DataONE Operations

The Data Observation Network for Earth (DataONE) is an NSF funded project in the DataNet program charged with implementing and deploying core services and infrastructure to improve the long term accessibility and re-use of data collected through NSF funded and other research projects.

These documents are primarily for internal use and provide information about operations of the DataONE service infrastructure.

The source for these documents is located in the subversion repository under:

/documents/Projects/cicore/operations

The HTML version of the documentation is generated from the current version of the source documents and is available from:

https://purl.dataone.org/operations/index.html

Contents

• Member Node Deployment Checklist
  • Overview
  • Planning
  • Developing
  • Testing
  • Operating
• Coordinating Nodes
  • Architecture
  • Connectivity and Certificates
  • Service Monitoring
  • Maintenance and Administration

Warning

2015-07-30 refactoring of the layout and content of thesedocuments is in progress.

Administration Topics

• Server Administration and Contacts
  • Organizations and Contacts
  • Administrative Procedures
• DNS Management for DataONE Nodes
  • Naming of VM Host Hardware
  • Naming of Coordinating Nodes
  • Current Situation (mid 2010)
  • Comments Section
• DataONE Cybersecurity Plan
  • General Principles
  • DataONE Institutional Components
  • Institutional Cybersecurity Requirements
  • DataONE Wide Cybersecurity Requirements
  • DataONE Cybersecurity Planning Posture Progression Through Project Lifetime
  • Cybersecurity Milestones in DataONE Project Year One
  • Approval Workflow
• Server Layout
• DataONE Deployment Environments
  • Dev Environment
  • Sandbox Environment
  • Staging Environment
  • Production Environment
• Coordination Node Software Stack Administration
  • Installed Software
  • Software Administrative Procedures
• Recipes for Baking Servers
  • Server Management
  • Coordinating Nodes

Virtual Machine Building

• Specifications for Hardware Implementing the CNs
• Configuration of Host Hardware and OS for Coordinating Nodes
  • Hardware
• Creating Virtual Machines
  • Create a KVM Guest
  • Create an OpenVZ Guest
  • Remote Administration of Virtual Machines
• Puppet
  • Install on the Server
  • Install on the Client
  • Create Certificate
• Timezone and NTP
  • Timezone
  • Network Time Protocol
• LDAP Authentication
  • Enable LDAP Authentication
  • Listing Group Members
• Deny Hosts
• Firewall Configuration
  • Suggested Member Node Firewall Configuration
  • Coordinating Node Firewall Configuration
• Monitoring Systems
  • Synopsis
  • Java JMX
  • Statsd
  • Check_mk
• Outgoing Email

Building Coordinating Nodes

• Create a Coordinating Node
  • Create the Virtual Machine
  • Install the Coordinating Node Software
  • Configuring a Coordinating Node
  • Updating a Coordinating Node
• CN Purge and Installation Instructions –Version 858
• CN Reset and/or Upgrade –Version 3268
• CN KNB Service Update Instructions –Version 0
• CN Approve Member Node Registration
• Create the Solr Cloud on the CN cluster
• Coordinating Node Deployment
  • Overview of Coordinating Node Software
  • Prerequisites
  • Installing a New Coordinating Node
  • Coordinating Node Instance Upgrade
  • Clearing the Object Store
  • Troubleshooting CN Cluster Communications

Development Notes

• Software Development Guidelines
  • General
  • Java
• Software Methodology
  • Modularization
• Programming Style
  • Python
• Java and Maven Setup
  • Developing Java Projects with Maven
• Product Version Control
  • Baseline assumptions
  • Product Deliverables
• Cutting a stable software release from development
  • Development or Maintenance of a Software Component
  • Release Control Tools
  • Testing and Staging of a Software Component
  • Production Release of a Software Component
  • Deliver Product to the Public
  • Miscellaneous Notes
• CN Search Index
  • Index Task Generator
  • Index Task Processor
  • Index Processor Configuration
  • Adding new science metadata format
  • Adding a new science metadata rule

Site Specific Notes

• Configuring NFS on VMs
  • KVM VMs
  • OpenVZ VMs
• UCSB KVM Host Setup
  • KVM Host Descriptions
  • CentOS Configuration
  • Create a New Virtual Server
  • Delete a Virtual Server
  • Resize a Virtual Server Disk Image
  • Live Migration
  • Renaming Hosts
  • Other Notes
• UCSB Storage Layout
  • iSCSI Block Device Mapping (KVM)
  • Physical Array Layout
• UCSB Virtual Server Inventory
  • UCSB/Campus
  • Unsupported SAN Storage

Installed Systems and Monitoring

• Hardware and Virtual Machine Inventory
  • (Potentially) Interesting URLs
• Monitoring Java Processes with JMX
  • Watching Hazelcast
  • Watching Tomcat
  • Listing JMX Beans
• Network Inventory
  • UCSB/Campus

Miscellaneous Tools

• Useful Utilities
  • Using dsh for System Monitoring
  • Using the Sparkle Share Space

Miscellaneous Designs

• Rough Draft of CN Audit SystemMetadata Design

Indices and tables

• Index
• Module Index
• Search Page
• glossary