Use Case 35 - Query Coordinating Node for Metadata Describing a Member Node

Use Case 34 - CNs Support Other Discovery Mechanisms (e.g. Google)

Operations to assist with diagnosing authentication and content
formatting.

Handles CN directed replication of content between Member Nodes.

Supports operations for replication of content between Member Nodes.

Core operations necessary for basic interaction with
Coordinating Nodes

Provide tier 1 operations for sharing content from a Member Node.

An optional tier 1 API for Member Node support of downloads of whole packages.

An optional tier 1 API for Member Node support of metadata and data views.

View operations to see formatted versions of metadata and data for Coordinating Nodes

The *xml* command is implemented by XMLStarlet_, and is used here to
format the output for easier reading. The *urlencode* command is a
script available with the `d1_client_bash`_ itk tool

, and is used here to
format the output for easier reading. The 

The register API methods are used to maintain a registry of nodes participating
in the DataONE infrastructure.

Note that the node registry is much the same as the Object collection with a
restriction on the returned object formats to be Member Nodes or Coordinating
Nodes. It may be prudent for the implementation of the registration API to
leverage the existing functionality of the object collection rather than
implementing a parallel data store. In this case, the "science metadata" could
be a DC description of the node, and the "data" might be the detailed
registration information including node capabilities, scheduling and so forth.

Warning: this file is automatically generated.  Edits will be lost

(:class:`session<Types.Session>`, :class:`nodeid<Types.NodeReference>`, :class:`node<Types.Node>`) ``->`` boolean

(:class:`nodeid<Types.NodeReference>`) ``->`` :class:`Types.Node`

(:class:`session<Types.Session>`, :class:`node<Types.Node>`) ``->`` :class:`Types.NodeReference`

For updating the capabilities of the specified node. Most information is replaced by information in the new node, however, the node identifier, nodeType, ping, syncrhonization.lastHarvested, and synchronization.lastCompleteHarvest are preserved from the existing entry.  Services in the old record not included in the new Node will be removed.

Successful completion of this operation is indicated by a HTTP response status code of 200.

Unsuccessful completion of this operation MUST be indicated by returning an appropriate exception.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session defaults to the public user if the certificate was not provided with the request. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session defaults to the 

 if the certificate was not provided with the request. Transmitted as part of the SSL handshake process.

nodeid (Types.NodeReference) -- The identifier of the existing node entry being updated. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the existing node entry being updated. Transmitted as part of the URL path and must be escaped accordingly.

node (Types.Node) -- An instance of :class`Types.Node` that contains the updated information.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

An instance of :class`Types.Node` that contains the updated information.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

Exceptions.NotImplemented -- (errorCode=501, detailCode=4820)The service is not implemented.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4821)The Subject does not have authority to modify the node registration information.

The :term:`Subject` does not have authority to modify the node registration information.

 does not have authority to modify the node registration information.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4823)The request was malformed.

Exceptions.NotFound -- (errorCode=404, detailCode=4824)The requested nodeid is not available in the registry.

For retrieving the capabilities of the specified node if it is registered on the Coordinating Node being called.

nodeid (Types.NodeReference) -- The identifier of the existing node entry being looked up. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the existing node entry being looked up. Transmitted as part of the URL path and must be escaped accordingly.

An instance of :class`Types.Node` that contains the Node information.

Exceptions.NotImplemented -- (errorCode=501, detailCode=4826)The service is not implemented.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4828)The request was malformed.

Exceptions.NotFound -- (errorCode=404, detailCode=4829)The requested nodeid is not available in the registry.

internal padding after source/apis/generated/generated_CNRegister.txt

Register a new node in the system. If the node already exists, then a :exc:`IdentifierNotUnique` exception MUST be returned.

Register a new node in the system. If the node already exists, then a 

node (Types.Node) -- An instance of Types.Node that fully describes the node being registered. Note that some attributes will be set by the Coordinating Node.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

 that fully describes the node being registered. Note that some attributes will be set by the Coordinating Node.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

The identifier of the new node entry if successful, otherwise an error is raised.

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=4844)

xxxx notes xxxx

.. function:: addNodeCapabilities (nodeID, capabilities) -> boolean

   Adds the supplied capabilities to the system metadata describing the Member Node.

   :param nodeID: A PID that identifies the Member Node.

   :param capabilities:
     A capabilities document that describes the resources and supported
     services of the Member Node.

   :returns: True if the operation succeeds

   :rtype: Boolean

.. function:: register (token, capabilitiesURL) -> boolean

   Used to register a new Member Node.

   :param token: An authentication token with priviledges to rgister a new Member Node.

   :param capabilitiesURL: The URL from which capabilities will be retrieved.

   :raises exceptions.NotAuthorized: The supplied identity does not have
     authority to register a node.

   :raises Exceptions.NotFound: The supplied capabilitiesURL does not resolve
     to a capabilities document

   :returns: True if the registration operation succeeds

.. function:: registerForNotification (token, dataOwnerID)

.. function:: verifyCapabilities (nodeID, capabilities) -> boolean

   Evaluates the supplied capabilities document. This may involve additional
   interaction with the MN as various aspects are evaluated.

.. todo::
   The :func:`verifyCapabilities` will need to check functionality of the MN so
   need to define a few service hooks for checking these properties.

State of Health API
~~~~~~~~~~~~~~~~~~~

The Health API contains methods for determining the status of the DataONE
infrastructure, including both the CNs and MNs.

.. module:: CN_health
   :synopsis: Methods for reporting on the state of health of a CN and on the entire DataONE infrastructure

::

  generateReport(token) -> statusReport

.. function:: actOnPendingEvent (event)

.. function:: reportPendingEvent (event)

.. function:: Admin Application

.. function:: notify (adminID, nodeID, service)

.. function:: generateReport(token) -> statusReport

   Returns a status report for all of the registered Member Nodes. The
   informatio returned is used to give administrators information necessary to
   maintain the infrastructure.

   .. note::
      This method is used by Use Case 10 "MN Status Reports" for V0.3. However
      instead of a complete set of detailed status reports, the V0.3
      implementation of the use case will utilize the ``MN_health.heartbeat()``
      method to compile very basic information about the MNs. Later versions of
      this method will use the richer ``MN_health.getStatus(token)`` method.

   :param token: Identity with authority to generate reports.

   :rtype: Status report, ideally in XML + stylsheet or at least XHTML.

setReplicationStatus(session,pid,nodeRef,status,failure) -> boolean

The Data Replication API operates in conjunction with the
:mod:`MNReplication` API to assist with the replication of data and
science metadata content between Member Nodes to ensure that copies of data
and metadata can be retrieved from more than one Member Node where possible.

 API to assist with the replication of data and

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`nodeRef<Types.NodeReference>`, :class:`status<Types.ReplicationStatus>`, :class:`failure<Types.BaseException>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`replicaMetadata<Types.Replica>`, ``serialVersion``) ``->`` boolean

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`, :class:`policy<Types.ReplicationPolicy>`, ``serialVersion``) ``->`` boolean

(:class:`session<Types.Session>`, :class:`targetNodeSubject<Types.Subject>`, :class:`pid<Types.Identifier>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`nodeId<Types.NodeReference>`, ``serialVersion``) ``->`` boolean

updateReplicationMetadata(session,pid,replicaMetadata,serialVersion) -> boolean

Replaces the replica with matching nodeRef in the system metadata of the specified object.  Adds a new replica if the nodeRef of passed in Replica is not already present.  Changes the date sys meta modified.

Successful completion of the operation is indicated by returning a HTTP status of 200.

Failure of the operation MUST be indicated by returning an appropriate exception.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service.  Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service.  Transmitted as part of the SSL handshake process.

pid (Types.Identifier) -- Transmitted as part of the URL path and must be escaped accordingly.

Transmitted as part of the URL path and must be escaped accordingly.

replicaMetadata (Types.Replica) -- Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

serialVersion (unsigned long) -- The serialVersion of the system metadata that is the intended target for the change.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The serialVersion of the system metadata that is the intended target for the change.  Transmitted as a UTF-8 String as a 

Exceptions.NotFound -- (errorCode=404, detailCode=4854)The specified pid does not exist

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4855)The serialVersion supplied with the request does not match the serialVersion of the target

The serialVersion supplied with the request does not match the serialVersion of the target

setReplicationPolicy(session,id,policy,serialVersion) -> boolean

Updates the replication policy entry for an object by updating the system metadata.

id (Types.Identifier) -- The identifier of the policy being updated. May be either a PID or a SID, the latter acting only on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the policy being updated. May be either a PID or a SID, the latter acting only on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

policy (Types.ReplicationPolicy) -- Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

serialVersion (unsigned long) -- The serialVersion of the system metadata that is the intended target for the change. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The serialVersion of the system metadata that is the intended target for the change. Transmitted as a UTF-8 String as a 

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4886)The serialVersion supplied with the request does not match the serialVersion of the target

Verifies that a replication event was initiated by a CN by comparing the target node's identifiying subject with a known list of scheduled replication tasks.

targetNodeSubject (Types.Subject) -- The subject that identifies the target node, with a value extracted from the X.509 certificate passed in during the call to MNReplication.replicate(). Transmitted as a URL query parameter, and so must be escaped accordingly.

The subject that identifies the target node, with a value extracted from the X.509 certificate passed in during the call to MNReplication.replicate(). Transmitted as a URL query parameter, and so must be escaped accordingly.

pid (Types.Identifier) -- The identifier of the object to be replicated. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to be replicated. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotImplemented -- (errorCode=501, detailCode=4870)The method functionality is not implemented.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4871)Replication of PID is not authorized for the subject listed.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4872)An internal server error occurred.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4873)The replication request is invalid.

Exceptions.NotFound -- (errorCode=404, detailCode=4874)The PID does not exist as an existing object.

Exceptions.InvalidToken -- (errorCode=401, detailCode=4875)The session information is invalid.

deleteReplicationMetadata(session,pid,nodeId,serialVersion) -> boolean

internal padding after source/apis/generated/generated_CNReplication.txt

Removes the replication information for the specified node from the object system metadata identified by *pid*.

Removes the replication information for the specified node from the object system metadata identified by 

Removal of replication metadata is necessary if the Member Node goes offline permanently or for an extended period, or when it is deeemed prudent to migrate an object from one node to another to address resource management issues.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session MUST be a Coordinating Node. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session MUST be a Coordinating Node. Transmitted as part of the SSL handshake process.

pid (Types.Identifier) -- The identifier of the object whose replication metadata is being modified.  Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object whose replication metadata is being modified.  Transmitted as part of the URL path and must be escaped accordingly.

nodeId (Types.NodeReference) -- The identifier of the node replication information that is being removed from the system metadata record. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The identifier of the node replication information that is being removed from the system metadata record. Transmitted as a UTF-8 String as a 

serialVersion (unsigned long) -- The Types.SystemMetadata.serialVersion of the system metadata being updated. This MUST match the latest version of system metadata available for the object on the Coordinating Node. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

 of the system metadata being updated. This MUST match the latest version of system metadata available for the object on the Coordinating Node. Transmitted as a UTF-8 String as a 

Exceptions.NotImplemented -- (errorCode=501, detailCode=4950)The method functionality if not fully implemented

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4951)An internal server error occurred that prevented the operation from completing.

An internal server error occurred that prevented the operation from completing.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4952)The request parameters are malformed

Exceptions.InvalidToken -- (errorCode=401, detailCode=4953)The supplied session is invalid

Exceptions.NotFound -- (errorCode=404, detailCode=4956)The object identified by pid or the node reference specified by nodeId could not be located in the system metadata for the object.

The object identified by *pid* or the node reference specified by *nodeId* could not be located in the system metadata for the object.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4954)The subject identified by the session information does not have appropriate priviledges for modifiying the content or accessing the service.

The subject identified by the *session* information does not have appropriate priviledges for modifiying the content or accessing the service.

 information does not have appropriate priviledges for modifiying the content or accessing the service.

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4955)The serialVersion does not match the current Types.SystemMetadata.serialVersion value of the object system metadata.

The *serialVersion* does not match the current :attr:`Types.SystemMetadata.serialVersion` value of the object system metadata.

This old stuff is commented out but kept here for revisiting later
.. function:: getDefaultReplicationPolicy (sess)

Update the replication status of the system metadata, ensuring that the change is appropriate for the given state of system metadata.  For example, a MN can not change the status to *COMPLETED* unless the CN previously requested replication of the object and the replications status of the object (as indicated in the system metadata) is set to *QUEUED*.

Update the replication status of the system metadata, ensuring that the change is appropriate for the given state of system metadata.  For example, a MN can not change the status to 

 unless the CN previously requested replication of the object and the replications status of the object (as indicated in the system metadata) is set to 

The nodeRef, status, and failure parameters are transmitted as part of the HTTP request body encoded as a MIME Multipart/form-data encoded payload.

This method can be only called by Coordinating Nodes and trusted Member Nodes.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session defaults to the public user if the certificate was not provided with the request. Transmitted as part of the SSL handshake process.

pid (Types.Identifier) -- Identifier of the object to be replicated between Member Nodes. Transmitted as part of the URL path and must be escaped accordingly.

Identifier of the object to be replicated between Member Nodes. Transmitted as part of the URL path and must be escaped accordingly.

nodeRef (Types.NodeReference) -- Reference to the Node which made the setReplicationStatus call. If this is a Member Node, the checksum must be compared with that of the authoritative Member Node. If not, this step can be ignored as the call is not signifying a replication is complete. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

Reference to the Node which made the setReplicationStatus call. If this is a Member Node, the checksum must be compared with that of the authoritative Member Node. If not, this step can be ignored as the call is not signifying a replication is complete. Transmitted as a UTF-8 String as a 

status (Types.ReplicationStatus) -- Replication status. See system metadata schema for possible values.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

Replication status. See system metadata schema for possible values.  Transmitted as a UTF-8 String as a 

failure (Types.BaseException) -- A BaseException object or one of it's subclasses, or null.  If the status is set to 'failed', this exception object can provide more detail. Appropriate sub-classes include InsufficientResource, NotAuthorized, ServiceFailure, etc. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

A BaseException object or one of it's subclasses, or null.  If the status is set to 'failed', this exception object can provide more detail. Appropriate sub-classes include InsufficientResource, NotAuthorized, ServiceFailure, etc. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

True if the operation is allowed and expected, otherwise an exception should be raised.

The *CN_read* API implements methods that enable object retrieval operations
on a :term:`Coordinating Node`. It includes searches of science metadata and
system metadata and exposes log records held by CNs.

 API implements methods that enable object retrieval operations
on a 

. It includes searches of science metadata and

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.OctetStream`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.SystemMetadata`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.DescribeResponse`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.ObjectLocationList`

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`) ``->`` :class:`Types.Checksum`

(:class:`session<Types.Session>`, :class:`[fromDate]<Types.DateTime>`, :class:`[toDate]<Types.DateTime>`, :class:`[formatId]<Types.ObjectFormatIdentifier>`, :class:`[identifier]<Types.Identifier>`, ``[start=0]``, ``[count=1000]``, :class:`[nodeId]<Types.NodeReference>`) ``->`` :class:`Types.ObjectList`

(:class:`session<Types.Session>`, ``queryType``, ``query``) ``->`` :class:`Types.ObjectList`

(:class:`session<Types.Session>`, ``queryEngine``, ``query``) ``->`` :class:`Types.OctetStream`

(:class:`session<Types.Session>`, ``queryEngine``) ``->`` :class:`Types.QueryEngineDescription`

(:class:`session<Types.Session>`) ``->`` :class:`Types.QueryEngineList`

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`) ``->`` boolean

Retrieves the object identified by *id* from the node. If the object is not present on the node, then an :exc:`Exceptions.NotFound` error is raised, regardless of whether the object exists on another node in the DataONE system.

 from the node. If the object is not present on the node, then an 

 error is raised, regardless of whether the object exists on another node in the DataONE system.

v2.0: The supplied identifier may be a :term:`PID` or a :term:`SID`.

:doc:`UC01 </design/UseCases/01_uc>`, :doc:`UC16 </design/UseCases/16_uc>`, :doc:`UC09 </design/UseCases/09_uc>`

id (Types.Identifier) -- Identifier of the object to be retrieved. May be either a PID or a SID, the latter acting as if called using the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier of the object to be retrieved. May be either a PID or a SID, the latter acting as if called using the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

For science metadata objects, this will be the exact byte stream of the science metadata object as it was original ingested. Note that additional object types may in the future be returned by the get method.

Exceptions.NotFound -- (errorCode=404, detailCode=1020)The object specified by the identifier is not present on this (or any) CN. The response body should contain a reference to CNRead.resolve().

The object specified by the identifier is not present on this (or any) CN. The response body should contain a reference to :func:`CNRead.resolve`.

The object specified by the identifier is not present on this (or any) CN. The response body should contain a reference to 

See :func:`MNRead.getSystemMetadata` for additional details and examples.

This method provides a lighter weight mechanism than :func:`CNRead.getSystemMetadata` for a client to determine basic properties of the referenced object. The response should indicate properties that are typically returned in a HTTP HEAD request: the date late modified, the size of the object, the type of the object (the :attr:`SystemMetadata.formatId`).

 for a client to determine basic properties of the referenced object. The response should indicate properties that are typically returned in a HTTP HEAD request: the date late modified, the size of the object, the type of the object (the 

The principal indicated by *token* must have read privileges on the object, otherwise :exc:`Exceptions.NotAuthorized` is raised.

If the object does not exist on the node servicing the request, then :exc:`Exceptions.NotFound` must be raised even if the object exists on another node in the DataONE system.

If the object does not exist on the node servicing the request, then 

 must be raised even if the object exists on another node in the DataONE system.

Note that this method is likely to be called frequently and so efficiency should be taken into consideration during implementation.

id (Types.Identifier) -- Identifier for the object in question. May be either a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier for the object in question. May be either a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

(HEAD) Retrieve information about the object with identifier "ABC123"::

(HEAD) Retrieve information about the object with identifier "ABC123":

curl -I http://mn1.dataone.org/mn/v1/object/ABC123

HTTP/1.1 200 OK
Last-Modified: Wed, 16 Dec 2009 13:58:34 GMT
Content-Length: 10400
Content-Type: application/octet-stream
DataONE-ObjectFormat: eml://ecoinformatics.org/eml-2.0.1
DataONE-Checksum: SHA-1,2e01e17467891f7c933dbaa00e1459d23db3fe4f
DataONE-SerialVersion: 1234

(HEAD) An error response to a describe() request for object "IDONTEXIST"::

(HEAD) An error response to a describe() request for object "IDONTEXIST":

curl -I http://mn1.dataone.org/mn/v1/object/IDONTEXIST

HTTP/1.1 404 Not Found
Content-Length: 1182
Content-Type: text/xml
DataONE-Exception-Name: NotFound
DataONE-Exception-DetailCode: 1380
DataONE-Exception-Description: The specified object does not exist on this node.
DataONE-Exception-PID: IDONTEXIST

Returns a list of nodes (MNs or CNs) known to hold copies of the object identified by *id*. The object resolution process is intended to provide a simple mechanism for a client to discover from which node(s) a particular object may be retrieved. Details about method interfaces (i.e. REST URLs) exposed by a particular node can be determined by examining the response from the *node* collection. For convenience, the :func:`MNRead.get` URL is included in the response as is the base URL of the node REST services.

Returns a list of nodes (MNs or CNs) known to hold copies of the object identified by 

. The object resolution process is intended to provide a simple mechanism for a client to discover from which node(s) a particular object may be retrieved. Details about method interfaces (i.e. REST URLs) exposed by a particular node can be determined by examining the response from the 

 URL is included in the response as is the base URL of the node REST services.

Note also that the same functionality as *resolve()* can be implemented by retrieving a copy of the system metadata for the object and utilizing the node registry to discover the base URL from which the client can construct the *get()* URL. Resolve is provided for efficiency since the response size is much smaller.

 can be implemented by retrieving a copy of the system metadata for the object and utilizing the node registry to discover the base URL from which the client can construct the 

 URL. Resolve is provided for efficiency since the response size is much smaller.

Resolve will return a HTTP status of 303 (see other) on success. The HTTP header "Location" MUST be set, and it's value SHOULD be the full get() URL for retrieving the object from the first location in the resolve response.

v2.0: The supplied identifier may be a :term:`PID` or a :term:`SID`. If the identifier is a SID, then resolution is for the latest version of an object (i.e. the head of the obsolescence chain).

. If the identifier is a SID, then resolution is for the latest version of an object (i.e. the head of the obsolescence chain).

:doc:`UC36 </design/UseCases/36_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

id (Types.Identifier) -- Identifier being resolved. May be either a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier being resolved. May be either a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

A list of nodes known to contain copies of the target object, plus the URLs known to resolve to the node get methods.

Exceptions.NotFound -- (errorCode=404, detailCode=4140)There is no object identified by the given identifier

curl -X GET -H "Accept: text/xml" http://cn.dataone.org/object/resolve/1234
or
curl -X GET -H "Accept: */*" http://cn.dataone.org/object/resolve/1234

<objectLocationList xmlns:d1="http://ns.dataone.org/service/types/v1">
  <identifier>1234</identifier>
  <objectLocation>
    <nodeIdentifier>mn1</nodeIdentifier>
    <baseURL>http://mn1.dataone.org/mn</baseURL>
    <version>v1</version>
    <url>http://mn1.dataone.org/mn/object/1234</url>
  </objectLocation>
    <nodeIdentifier>mn2</nodeIdentifier>
    <baseURL>http://mn2.dataone.org/mn</baseURL>
    <url>http://mn2.dataone.org/some_base/object/1234</url>
    <nodeIdentifier>cn1</nodeIdentifier>
    <baseURL>http://cn1.dataone.org/mn</baseURL>
    <url>http://cn1.dataone.org/cn/object/1234</url>
    <nodeIdentifier>cn2</nodeIdentifier>
    <baseURL>http://cn2.dataone.org/mn</baseURL>
    <url>http://cn2.dataone.org/cn/object/1234</url>
    <nodeIdentifier>cn3</nodeIdentifier>
    <baseURL>http://cn3.dataone.org/mn</baseURL>
    <url>http://cn3.dataone.org/cn/object/1234</url>
</objectLocationList>

Returns the checksum for the specified object as reported in the system metadata.

Note that the signature of this method differs from :func:`MNRead.getChecksum` as that method takes an optional algorithm parameter.

pid (Types.Identifier) -- Identifier of the object for which checksum is being requested Transmitted as part of the URL path and must be escaped accordingly.

Identifier of the object for which checksum is being requested Transmitted as part of the URL path and must be escaped accordingly.

session, [fromDate], [toDate], [formatId], [identifier], [start=0], [count=1000], [nodeId]

Retrieve the list of objects present on the CN that match the calling parameters. At a minimum, this method should be able to return a list of objects that match::

Retrieve the list of objects present on the CN that match the calling parameters. At a minimum, this method should be able to return a list of objects that match:

but is expected to also support date range (by also specifying *toDate*), and should also support slicing of the matching set of records by indicating the starting *index* of the response (where 0 is the index of the first item) and the *count* of elements to be returned.

), and should also support slicing of the matching set of records by indicating the starting 

 of the response (where 0 is the index of the first item) and the 

Note that date time precision is limited to one millisecond. If no timezone information is provided, the UTC will be assumed.

:doc:`UC06 </design/UseCases/06_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

fromDate (Types.DateTime) -- Entries with SystemMetadata.dateSysMetadataModified greater than or equal to (>=) fromDate must be returned.  Transmitted as a URL query parameter, and so must be escaped accordingly.

 must be returned.  Transmitted as a URL query parameter, and so must be escaped accordingly.

toDate (Types.DateTime) -- Entries with SystemMetadata.dateSysMetadataModified less than (<) toDate must be returned. Transmitted as a URL query parameter, and so must be escaped accordingly.

 must be returned. Transmitted as a URL query parameter, and so must be escaped accordingly.

formatId (Types.ObjectFormatIdentifier) -- Restrict results to the specified object format. Transmitted as a URL query parameter, and so must be escaped accordingly.

Restrict results to the specified object format. Transmitted as a URL query parameter, and so must be escaped accordingly.

identifier (Types.Identifier) -- Restrict results to the specified identifier. May be either a PID or a SID. If the latter, will return results for each PID in the series. Transmitted as a URL query parameter, and so must be escaped accordingly.

Restrict results to the specified identifier. May be either a PID or a SID. If the latter, will return results for each PID in the series. Transmitted as a URL query parameter, and so must be escaped accordingly.

start=0 (integer) -- The zero-based index of the first value, relative to the first record of the resultset that matches the parameters. Transmitted as a URL query parameter, and so must be escaped accordingly.

The zero-based index of the first value, relative to the first record of the resultset that matches the parameters. Transmitted as a URL query parameter, and so must be escaped accordingly.

count=1000 (integer) -- The maximum number of entries that should be returned in the response. The Member Node may return fewer and the caller should check the total in the response to determine if further pages may be retrieved. Transmitted as a URL query parameter, and so must be escaped accordingly.

The maximum number of entries that should be returned in the response. The Member Node may return fewer and the caller should check the 

 in the response to determine if further pages may be retrieved. Transmitted as a URL query parameter, and so must be escaped accordingly.

nodeId (Types.NodeReference) -- Restrict results to those with authoritativeMemberNode equal to nodeId. Transmitted as a URL query parameter, and so must be escaped accordingly.

Restrict results to those with authoritativeMemberNode equal to nodeId. Transmitted as a URL query parameter, and so must be escaped accordingly.

The list of PIDs that match the query criteria. If none match, an empty list is returned.

Exceptions.NotImplemented -- (errorCode=501, detailCode=1560)Raised if some functionality requested is not implemented. In the case of an optional request parameter not being supported, the errorCode should be 400. If the requested format (through HTTP Accept headers) is not supported, then the standard HTTP 406 error code should be returned.

Raised if some functionality requested is not implemented. In the case of an optional request parameter not being supported, the errorCode should be 400. If the requested format (through HTTP Accept headers) is not supported, then the standard HTTP 406 error code should be returned.

Search the metadata catalog and return identifiers of metadata records that match the criteria.

Search may be implemented by more than one type of search engine. The queryType parameter indicates which search engine should be targeted. The value and form of *query* is determined by the search engine.

Search may be implemented by more than one type of search engine. The queryType parameter indicates which search engine should be targeted. The value and form of 

:doc:`UC02 </design/UseCases/02_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

queryType (string) -- Indicates which search engine will be used to handle the query. Currently supported search engines include: "SOLR". Transmitted as part of the URL path and must be escaped accordingly.

Indicates which search engine will be used to handle the query. Currently supported search engines include: "SOLR". Transmitted as part of the URL path and must be escaped accordingly.

query (string) -- The remainder of the URL is passed verbatim to the respective search engine implementation. Hence it may contain additional path elements and query elements as determined by the functionality of the search engine. The caller is reponsible for providing a '?' to indicate the start of the query string portion of the URL, as well as proper URL escaping.   Transmitted as part of the URL path and must be escaped accordingly.

The remainder of the URL is passed verbatim to the respective search engine implementation. Hence it may contain additional path elements and query elements as determined by the functionality of the search engine. The caller is reponsible for providing a '?' to indicate the start of the query string portion of the URL, as well as proper URL escaping.   Transmitted as part of the URL path and must be escaped accordingly.

Submit a query against the specified *queryEngine* and return the response as formatted by the queryEngine.

The *query()* operation may be implemented by more than one type of search engine and the *queryEngine* parameter indicates which search engine is targeted. The value and form of *query* is determined by the specific query engine.

 operation may be implemented by more than one type of search engine and the 

 parameter indicates which search engine is targeted. The value and form of 

For example, the solr search engine will accept many of the standard parameters of solr, including field restrictions and faceting.

queryEngine (string) -- Indicates which search engine will be used to handle the query. Supported search engines can be determined through the CNRead.listQueryEngines API call. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which search engine will be used to handle the query. Supported search engines can be determined through the CNRead.listQueryEngines API call. Transmitted as part of the URL path and must be escaped accordingly.

The structure of the response is determined by the chosen search engine and parameters provided to it.

Exceptions.NotFound -- (errorCode=404, detailCode=4325)The specified queryEngine does not exist

getQueryEngineDescription(session,queryEngine) -> QueryEngineDescription

Provides metadata about the query service of the specified *queryEngine*. The metadata provides a brief description of the query engine, its version, its schema version, and an optional list of fields supported by the query engine.

. The metadata provides a brief description of the query engine, its version, its schema version, and an optional list of fields supported by the query engine.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate provided with the request. The certificate must be traceable to an authority recognized by DataONE, currently CILogon. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate provided with the request. The certificate must be traceable to an authority recognized by DataONE, currently CILogon. Transmitted as part of the SSL handshake process.

queryEngine (string) -- Indicates which query engine for which to provide descriptive metadata. Currently supported search engines can be determined through CNRead.listQueryEngines. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which query engine for which to provide descriptive metadata. Currently supported search engines can be determined through CNRead.listQueryEngines. Transmitted as part of the URL path and must be escaped accordingly.

A list of fields that are supported by the search index and additional metadata.

Exceptions.NotFound -- (errorCode=404, detailCode=4414)The specified queryEngine does not exist

Returns a list of query engines, i.e. supported values for the *queryEngine* parameter of the *getQueryEngineDescription* and *query* operations.

The list of search engines available may be influenced by the authentication status of the request.

A list of names of queryEngines available to the user identified by session.

A list of names of queryEngines available to the user identified by 

internal padding after source/apis/generated/generated_CNRead.txt

Indicates to the CN that a new or existing object identified by PID requires synchronization. Note that this operation is asynchronous, a successful return indicates that the synchronization task was successfully queued.

This method may be called by any Member Node for new content or the authoritative Member Node for updates to existing content.

The CN will schedule the synchronization task which will then be processed in the same way as content changes identified through the listObjects polling mechanism.

pid (Types.Identifier) -- Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

True if the synchronization request was successfully queued, otherwise False

Returns the :term:`system metadata` that contains DataONE specific information about the object identified by *id*. Authoritative copies of system metadata are only available from the Coordinating Nodes.

 that contains DataONE specific information about the object identified by 

. Authoritative copies of system metadata are only available from the Coordinating Nodes.

v2.0: The supplied identifier may be a :term:`PID` or a :term:`SID` and the returned :class:`v2_0.Types.SystemMetadata` structure has changed.

:doc:`UC36 </design/UseCases/36_uc>`, :doc:`UC37 </design/UseCases/37_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

id (Types.Identifier) -- Identifier for the object of interest. May be either a PID or a SID, the latter acts as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier for the object of interest. May be either a PID or a SID, the latter acts as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=1040)The principal identified by token does not have READ permission on the object.

The principal identified by token does not have READ permission on the object.

Exceptions.NotFound -- (errorCode=404, detailCode=1060)There is no object identified by pid and so no system metadata can be returned.

There is no object identified by *pid* and so no system metadata can be returned.

(:class:`session<Types.Session>`, :class:`person<Types.Person>`) ``->`` :class:`Types.Subject`

(:class:`session<Types.Session>`, :class:`subject<Types.Subject>`, :class:`person<Types.Person>`) ``->`` :class:`Types.Subject`

(:class:`session<Types.Session>`, :class:`subject<Types.Subject>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`subject<Types.Subject>`) ``->`` :class:`Types.SubjectInfo`

(:class:`session<Types.Session>`, ``query``, ``status``, ``start``, ``count``) ``->`` :class:`Types.SubjectInfo`

(:class:`session<Types.Session>`, :class:`primarySubject<Types.Subject>`, :class:`secondarySubject<Types.Subject>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`group<Types.Group>`) ``->`` :class:`Types.Subject`

(:class:`session<Types.Session>`, :class:`group<Types.Group>`) ``->`` boolean

Note that there should probably be a lot more metadata captured about the new user, and there should be a mechanism for specifying the default access control rules for the new account.

person (Types.Person) -- Information about the Person to be registered with the account, including the real name and email address for the individual.  The Subject with the Person must match the subject of the X.509 certificate associated with the authenticated SSL session via client-side authentication, and must not have been registered previously. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

Information about the Person to be registered with the account, including the real name and email address for the individual.  The Subject with the Person must match the subject of the X.509 certificate associated with the authenticated SSL session via client-side authentication, and must not have been registered previously. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

The new subject in the DataONE system.  This may be a simple identifier.

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=4521)The chosen identity already exists in the system

Exceptions.InvalidCredentials -- (errorCode=401, detailCode=4522)Raised if the supplied credentials are invalid, such as an invalid X.509 certificate.

Raised if the supplied credentials are invalid, such as an invalid X.509 certificate.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4525)The subject of the session does not match the subject of the person.

The subject of the session does not match the subject of the person.

Update an existing :term:`subject` in the DataONE system. The target subject is determined from subject provided in the URL.

 in the DataONE system. The target subject is determined from subject provided in the URL.

The use calling this method must have write access to the account details.

Note that there should be a policy for verifying the details that change via this method.

subject (Types.Subject) -- The subject of the person being updated. Transmitted as part of the URL path and must be escaped accordingly.

The subject of the person being updated. Transmitted as part of the URL path and must be escaped accordingly.

person (Types.Person) -- New information about the Person. The subject of the Person cannot be updated with this method and must match the subject of the X.509 certificate associated with the authenticated SSL session via client-side authentication. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

New information about the Person. The subject of the Person cannot be updated with this method and must match the subject of the X.509 certificate associated with the authenticated SSL session via client-side authentication. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

Exceptions.InvalidCredentials -- (errorCode=401, detailCode=4531)Raised if the supplied credentials are invalid, such as an invalid X.509 certificate.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4534)Raised if the subject of the session does not match that of the person.

Raised if the subject of the session does not match that of the person.

Exceptions.NotFound -- (errorCode=404, detailCode=4535)Raised if the account does not exist.

Verify that the Person data associated with this Subject is a true representation of the real world person.

This service can only be called by users who have an administrative role for the domain of users in question.

A successful completion of this operation is indicated by returning a HTTP status of 200.

An exeption MUST be returned if the account verification is not successful.

subject (Types.Subject) -- The Subject identifier of the Person to be verified.  After this service is called for a subject by an authorized user, the account is marked as verified by this user. Transmitted as part of the URL path and must be escaped accordingly.

The Subject identifier of the Person to be verified.  After this service is called for a subject by an authorized user, the account is marked as verified by this user. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4541)This error is raised if the person attempting to validate the account is not authorized to verify accounts.

This error is raised if the person attempting to validate the account is not authorized to verify accounts.

Get the information about a Person (their equivalent identities, and the Groups to which they belong) or the Group (including members).

subject (Types.Subject) -- The Subject identifier of the Person or Group details to be returned.  Transmitted as part of the URL path and must be escaped accordingly.

The Subject identifier of the Person or Group details to be returned.  Transmitted as part of the URL path and must be escaped accordingly.

The Person or Group details are contained in the returned SubjectList for the given Subject

Exceptions.NotFound -- (errorCode=404, detailCode=4564)Raised if the requested subject is not registered with the DataONE system

Raised if the requested subject is not registered with the DataONE system

Create a new mapping between the two identities, asserting that they represent the same subject.

Mapping identities with this method requires explicit authorization for the user given in the Session object. The caller must have made sure that the primary and secondary identities represent one and the same individual.

Successful completion of the request is indicated by returning a HTTP status of 200.

A failed request MUST be indicated by returning an appropriate exception and setting the response HTTP status accordingly.

primarySubject (Types.Subject) -- The Subject identifier that will have a mapped identity to the other given subject.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The Subject identifier that will have a mapped identity to the other given subject.  Transmitted as a UTF-8 String as a 

secondarySubject (Types.Subject) -- The Subject identifier that will have a mapped identity to the other given subject.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

True if the mapping was successfully initiated, false otherwise.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=2360)The supplied principal does not have permission to map these two identities

The supplied principal does not have permission to map these two identities

Exceptions.NotFound -- (errorCode=404, detailCode=2340)The specified principal does not exist in the DataONE system

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=2343)The subject of the session and the provided subject are the same

The subject of the session and the provided subject are the same

Removes a previously asserted identity mapping from the Subject in the Session to the Subject given by the parameter. The reciprocol mapping entry is also removed.

A successful request is indicated by returning a HTTP status of 200.

subject (Types.Subject) -- The Subject identifier to be used for equivalentIdentity. This Subject will not match the Subject named in the certificate. Transmitted as part of the URL path and must be escaped accordingly.

The Subject identifier to be used for equivalentIdentity. This Subject will not match the Subject named in the certificate. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2340)The specified principal does not exist in the DataONE system, or the mapping between the subjects has no yet been initiated.

The specified principal does not exist in the DataONE system, or the mapping between the subjects has no yet been initiated.

Request a new mapping between the authenticated identity in the session and the given identity, asserting that they represent the same subject.

Mapping identities is a two-step process wherein a map request is made by a primary Subject and a subsequent (confirmation) map request is made by the secondary Subject. This ensures that mappings are performed only by those that have authority to do so.

subject (Types.Subject) -- The Subject identifier to be used for equivalentIdentity. This Subject will not match the Subject named in the certificate. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The Subject identifier to be used for equivalentIdentity. This Subject will not match the Subject named in the certificate. Transmitted as a UTF-8 String as a 

Confirms a previously initiated identity mapping. If subject A asserts that B is the same identity through :func:`CNIdentity.requestMapIdentity`, then this method is called by B to confirm that assertion.

Confirms a previously initiated identity mapping. If subject A asserts that B is the same identity through 

Gets the SubjectInfo of a previously initiated identity mapping.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=2360)The supplied principal does not have permission to get the SubjectInfo

The supplied principal does not have permission to get the SubjectInfo

Denies a previously initiated identity mapping. If subject A asserts that B is the same identity through :func:`CNIdentity.requestMapIdentity`, then this method is called by B to deny that assertion.

Denies a previously initiated identity mapping. If subject A asserts that B is the same identity through 

Groups are lists of subjects that allow all members of the group to be referenced by listing solely the subject name of the group.  Group names must be unique within the DataONE system. Groups can only be modified by Subjects listed as rightsHolders.

group (Types.Group) -- The Group to be created. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

The Group to be created. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

Exceptions.NotAuthorized -- (errorCode=401, detailCode=2460)The supplied principal does not have permission to create a group

The supplied principal does not have permission to create a group

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=2400)A group by this name already exists

internal padding after source/apis/generated/generated_CNIdentity.txt

Group members can be modified only by the original creator of the group, otherwise a NotAuthorized exception is thrown.  Group members are provided as a list of subjects that replace the group membership.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session is used to determine  if the request is being made by the creator of the group. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The subject of the session is used to determine  if the request is being made by the creator of the group. Transmitted as part of the SSL handshake process.

group (Types.Group) -- The new Group object that will replace the old Group. The Group.Subject must match the groupName and an update cannot modify this value. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

The new Group object that will replace the old Group. The Group.Subject must match the groupName and an update cannot modify this value. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

True if the group that was modified successfully, false otherwise

Exceptions.NotAuthorized -- (errorCode=401, detailCode=2560)The supplied principal does not have permission to add to a group

The supplied principal does not have permission to add to a group

Exceptions.NotFound -- (errorCode=404, detailCode=2540)The specified group does not exist in the DataONE system

List the subjects, including users, groups, and systems, that match search criteria.

The list can be restricted to subjects whose identifier matches certain substrings, and the size of the resultset can be paged through.

query (string) -- A query string criteria to be matched using a case-insensitive substring match against the identifier for the principal, the givenName or familyName of users, and the groupName of groups.  The function returns the union of all successful matches against these fields. Transmitted as a URL query parameter, and so must be escaped accordingly.

A query string criteria to be matched using a case-insensitive substring match against the identifier for the principal, the givenName or familyName of users, and the groupName of groups.  The function returns the union of all successful matches against these fields. Transmitted as a URL query parameter, and so must be escaped accordingly.

status (string) -- When provided, the status field can limit the returned list of Subjects to only those with the given status. Currently the 'verified' status is supported. Transmitted as a URL query parameter, and so must be escaped accordingly.

When provided, the status field can limit the returned list of Subjects to only those with the given status. Currently the 'verified' status is supported. Transmitted as a URL query parameter, and so must be escaped accordingly.

start (integer) -- The starting record number for the records to be returned (default = 0). Transmitted as a URL query parameter, and so must be escaped accordingly.

The starting record number for the records to be returned (default = 0). Transmitted as a URL query parameter, and so must be escaped accordingly.

count (integer) -- The maximum number of entries that should be returned in the response. The Member Node may return fewer and the caller should check the total in the response to determine if further pages may be retrieved (default = 100). Transmitted as a URL query parameter, and so must be escaped accordingly.

 in the response to determine if further pages may be retrieved (default = 100). Transmitted as a URL query parameter, and so must be escaped accordingly.

The *CNView* API implements methods that enable viewing content
on a :term:`Coordinating Node`. Like the MNView service, the CNView service provides a transformed view of a metadata file, data file, or package.  The CNView service provides a default view for all content, and may choose to redirect a review request to the authoritative Member Node for a given PID.

. Like the MNView service, the CNView service provides a transformed view of a metadata file, data file, or package.  The CNView service provides a default view for all content, and may choose to redirect a review request to the authoritative Member Node for a given PID.

(:class:`session<Types.Session>`, ``theme``, :class:`id<Types.Identifier>`) ``->`` :class:`Types.OctetStream`

(:class:`session<Types.Session>`) ``->`` :class:`Types.OptionList`

Provides a formatted view of an object (science metadata, data, resource, or other) using the given named theme.

The service :func:`CNView.view` operation will implement at least one {theme} named 'default' to provide a standard (possibly minimalistic) view of the content in HTML format. In addition, a CN may redirect a client to the view service of the authoritative Member Node for a PID if that node has implemented the :func:`MNView.view` service and implements a compatible theme.

 operation will implement at least one {theme} named 'default' to provide a standard (possibly minimalistic) view of the content in HTML format. In addition, a CN may redirect a client to the view service of the authoritative Member Node for a PID if that node has implemented the 

If the {theme} parameter is not recognized, the service must render the object using the default theme rather than throwing an error.  Note that the return type of Types.OctetStream requires that the consuming client has a priori knowledge of the theme being returned (like HTML). Response headers must include the correct mime-type of the view being returned.

theme (string) -- Indicates which themed view will be used to handle the query. All implementations must support a 'default' HTML theme, but are free to implement additional themes that return both HTML and non-HTML responses. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which themed view will be used to handle the query. All implementations must support a 'default' HTML theme, but are free to implement additional themes that return both HTML and non-HTML responses. Transmitted as part of the URL path and must be escaped accordingly.

id (Types.Identifier) -- The identifier of the object to render in a view. May be a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to render in a view. May be a PID or a SID, the latter acting as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Any return type is allowed, including application/octet-stream, but the format of the response should be specialized by the requested theme.

Exceptions.NotFound -- (errorCode=404, detailCode=2855)The specified pid does not exist.

internal padding after source/apis/generated/generated_CNView.txt

Provides a list of usable themes for rendering content in a view, including a required 'default' theme. The list of themes is provided as an OptionList, where the option key should be used as the theme name in calls to MNView.view, and the description provides a human readable description of what will be returned fo rthat theme.

A list of available themes that can be used with the MNView.view service.

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`, :class:`userId<Types.Subject>`, ``serialVersion``) ``->`` :class:`Types.Identifier`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`, :class:`action<Types.Permission>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`, :class:`accessPolicy<Types.AccessPolicy>`, ``serialVersion``) ``->`` boolean

Changes ownership (RightsHolder) of the specified object to the :term:`subject` specified by `userId`

Changes ownership (RightsHolder) of the specified object to the 

id (Types.Identifier) -- Identifier of the object to be modified. May be either a PID or a SID, the latter acting on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier of the object to be modified. May be either a PID or a SID, the latter acting on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

userId (Types.Subject) -- The subject that will be taking ownership of the specified object.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The subject that will be taking ownership of the specified object.  Transmitted as a UTF-8 String as a 

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4440)The supplied subject does not have permission to change ownership of the object

The supplied subject does not have permission to change ownership of the object

Exceptions.NotFound -- (errorCode=404, detailCode=4460)The specified object does not exist in the DataONE system

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4443)The serialVersion supplied with the request does not match the serialVersion of the target

Test if the user identified by the provided token has authorization for operation on the specified object.

A successful operation is indicated by a return HTTP status of 200.

Failure is indicated by an exception such as :exc:`NotAuthorized` being returned.

A successful response is indicated by a response HTTP status of 200. The body of the response is arbitrary and SHOULD be ignored by the caller.

If the action is not authorized, then a :exc:`NotAuthorized` exception MUST be raised.

:doc:`UC01 </design/UseCases/01_uc>`, :doc:`UC02 </design/UseCases/02_uc>`, :doc:`UC36 </design/UseCases/36_uc>`, :doc:`UC37 </design/UseCases/37_uc>`

id (Types.Identifier) -- The identifer of the resource for which access is being checked. May be either a PID or a SID, the latter returning results as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

The identifer of the resource for which access is being checked. May be either a PID or a SID, the latter returning results as if called with the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

action (Types.Permission) -- The type of operation which is being requested for the given identifier. Transmitted as a URL query parameter, and so must be escaped accordingly.

The type of operation which is being requested for the given identifier. Transmitted as a URL query parameter, and so must be escaped accordingly.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=1820)This error is raised if the request comes from a black listed source (e.g. a temporary block may be imposed on a source that calls this method too many times within some time interval)

This error is raised if the request comes from a black listed source (e.g. a temporary block may be imposed on a source that calls this method too many times within some time interval)

setAccessPolicy(session,id,accessPolicy,serialVersion) -> boolean

internal padding after source/apis/generated/generated_CNAuthorization.txt

Unsuccessful completion of this operation MUST be indicated by returning an appropriate exception such as :exc:`NotAuthorized`.

Unsuccessful completion of this operation MUST be indicated by returning an appropriate exception such as 

id (Types.Identifier) -- The object for which access control is being updated. May be either a PID or a SID, the latter acting on the HEAD PID only. Transmitted as part of the URL path and must be escaped accordingly.

The object for which access control is being updated. May be either a PID or a SID, the latter acting on the HEAD PID only. Transmitted as part of the URL path and must be escaped accordingly.

accessPolicy (Types.AccessPolicy) -- The desired privileges to be assigned to the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

The desired privileges to be assigned to the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

Exceptions.InvalidToken -- (errorCode=401, detailCode=4410)The supplied token is invalid

Exceptions.NotFound -- (errorCode=404, detailCode=4400)The specified object does not exist in the DataONE system

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4420)The Subject does not have permission to alter access control rules for the object.

The :term:`Subject` does not have permission to alter access control rules for the object.

 does not have permission to alter access control rules for the object.

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4403)The serialVersion supplied with the request does not match the serialVersion of the target

Returns a list of all object formats registered in the DataONE Object Format Vocabulary.

v2.0: The structure of :class:`v2_0.Types.ObjectFormat` has changed.

The list of object formats registered in the DataONE Object Format Vocabulary

Exceptions.NotImplemented -- (errorCode=501, detailCode=4840)The service is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4841)An error occurred when attempting to service the request.

echoIndexedObject(session,queryEngine,sysmeta,object) -> OctetStream

(:class:`session<Types.Session>`) ``->`` :class:`Types.SubjectInfo`

(:class:`session<Types.Session>`, :class:`sysmeta<Types.SystemMetadata>`) ``->`` :class:`Types.SystemMetadata`

(:class:`session<Types.Session>`, ``queryEngine``, :class:`sysmeta<Types.SystemMetadata>`, ``object``) ``->`` :class:`Types.OctetStream`

Echo the credentials used to make the call. This method can be used to verify the client certificate is valid and contains the expected information.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate. Transmitted as part of the SSL handshake process.

The subjects and groups parsed from the supplied session information.

Exceptions.NotImplemented -- (errorCode=501, detailCode=4965)The service is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4966)An internal failure prevented a successful response.

Exceptions.InvalidToken -- (errorCode=401, detailCode=4967)The supplied session information could not be parsed.

On successful parsing, a copy of the system metadata is returned, otherwise an exception is returned indicating an error condition.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the x.509 certificate. The service MAY choose to limit access to only authorized users. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the x.509 certificate. The service MAY choose to limit access to only authorized users. Transmitted as part of the SSL handshake process.

sysmeta (Types.SystemMetadata) -- A SystemMetadata object to be examined. The object is parsed and error conditions reported by an exception response. On successful parsing, the SystemMetadata object is echoed back with a HTTP 200 status. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

A SystemMetadata object to be examined. The object is parsed and error conditions reported by an exception response. On successful parsing, the SystemMetadata object is echoed back with a HTTP 200 status. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=4975)

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=4976)

internal padding after source/apis/generated/generated_CNDiagnostic.txt

Parse and echo the provided science metadata or resource map document. The response is governed by the type of object provided in the request, and on success is one or more documents that are the result of parsing for indexing.

Since DataONE supports multiple types of query engine, the query engine to be used for parsing is specified in the request.

The servce may terminate the POST operation if the size of the object is beyond a reasonable size.

queryEngine (string) -- A valid query engine name as reported by listQueryEngines() Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

sysmeta (Types.SystemMetadata) -- A SystemMetadata object that passes the echoSystemMetadata diagnostic. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

A SystemMetadata object that passes the echoSystemMetadata diagnostic. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

object (bytes) -- A document (e.g. science metadata or resource map) that is to be evalauted for indexing.

A document (e.g. science metadata or resource map) that is to be evalauted for indexing.

A document representing the parsed object as it would be prior to being added to a search index. For the solr query engine for example, this would be the equivalent of a <add><doc> .. </doc> .. </add> structure with possibly multiple documents.

A document representing the parsed object as it would be prior to being added to a search index. For the solr query engine for example, this would be the equivalent of a 

Exceptions.NotImplemented -- (errorCode=501, detailCode=4980)The service is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4981)An internal failure prevented a successful response.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4982)The supplied credentials are not authorized for this operation.

Exceptions.InvalidToken -- (errorCode=401, detailCode=4983)The supplied session information could not be parsed.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4984)The structure of the request is invalid.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=4985)The system metadata could not be parsed.

Exceptions.UnsupportedType -- (errorCode=400, detailCode=4986)The supplied object was not of a supported type.

Exceptions.UnsupportedMetadataType -- (errorCode=400, detailCode=4987)The provided metadata format is not supported by the query engine.

The provided metadata format is not supported by the query engine.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=4988)Insufficient resources could be allocated to support the request. The provided object may be too large to process.

Insufficient resources could be allocated to support the request. The provided object may be too large to process.

Used internally within a Coordinating Node to add a new object to the object store.

v2.0: The structure of :class:`v2_0.Types.SystemMetadata` has changed from Version 1.

:doc:`UC04 </design/UseCases/04_uc>`, :doc:`UC09 </design/UseCases/09_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

pid (Types.Identifier) -- The identifier that should be used in DataONE to identify and access the object. This is an Unicode string that follows the constraints on identifiers described in /design/PIDs. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The identifier that should be used in DataONE to identify and access the object. This is an Unicode string that follows the constraints on identifiers described in 

sysmeta (Types.SystemMetadata) -- The complete system metadata document describing the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

The complete system metadata document describing the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

The identifier that was used to insert the document into the system. This should be the same as the identifier provided as the pid parameter.

The identifier that was used to insert the document into the system. This should be the same as the identifier provided as the 

Exceptions.NotAuthorized -- (errorCode=401, detailCode=1100)The provided identity does not have permission to WRITE to the Member Node.

The provided identity does not have permission to WRITE to the Member Node.

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=1120)The requested identifier is already used by another object and therefore can not be used for this object. Clients should choose a new identifier that is unique and retry the operation or use CNCore.reserveIdentifier() to reserve one.

The requested identifier is already used by another object and therefore can not be used for this object. Clients should choose a new identifier that is unique and retry the operation or use :func:`CNCore.reserveIdentifier` to reserve one.

The requested identifier is already used by another object and therefore can not be used for this object. Clients should choose a new identifier that is unique and retry the operation or use 

Exceptions.UnsupportedType -- (errorCode=400, detailCode=4895)The object store is unable to store the provided content.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=4897)The CN object store is unable to execute the transfer because of resource limitations.

The CN object store is unable to execute the transfer because of resource limitations.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=4896)The supplied system metadata is invalid. This could be because some required field is not set, the metadata document is malformed, or the value of some field is not valid.

The supplied system metadata is invalid. This could be because some required field is not set, the metadata document is malformed, or the value of some field is not valid.

Given a scheme and optional fragment, generates an identifier with that scheme and fragment that is unique. Returned identifier may be used as either a PID or a SID.

scheme (string) -- The name of the identifier scheme to be used, drawn from a DataONE-specific vocabulary of identifier scheme names, including several common syntaxes such as DOI, ARK, LSID, UUID, and LSRN, among others. The first version of this method only supports the UUID scheme, and ignores the fragment parameter.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The name of the identifier scheme to be used, drawn from a DataONE-specific vocabulary of identifier scheme names, including several common syntaxes such as DOI, ARK, LSID, UUID, and LSRN, among others. The first version of this method only supports the UUID scheme, and ignores the fragment parameter.  Transmitted as a UTF-8 String as a 

fragment (string) -- The optional fragment to include in the generated Identifier. This parameter is optional and may not be present in the message body. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The optional fragment to include in the generated Identifier. This parameter is optional and may not be present in the message body. Transmitted as a UTF-8 String as a 

Exceptions.InvalidToken -- (errorCode=401, detailCode=4190)The supplied authentication token is not a proper certificate, or missing required fields, or otherwise proves invalid.

The supplied authentication token is not a proper certificate, or missing required fields, or otherwise proves invalid.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4180)Supplied credentials does not have WRITE permission

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4200)The schme requested is not a valid schme accepted by this service

The schme requested is not a valid schme accepted by this service

Returns a document describing the capabilities of the Coordinating Node.

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, ``object``, :class:`sysmeta<Types.SystemMetadata>`) ``->`` :class:`Types.Identifier`

(:class:`session<Types.Session>`, :class:`[fromDate]<Types.DateTime>`, :class:`[toDate]<Types.DateTime>`, :class:`[event]<Types.Event, string>`, ``[idFilter]``, ``[start]``, ``[count]``) ``->`` :class:`Types.Log`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.Identifier`

(:class:`session<Types.Session>`, ``scheme``, ``[fragment]``) ``->`` :class:`Types.Identifier`

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`obsoletedByPid<Types.Identifier>`, ``serialVersion``) ``->`` boolean

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`sysmeta<Types.SystemMetadata>`) ``->`` :class:`Types.Identifier`

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, :class:`sysmeta<Types.SystemMetadata>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`subject<Types.Subject>`, :class:`id<Types.Identifier>`) ``->`` boolean

Provides a mechanism for updating system metadata for any objects held in the federation.

Usage of this method SHOULD be restricted to CNs for updating the system metadata in the underlying CN storage sub-system.

v2.0: The structure of :class:`v2_0.Types.SystemMetadata` has changed.

Note: the serial version and the replica list in the new system metadata will be ignored.

sysmeta (Types.SystemMetadata) -- Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=4956)

Returns the object format registered in the DataONE Object Format Vocabulary for the given format identifier.

formatId (Types.ObjectFormatIdentifier) -- Unique ObjectFormatIdentifier for the object format Transmitted as part of the URL path and must be escaped accordingly.

Unique ObjectFormatIdentifier for the object format Transmitted as part of the URL path and must be escaped accordingly.

The object format registered in the DataONE Object Format Vocablulary

Exceptions.NotImplemented -- (errorCode=501, detailCode=4845)The service is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4846)Unexpected exception from the service.

Exceptions.NotFound -- (errorCode=404, detailCode=4848)The format specified by formatId does not exist at this node.

Retrieves consolidated log information for the specified date range (fromDate < timestamp <= toDate) for the entire DataONE infrastructure

Note that full access to log records requires access through a priviledged account. A public user may be presented with an empty response.

v2.0: The event parameter has changed from :class:`v1_0.Types.Event` to a plain *string*

fromDate (Types.DateTime) -- Starting time for records in response, entries with timestamp greater than or equal to (>=) this value will be returned. Defaults to include all records. Transmitted as a URL query parameter, and so must be escaped accordingly.

Starting time for records in response, entries with timestamp greater than or equal to (>=) this value will be returned. Defaults to include all records. Transmitted as a URL query parameter, and so must be escaped accordingly.

toDate (Types.DateTime) -- End time for records in response, entries with timestamp less than (<) this value will be returned. If not specified, then defaults to now. Transmitted as a URL query parameter, and so must be escaped accordingly.

End time for records in response, entries with timestamp less than (<) this value will be returned. If not specified, then defaults to now. Transmitted as a URL query parameter, and so must be escaped accordingly.

event (Types.Event, string) -- Return only log records for the specified type of event.  Default is all. Transmitted as a URL query parameter, and so must be escaped accordingly.

Return only log records for the specified type of event.  Default is all. Transmitted as a URL query parameter, and so must be escaped accordingly.

idFilter (string) -- Return only log records for identifiers that start with the supplied identifier string. Support for this parameter is optional and MAY be ignored by the Coordinating Node implementation with no warning. Supports PID and SID values. Only PID values will be included in the returned entries. Transmitted as a URL query parameter, and so must be escaped accordingly.

Return only log records for identifiers that start with the supplied identifier string. Support for this parameter is optional and MAY be ignored by the Coordinating Node implementation with no warning. Supports PID and SID values. Only PID values will be included in the returned entries. Transmitted as a URL query parameter, and so must be escaped accordingly.

start (integer) -- The zero based index of the first log record to return. Default is 0. Transmitted as a URL query parameter, and so must be escaped accordingly.

The zero based index of the first log record to return. Default is 0. Transmitted as a URL query parameter, and so must be escaped accordingly.

count (integer) -- The maximum number of log records that should be returned in the response. The Member Node may return fewer and the caller should check the total in the response to determine if further pages may be retrieved. Transmitted as a URL query parameter, and so must be escaped accordingly.

The maximum number of log records that should be returned in the response. The Member Node may return fewer and the caller should check the 

Exceptions.InsufficientResources -- (errorCode=413, detailCode=1481)The request could not be serviced due to a limitation of resources - too many requests, internal service timeout, or another similar failure.

The request could not be serviced due to a limitation of resources - too many requests, internal service timeout, or another similar failure.

session, [fromDate], [toDate], [event], [idFilter], [start], [count]

Deletes an object from the entire DataONE system, including all nodes known to hold a copy of the object. The PID and/or SID of the object will continue to be shown as in use (preventing its reuse for other objects), however the object should not be resolvable (NotFound) or retrievable.

The delete operation is used only by administrators in response to a request to remove an object from DataONE, perhaps because of legal requirements or the object has been identified as containing malicious content.

id (Types.Identifier) -- The identifier of the object to be deleted. May be either a PID or SID, the latter will operate on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to be deleted. May be either a PID or SID, the latter will operate on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=4961)The delete operation failed because the object does not exist.

internal padding after source/apis/generated/generated_CNCore.txt

Checks to determine if the supplied *subject* is the owner of the reservation of *id*.

A positive response (that the *pid* is reserved and owned by *subject*) is indicated by a return of a HTTP status of 200.

A negative response is indicated by an exception and the associated HTTP status code.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to an authority trusted by the DataONE Coordinating Nodes.  Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to an authority trusted by the DataONE Coordinating Nodes.  Transmitted as part of the SSL handshake process.

subject (Types.Subject) -- The subject of the principal (user) that made the reservation.  Transmitted as a URL query parameter, and so must be escaped accordingly.

 (user) that made the reservation.  Transmitted as a URL query parameter, and so must be escaped accordingly.

id (Types.Identifier) -- The identifier that is being checked for existing as a reserved identifier or is in use as an identifier for an existing object. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier that is being checked for existing as a reserved identifier or is in use as an identifier for an existing object. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

True - subject has the reservation on the PID; False - the PID does not exist; False - the PID is already in use; False - the PID is reserved by somone else.

Exceptions.NotImplemented -- (errorCode=501, detailCode=4920)The method functionality is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4921)An internal server error occurred.

Exceptions.InvalidToken -- (errorCode=401, detailCode=4922)The session information is invalid.

Exceptions.NotFound -- (errorCode=404, detailCode=4923)The PID does not exist as a reservation or an existing object and is not in use as an identifier.

The PID does not exist as a reservation or an existing object and is not in use as an identifier.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4924)The PID is reserved but the owner is not the principal identified by the subjectInfo OR the provide session does not have authority to access this service.

The PID is reserved but the owner is not the :term:`principal` identified by the *subjectInfo* OR the provide *session* does not have authority to access this service.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4925)The request was malformed and could not be processed

Low level "are you alive" operation. A valid ping response is indicated by a HTTP status of 200. A timestmap indicating the current system time (UTC) on the node MUST be returned in the HTTP Date header.

The Member Node should perform some minimal internal functionality testing before answering. However, ping checks will be frequent (every few minutes) so the internal functionality test should not be high impact.

Any status response other than 200 indicates that the node is offline for DataONE operations.

Note that the timestamp returned in the Date header should follow the semantics as described in the HTTP specifications, http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18

Note that the timestamp returned in the Date header should follow the semantics as described in the HTTP specifications, 

The response body will be ignored by the caller expect in the case of an error, in which case the response body should contain the appropriate DataONE exception.

Null body or Exception. The body of the message is ignored by the caller. The HTTP header Date MUST be set in the response.

Null body or Exception. The body of the message is ignored by the caller. The HTTP header 

Exceptions.InsufficientResources -- (errorCode=413, detailCode=2045)A ping response may return InsufficientResources if for example the system is in a state where normal DataONE operations may be impeded by an unusually high load on the node.

A ping response may return InsufficientResources if for example the system is in a state where normal DataONE operations may be impeded by an unusually high load on the node.

Returns a list of checksum algorithms that are supported by DataONE.

Exceptions.NotImplemented -- (errorCode=501, detailCode=4880)The service is not implemented.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4881)A problem occurred with the service that prevented it from returning the expected response.

A problem occurred with the service that prevented it from returning the expected response.

setObsoletedBy(session,pid,obsoletedByPid,serialVersion) -> boolean

Updates the :attr:`Types.SystemMetadata.obsoletedBy` property for an object, indicating that the object specified by *pid* has been obsoleted by the identifier in *obsoletedByPid*.

 property for an object, indicating that the object specified by 

v2.0: Method implementation has changed to ensure that the obsolescence chain is consistent with use of any SID assigned to the object.

session (Types.Session) -- Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The Subject of the session MUST have write permission on the object. Transmitted as part of the SSL handshake process.

Session information that contains the identity of the calling user as retrieved from the X.509 certificate which must be traceable to the CILogon service. The 

 permission on the object. Transmitted as part of the SSL handshake process.

pid (Types.Identifier) -- Identifier of the object system metadata being updated. Transmitted as part of the URL path and must be escaped accordingly.

Identifier of the object system metadata being updated. Transmitted as part of the URL path and must be escaped accordingly.

obsoletedByPid (Types.Identifier) -- Identifier of the object that obsoletes the object identified by pid.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

Identifier of the object that obsoletes the object identified by 

serialVersion (unsigned long) -- The serial version of the system metadata being updated. If the specified serialVersion does not match the current version at the Coordinating Nodes, then a Exceptions.VersionMismatch error is raised and no changes are made. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The serial version of the system metadata being updated. If the specified 

 does not match the current version at the Coordinating Nodes, then a 

 error is raised and no changes are made. Transmitted as a UTF-8 String as a 

Exceptions.NotImplemented -- (errorCode=501, detailCode=4940)The service endpoint has not yet been fully implemented

Exceptions.ServiceFailure -- (errorCode=500, detailCode=4941)A problem occurred with the service that prevented it from returning the expected response.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4942)The request was malformed and could not be processed.

Exceptions.InvalidToken -- (errorCode=401, detailCode=4943)The supplied session information could not be verified as a valid DataONE session.

The supplied session information could not be verified as a valid DataONE session.

Exceptions.NotFound -- (errorCode=404, detailCode=4944)The specified pid does not exist.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=4945)The credentials provided with the request in the session do not have write privileges on pid.

The credentials provided with the request in the *session* do not have *write* privileges on *pid*.

Exceptions.VersionMismatch -- (errorCode=409, detailCode=4946)The provided serialVersion does not match the latest version that is held by the CN. The client should refresh it's copy, verify that the update is still necessary, and resubmit the request with the updated information.

The provided *serialVersion* does not match the latest version that is held by the CN. The client should refresh it's copy, verify that the update is still necessary, and resubmit the request with the updated information.

 does not match the latest version that is held by the CN. The client should refresh it's copy, verify that the update is still necessary, and resubmit the request with the updated information.

Provides a mechanism for adding system metadata independently of its associated object, such as when adding system metadata for data objects.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=4864)

Returns a list of nodes that have been registered with the DataONE infrastructure.

Hides an object managed by DataONE from search operations, effectively preventing its discovery during normal operations.

The operation does not delete the object bytes, but instead sets the :attr:`Types.SystemMetadata.archived` flag to True. This ensures that the object can still be resolved (and hence remain valid for existing citations and cross references), though will not appear in searches.

The operation does not delete the object bytes, but instead sets the 

 flag to True. This ensures that the object can still be resolved (and hence remain valid for existing citations and cross references), though will not appear in searches.

Objects that are archived can not be updated through the :func:`MNStorage.update` operation.

Archived objects can not be un-archived. This behavior may change in future versions of the DataONE API.

The CN should ensure that all MNs holding a copy of the object are informed of the change so that they may update their information about the object.

id (Types.Identifier) -- The identifier of the object to be archived. May be either a PID or a SID, the latter will act on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to be archived. May be either a PID or a SID, the latter will act on the HEAD PID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=4971)The archive operation failed because the object does not exist.

Reserves the identifier that is unique and can not be used by any other sessions. Future calls to :func:`MNStorage.create` and :func:`MNStorage.update` that reference this ID must be made by the same :term:`principal` making the reservation, otherwise an error is raised on those methods.

Reserves the identifier that is unique and can not be used by any other sessions. Future calls to 

 making the reservation, otherwise an error is raised on those methods.

The requested identifier is transmitted in a MIME Multipart/form-data body with *id* as key, and the identifier string as value.

The requested identifier is transmitted in a MIME Multipart/form-data body with 

v2.0: The identifier being reserved may be used as a :term:`PID` or :term:`SID`.

id (Types.Identifier) -- The identifier that is to be reserved. May be a PID or a SID value. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The identifier that is to be reserved. May be a PID or a SID value. Transmitted as a UTF-8 String as a 

Exceptions.InvalidRequest -- (errorCode=400, detailCode=4200)The identifier requested is not a valid format accepted by this service

The identifier requested is not a valid format accepted by this service

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=4210)The requested identifier can not be reserved because it already exists in the DataONE system or has already been reserved.

The requested identifier can not be reserved because it already exists in the DataONE system or has already been reserved.

The service interfaces described here are exposed through the Coordinating
Node REST interface to support interactions with Member Nodes and DataONE
clients.

The following table provides a list of API methods exposed by Coordinating
Nodes.

Version of API method is available. The lowest version number indicates when
the method was added. A version number in parentheses indicates the method is
available in that version and is unchanged from the previous version. If more
than one version number is present, then the method signature or functionality
has changed between API versions. e.g. "1.0, 2.0" indicates that the method
was first introduced in Version 1.0 and has been modified in Version 2.0.

The HTTP method and path relative to the Base URL. Parameters specified in the
URL are indicatd by braces. Note that parameters included in a path MUST be
properly path encoded, and parameters included as key, value pairs MUST also
be properly encoded.

Indicates the parameters used when calling the method (sent in the message
payload) and the return type.

The response should be a valid HTTP response with a blank or arbitrary body.
Only the HTTP header information is considered by the requestor. A successful
response MUST have a HTTP status code of 200. In case of an error condition,
the appropriate HTTP status code MUST be set, and an exception or error
information MAY be returned in the response body.

Example of ping request and response for a Member Node (Coordinating Nodes
implement the same functionality). Lines prefixed with ">" indicate outgoing
information, lines prefixed with "<" show content returned from the server.
Lines associated with SSL connection initiation and close are not shown here.
Note that the actual response headers may vary, the only required header
fields are the first status line and a ``Date`` entry. However, in order to
fully support clients that may cache the response, it is recommended that the
``Expires``, and ``Cache-Control`` headers are returned.

fields are the first status line and a 

 entry. However, in order to

export NODE="https://demo2.test.dataone.org/knb/d1/mn"
curl -k -v "$NODE/v1/monitor/ping"

> GET /knb/d1/mn/v1/monitor/ping HTTP/1.1
> User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6
  OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> Host: demo2.test.dataone.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 06 Mar 2012 14:19:59 GMT
< Server: Apache/2.2.14 (Ubuntu)
< Content-Length: 0
< Content-Type: text/plain
<

curl "http://cn.dataone.org/cn/v1/formats"

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="/cn/xslt/dataone.types.v1.xsl"?>
<d1:objectFormatList xmlns:d1="http://ns.dataone.org/service/types/v1" count="67" start="0" total="67">
  <objectFormat>
    <formatId>-//ecoinformatics.org//eml-access-2.0.0beta4//EN</formatId>
    <formatName>Ecological Metadata Language, Access module, version 2.0.0beta4</formatName>
    <formatType>METADATA</formatType>
  </objectFormat>
    <formatId>-//ecoinformatics.org//eml-attribute-2.0.0beta4//EN</formatId>
    <formatName>Ecological Metadata Language, Attribute module, version 2.0.0beta4</formatName>
    <formatId>-//ecoinformatics.org//eml-constraint-2.0.0beta4//EN</formatId>
    <formatName>Ecological Metadata Language, Constraint module, version 2.0.0beta4</formatName>
  ...
</d1:objectFormatList>

Retrieve information about the ``http://www.openarchives.org/ore/terms``
formatId. Note that formatId has characters that should be escaped when added
as a URL path element. This is done using the urlencode script.

formatid=$(echo "http://www.openarchives.org/ore/terms" | urlencode)
curl -s "http://cn-dev.dataone.org/cn/v1/formats/${formatid}" | xml fo

<d1:objectFormat xmlns:d1="http://ns.dataone.org/service/types/v1">
  <formatId>http://www.openarchives.org/ore/terms</formatId>
  <formatName>Object Reuse and Exchange Vocabulary</formatName>
  <formatType>RESOURCE</formatType>
</d1:objectFormat>

Example of retrieving 3 log records from a Member Node. The *xml* command is
provided by xmlstarlet_ and is used to format the output.

The *MN_core* API provides mechanisms for a Member Node to report on the level
of service compliance and to specify replication policies. The capabilities
information is used in the Member Node registration process by the
Coordinating Nodes.

 API provides mechanisms for a Member Node to report on the level

The state of health API provides mechanisms for the monitoring infrastructure
to report on the current state of the DataONE infrastructure and for the
Coordinating Nodes to track the current operating state of the Member Node.

(:class:`session<Types.Session>`, :class:`[fromDate]<Types.DateTime>`, :class:`[toDate]<Types.DateTime>`, :class:`[event]<Types.Event, string>`, ``[idFilter]``, ``[start=0]``, ``[count=1000]``) ``->`` :class:`Types.Log`

The response body will be ignored by the caller except in the case of an error, in which case the response body should contain the appropriate DataONE exception.

Null body or Exception. The body of the message may be ignored by the caller. The HTTP header Date MUST be set in the response.

Null body or Exception. The body of the message may be ignored by the caller. The HTTP header 

Exceptions.NotImplemented -- (errorCode=501, detailCode=2041)Ping is a required operation and so an operational member node should never return this exception unless under development.

Ping is a required operation and so an operational member node should never return this exception unless under development.

Exceptions.ServiceFailure -- (errorCode=500, detailCode=2042)A ServiceFailure exception indicates that the node is not currently operational as a member node. A coordinating node or monitoring service may use this as an indication that the member node should be taken out of the pool of active nodes, though ping should be called on a regular basis to determine when the node might b ready to resume normal operations.

A ServiceFailure exception indicates that the node is not currently operational as a member node. A coordinating node or monitoring service may use this as an indication that the member node should be taken out of the pool of active nodes, though ping should be called on a regular basis to determine when the node might b ready to resume normal operations.

session, [fromDate], [toDate], [event], [idFilter], [start=0], [count=1000]

Retrieve log information from the Member Node for the specified slice parameters. Log entries will only return PIDs.

This method is used primarily by the log aggregator to generate aggregate statistics for nodes, objects, and the methods of access.

The response MUST contain only records for which the requestor has permission to read.

Note that date time precision is limited to one millisecond. If no timezone information is provided UTC will be assumed.

Access control for this method MUST be configured to allow calling by Coordinating Nodes and MAY be configured to allow more general access.

fromDate (Types.DateTime) -- Records with time stamp greater than or equal to (>=) this value will be returned. Transmitted as a URL query parameter, and so must be escaped accordingly.

Records with time stamp greater than or equal to (>=) this value will be returned. Transmitted as a URL query parameter, and so must be escaped accordingly.

toDate (Types.DateTime) -- Records with a time stamp less than (<) this value will be returned. If not specified, then defaults to now. Transmitted as a URL query parameter, and so must be escaped accordingly.

Records with a time stamp less than (<) this value will be returned. If not specified, then defaults to 

. Transmitted as a URL query parameter, and so must be escaped accordingly.

Return only log records for the specified type of event.  Default is 

idFilter (string) -- Return only log records for identifiers that start with the supplied identifier string. Support for this parameter is optional and MAY be ignored by the Member Node implementation with no warning. Accepts PIDs and SIDs Transmitted as a URL query parameter, and so must be escaped accordingly.

Return only log records for identifiers that start with the supplied identifier string. Support for this parameter is optional and MAY be ignored by the Member Node implementation with no warning. Accepts PIDs and SIDs Transmitted as a URL query parameter, and so must be escaped accordingly.

start=0 (integer) -- Optional zero based offset from the first record in the set of matching log records. Used to assist with paging the response. Transmitted as a URL query parameter, and so must be escaped accordingly.

Optional zero based offset from the first record in the set of matching log records. Used to assist with paging the response. Transmitted as a URL query parameter, and so must be escaped accordingly.

count=1000 (integer) -- The maximum number of log records that should be returned in the response. The Member Node may return fewer and the caller should check the total in the response to determine if further pages may be retrieved. Transmitted as a URL query parameter, and so must be escaped accordingly.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=1460)Raised if the user making the request is not authorized to access the log records. This is determined by the policy of the Member Node.

Raised if the user making the request is not authorized to access the log records. This is determined by the policy of the Member Node.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=1480)The request parameters were malformed or an invalid date range was specified.

The request parameters were malformed or an invalid date range was specified.

Returns a document describing the capabilities of the Member Node.

The response at the Member Node base URL is for convenience only. Clients of Member Nodes SHOULD use the /node URL to retrieve the node capabilities document.

internal padding after source/apis/examples/mn_getCapabilities.txt

curl -k -s "$NODE/v1/node" | xml fo

<d1:node xmlns:d1="http://ns.dataone.org/service/types/v1" replicate="true" synchronize="true" type="mn" state="up">
  <identifier>urn:node:DEMO2</identifier>
  <name>DEMO2 Metacat Node</name>
  <description>A DataONE member node implemented in Metacat.</description>
  <baseURL>https://demo2.test.dataone.org:443/knb/d1/mn</baseURL>
  <services>
    <service name="MNRead" version="v1" available="true"/>
    <service name="MNCore" version="v1" available="true"/>
    <service name="MNAuthorization" version="v1" available="true"/>
    <service name="MNStorage" version="v1" available="true"/>
    <service name="MNReplication" version="v1" available="true"/>
  </services>
  <synchronization>
    <schedule hour="*" mday="*" min="0/3" mon="*" sec="10" wday="?" year="*"/>
    <lastHarvested>2012-03-06T14:57:39.851+00:00</lastHarvested>
    <lastCompleteHarvest>2012-03-06T14:57:39.851+00:00</lastCompleteHarvest>
  </synchronization>
  <ping success="true"/>
  <subject>CN=urn:node:DEMO2, DC=dataone, DC=org</subject>
  <contactSubject>CN=METACAT1, DC=dataone, DC=org</contactSubject>
</d1:node>

curl -k -s "$NODE/v1/log?start=0&count=3" | xml fo

<d1:log xmlns:d1="http://ns.dataone.org/service/types/v1" count="3" start="0" total="1273">
  <logEntry>
    <entryId>1</entryId>
    <identifier>MNodeTierTests.201260152556757.</identifier>
    <ipAddress>129.24.0.17</ipAddress>
    <userAgent>null</userAgent>
    <subject>CN=testSubmitter,DC=dataone,DC=org</subject>
    <event>create</event>
    <dateLogged>2012-02-29T23:25:58.104+00:00</dateLogged>
    <nodeIdentifier>urn:node:DEMO2</nodeIdentifier>
  </logEntry>
    <entryId>2</entryId>
    <identifier>TierTesting:testObject:RightsHolder_Person.4</identifier>
    <dateLogged>2012-02-29T23:26:38.828+00:00</dateLogged>
    <entryId>3</entryId>
    <identifier>TierTesting:testObject:RightsHolder_Group.4</identifier>
    <dateLogged>2012-02-29T23:27:40.255+00:00</dateLogged>
</d1:log>

The system metadata included with the create call must contain values for the
elements required to be set by clients (see :doc:`/design/SystemMetadata`). The
system metadata document can be crafted by hand or preferably with a tool such
as *generate_sysmeta.py* which is available in the d1_instance_generator_ Python
package. See documentation included with that package for more information on
its operation.

elements required to be set by clients (see 

). The
as 

 Python

The outgoing request body *must* be encoded as `MIME multipart/form-data`_ with
the system metadata portion and the object as file attachments.

 with

The outgoing request body *must* be encoded as `MIME multipart/form-data`_
with the system metadata portion as a file attachment and the sourceNode
parameter as a form field.

The Replication API provides methods to support CN-directed replication of
content between MNs.

(:class:`session<Types.Session>`, :class:`sysmeta<Types.SystemMetadata>`, :class:`sourceNode<Types.NodeReference>`) ``->`` boolean

Called by a Coordinating Node to request that the Member Node create a copy of the specified object by retrieving it from another Member Nodeode and storing it locally so that it can be made accessible to the DataONE system.

A successful operation is indicated by a HTTP status of 200 on the response.

Access control for this method MUST be configured to allow calling by Coordinating Nodes.

sysmeta (Types.SystemMetadata) -- Copy of the CN held system metadata for the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

Copy of the CN held system metadata for the object. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

sourceNode (Types.NodeReference) -- A reference to node from which the content should be retrieved. The reference should be resolved by checking the CN node registry. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

A reference to node from which the content should be retrieved. The reference should be resolved by checking the CN node registry. Transmitted as a UTF-8 String as a 

Exceptions.InsufficientResources -- (errorCode=413, detailCode=2154)

response must have a HTTP status code of 200. In case of an error condition,
the appropriate HTTP status code must be set, and an exception or error
information *may* be returned in the response.

information 

curl -v -X POST "https://localhost:8000/mn/v1/replicate" \
  -H "Content-type: multipart/form-data" \
  -F "sysmeta=@systemmetadata.xml" \
  -F "sourceNode=urn:node:MN_B"

* About to connect() to localhost port 8000 (#0)
*   Trying ::1... Connection refused
*   Trying fe80::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> POST /mn/v1/replicate HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: localhost:8000
> Content-Length: 1021
> Expect: 100-continue
> Content-type: multipart/form-data; boundary=----------------------------88ffdd8070e9
>
* Done waiting for 100-continue
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Fri, 14 Jan 2011 22:01:13 GMT
< Server: WSGIServer/0.1 Python/2.6.1
< Content-Type: text/xml
<
<
* Closing connection #0

.. _MIME multipart/form-data: http://www.ietf.org/rfc/rfc2388.txt

The *MNRead* API implements methods that enable object management
operations on a :term:`Member Node`.

 API implements methods that enable object management
operations on a 

Retrieve an object identified by *id* from the node. Supports both PIDs and SIDs. SID will return HEAD PID.

 from the node. Supports both PIDs and SIDs. SID will return HEAD PID.

The response MUST contain the bytes of the indicated object, and the checksum of the bytes retrieved SHOULD match the :attr:`SystemMetadata.checksum` recorded in the  :class:`Types.SystemMetadata` when calling with PID.

The response MUST contain the bytes of the indicated object, and the checksum of the bytes retrieved SHOULD match the 

:doc:`UC01 </design/UseCases/01_uc>`, :doc:`UC06 </design/UseCases/06_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

id (Types.Identifier) -- The identifier for the object to be retrieved. May be a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier for the object to be retrieved. May be a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=1000)The provided identity does not have READ permission on the object.

The provided identity does not have READ permission on the object.

Exceptions.NotFound -- (errorCode=404, detailCode=1020)The object specified by id does not exist at this node. The description should include a reference to the resolve method.

The object specified by *id* does not exist at this node. The description should include a reference to the resolve method.

 does not exist at this node. The description should include a reference to the resolve method.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=1002)The node is unable to service the request due to insufficient resources such as CPU, memory, or bandwidth being over utilized.

The node is unable to service the request due to insufficient resources such as CPU, memory, or bandwidth being over utilized.

curl -k "$NODE/v1/object/XYZ332"

... data ...

(GET) Attempt to retrieve a non-existent object (and show headers in response)::

(GET) Attempt to retrieve a non-existent object (and show headers in response):

curl -D - "$NODE/v1/object/DOESNTEXIST"

Date: Tue, 06 Mar 2012 15:25:35 GMT
Server: Apache/2.2.14 (Ubuntu)
Content-Length: 196
Vary: Accept-Encoding

<error detailCode="1800" errorCode="404" name="NotFound">
   <description>No system metadata could be found for given PID: DOESNTEXIST</description>
</error>

Describes the object identified by *id* by returning the associated system metadata object.

If the object does not exist on the node servicing the request, then :exc:`Exceptions.NotFound` MUST be raised even if the object exists on another node in the DataONE system.

 MUST be raised even if the object exists on another node in the DataONE system.

:doc:`UC06 </design/UseCases/06_uc>`, :doc:`UC37 </design/UseCases/37_uc>`, :doc:`UC16 </design/UseCases/16_uc>`

id (Types.Identifier) -- Identifier for the science data or science metedata object of interest. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

Identifier for the science data or science metedata object of interest. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=1060)There is no data or science metadata identified by the given id on the node where the request was serviced. The error message should provide a hint to use the CNRead.resolve() mechanism.

There is no data or science metadata identified by the given *id* on the node where the request was serviced. The error message should provide a hint to use the :func:`CNRead.resolve` mechanism.

 on the node where the request was serviced. The error message should provide a hint to use the 

internal padding after source/apis/examples/getSystemMetadata.txt

(GET) Retrieve system metadata from a Member Node for object "XYZ332" which
happens to be science metadata (an EML document) that has been obsoleted by
a new version with identifier "XYZ33"::

a new version with identifier "XYZ33":

curl http://m1.dataone.org/mn/v1/meta/XYZ332

<d1:systemMetadata xmlns:d1="http://ns.dataone.org/service/types/v1">
  <serialVersion>1</serialVersion>
  <identifier>XYZ332</identifier>
  <formatId>eml://ecoinformatics.org/eml-2.1.0</formatId>
  <size>20875</size>
  <checksum algorithm="MD5">e7451c1775461b13987d7539319ee41f</checksum>
  <submitter>uid=mbauer,o=NCEAS,dc=ecoinformatics,dc=org</submitter>
  <rightsHolder>uid=mbauer,o=NCEAS,dc=ecoinformatics,dc=org</rightsHolder>
  <accessPolicy>
    <allow>
      <subject>uid=jdoe,o=NCEAS,dc=ecoinformatics,dc=org</subject>
      <permission>read</permission>
      <permission>write</permission>
      <permission>changePermission</permission>
    </allow>
    <allow>
      <subject>public</subject>
    </allow>
    <allow>
      <subject>uid=nceasadmin,o=NCEAS,dc=ecoinformatics,dc=org</subject>
    </allow>
  </accessPolicy>
  <replicationPolicy replicationAllowed="false"/>
  <obsoletes>XYZ331</obsoletes>
  <obsoletedBy>XYZ333</obsoletedBy>
  <archived>true</archived>
  <dateUploaded>2008-04-01T23:00:00.000+00:00</dateUploaded>
  <dateSysMetadataModified>2012-06-26T03:51:25.058+00:00</dateSysMetadataModified>
  <originMemberNode>urn:node:TEST</originMemberNode>
  <authoritativeMemberNode>urn:node:TEST</authoritativeMemberNode>
</d1:systemMetadata>

(GET) Attempt to retrieve system metadata for an object that does not exist.::

(GET) Attempt to retrieve system metadata for an object that does not exist.:

curl http://cn.dataone.org/cn/v1/meta/SomeObjectID

  <description>No system metadata could be found for given PID: SomeObjectID</description>
</error>

This method provides a lighter weight mechanism than :func:`MNRead.getSystemMetadata` for a client to determine basic properties of the referenced object. The response should indicate properties that are typically returned in a HTTP HEAD request: the date late modified, the size of the object, the type of the object (the :attr:`SystemMetadata.formatId`).

id (Types.Identifier) -- Identifier for the object in question. May be either a PID or a SID.  Transmitted as part of the URL path and must be escaped accordingly.

Identifier for the object in question. May be either a PID or a SID.  Transmitted as part of the URL path and must be escaped accordingly.

Returns a :class:`Types.Checksum` for the specified object using an accepted hashing algorithm. The result is used to determine if two instances referenced by a PID are identical, hence it is necessary that MNs can ensure that the returned checksum is valid for the referenced object either by computing it on the fly or by using a cached value that is certain to be correct.

 for the specified object using an accepted hashing algorithm. The result is used to determine if two instances referenced by a PID are identical, hence it is necessary that MNs can ensure that the returned checksum is valid for the referenced object either by computing it on the fly or by using a cached value that is certain to be correct.

pid (Types.Identifier) -- The identifier of the object the operation is being performed on. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object the operation is being performed on. Transmitted as part of the URL path and must be escaped accordingly.

checksumAlgorithm (string) -- The name of an algorithm that will be used to compute a checksum of the bytes of the object. This value is drawn from a DataONE controlled list of values as indicted in the Types.SystemMetadata. If not specified, then the system wide default checksum algorithm should be used. Transmitted as a URL query parameter, and so must be escaped accordingly.

The name of an algorithm that will be used to compute a checksum of the bytes of the object. This value is drawn from a DataONE controlled list of values as indicted in the 

. If not specified, then the system wide default checksum algorithm should be used. Transmitted as a URL query parameter, and so must be escaped accordingly.

The checksum value originally computed for the specified object.

Exceptions.InvalidRequest -- (errorCode=400, detailCode=1402)A supplied parameter was invalid, most likely an unsupported checksum algorithm was specified, in which case the error message should include an enumeration of supported checksum algorithms.

A supplied parameter was invalid, most likely an unsupported checksum algorithm was specified, in which case the error message should include an enumeration of supported checksum algorithms.

session, [fromDate], [toDate], [formatId], [identifier], [replicaStatus], [start=0], [count=1000]

Retrieve the list of objects present on the MN that match the calling parameters. This method is required to support the process of :term:`Member Node synchronization`. At a minimum, this method MUST be able to return a list of objects that match::

Retrieve the list of objects present on the MN that match the calling parameters. This method is required to support the process of 

. At a minimum, this method MUST be able to return a list of objects that match:

formatId (Types.ObjectFormatIdentifier) -- Restrict results to the specified object format identifier. Transmitted as a URL query parameter, and so must be escaped accordingly.

Restrict results to the specified object format identifier. Transmitted as a URL query parameter, and so must be escaped accordingly.

identifier (Types.Identifier) -- Restrict results to the specified identifier. May be a PID or a SID. In the case of the latter, returns a listing of all PIDs that share the given SID. Transmitted as a URL query parameter, and so must be escaped accordingly.

Restrict results to the specified identifier. May be a PID or a SID. In the case of the latter, returns a listing of all PIDs that share the given SID. Transmitted as a URL query parameter, and so must be escaped accordingly.

replicaStatus (boolean) -- Indicates if replicated objects should be returned in the list (i.e. any entries present in the SystemMetadata.replica, objects that have been replicated to this node). If false, then no objects that have been replicated should be returned. If true, then any objects can be returned, regardless of replication status. The default value is true. Transmitted as a URL query parameter, and so must be escaped accordingly.

Indicates if replicated objects should be returned in the list (i.e. any entries present in the 

, then no objects that have been replicated should be returned. If 

, then any objects can be returned, regardless of replication status. The default value is 

Retrieve an object list from a member node, and pipe the response through an
xml formatter for easier viewing::

xml formatter for easier viewing:

curl "https://gmn-dev.test.dataone.org/mn/v1/object?count=5" | xml fo

<?xml version="1.0"?>
<ns1:objectList xmlns:ns1="http://ns.dataone.org/service/types/v1" count="5" start="0" total="12">
  <objectInfo>
    <identifier>AnserMatrix.htm</identifier>
    <formatId>eml://ecoinformatics.org/eml-2.0.0</formatId>
    <checksum algorithm="MD5">0e25cf59d7bd4d57154cc83e0aa32b34</checksum>
    <dateSysMetadataModified>1970-05-27T06:12:49</dateSysMetadataModified>
    <size>11048</size>
  </objectInfo>

  ...

    <identifier>hdl:10255/dryad.218/mets.xml</identifier>
    <checksum algorithm="MD5">65c4e0a9c4ccf37c1e3ecaaa2541e9d5</checksum>
    <dateSysMetadataModified>1987-01-14T07:09:09</dateSysMetadataModified>
    <size>2796</size>
</ns1:objectList>

This is a callback method used by a CN to indicate to a MN that it cannot complete synchronization of the science metadata identified by *pid*. When called, the MN should take steps to record the problem description and notify an administrator or the data owner of the issue.

This is a callback method used by a CN to indicate to a MN that it cannot complete synchronization of the science metadata identified by 

. When called, the MN should take steps to record the problem description and notify an administrator or the data owner of the issue.

A successful response is indicated by a HTTP status of 200. An unsuccessful call is indicated by a returned exception and associated HTTP status code.

message (Types.Exception) -- An instance of the Exceptions.SynchronizationFailed exception with body appropriately filled.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

 exception with body appropriately filled.  Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

A successful response is indicated by a HTTP 200 status. An unsuccessful call is indicated by returing the appropriate exception.

Exceptions.InvalidToken -- (errorCode=401, detailCode=2164)Optionally raised by the receiving MN, depending on implementation.

Optionally raised by the receiving MN, depending on implementation.

Notifies the Member Node that the authoritative copy of system metadata on the Coordinating Nodes has changed.

The Member Node SHOULD schedule an update to its information about the affected object by retrieving an authoritative copy from a Coordinating Node.

id (Types.Identifier) -- Identifier of the object for which system metadata was changed. May be either a PID or a SID. Calling with SID is equivalent to calling with HEAD PID. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

Identifier of the object for which system metadata was changed. May be either a PID or a SID. Calling with SID is equivalent to calling with HEAD PID. Transmitted as a UTF-8 String as a 

serialVersion (unsigned long) -- The serialVersion of the system metadata. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The serialVersion of the system metadata. Transmitted as a UTF-8 String as a 

dateSysMetaLastModified (Types.DateTime) -- The time stamp for when the system metadata was changed. Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The time stamp for when the system metadata was changed. Transmitted as a UTF-8 String as a 

True if notification was received OK, otherwise an error is returned.

internal padding after source/apis/generated/generated_MNRead.txt

Called by a target Member Node to fullfill the replication request originated by a Coordinating Node calling :func:`MNReplication.replicate`. This is a request to make a replica copy of the object, and differs from a call to GET /object in that it should be logged as a replication event rather than a read event on that object.

Called by a target Member Node to fullfill the replication request originated by a Coordinating Node calling 

. This is a request to make a replica copy of the object, and differs from a call to GET /object in that it should be logged as a replication event rather than a read event on that object.

If the object being retrieved is restricted access, then a Tier 2 or higher Member Node MUST make a call to :func:`CNReplication.isNodeAuthorized` to verify that the Subject of the caller is authorized to retrieve the content.

If the object being retrieved is restricted access, then a Tier 2 or higher Member Node MUST make a call to 

 to verify that the Subject of the caller is authorized to retrieve the content.

pid (Types.Identifier) -- The identifier of the object to get as a replica Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to get as a replica Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=2184)The node is unable to service the request due to insufficient resources such as CPU, memory, or bandwidth being over utilized.

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, ``[checksumAlgorithm]``) ``->`` :class:`Types.Checksum`

(:class:`session<Types.Session>`, :class:`[fromDate]<Types.DateTime>`, :class:`[toDate]<Types.DateTime>`, :class:`[formatId]<Types.ObjectFormatIdentifier>`, :class:`[identifier]<Types.Identifier>`, ``[replicaStatus]``, ``[start=0]``, ``[count=1000]``) ``->`` :class:`Types.ObjectList`

(:class:`session<Types.Session>`, :class:`message<Types.Exception>`) ``->`` :class:`Types.Boolean`

(:class:`session<Types.Session>`, :class:`id<Types.Identifier>`, ``serialVersion``, :class:`dateSysMetaLastModified<Types.DateTime>`) ``->`` boolean

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`) ``->`` :class:`Types.OctetStream`

Provides mechanisms Member Nodes to verify access to resources for users
(:term:`subject`). See the document :doc:`/design/Authentication` for more
details on some authentication options.

(

internal padding after source/apis/generated/generated_MNAuthorization.txt

Test if the user identified by the provided session has authorization for operation on the specified object.

The body of the response is arbitrary and SHOULD be ignored by the caller.

Should perhaps add convenience methods for "canRead()" and "canWrite()" to verify that a user is able to read / write an object.

:doc:`UC01 </design/UseCases/01_uc>`, :doc:`UC37 </design/UseCases/37_uc>`

id (Types.Identifier) -- The identifer of the resource for which access is being checked. May be either a PID or a SID. Will use the HEAD PID when given a SID value. Transmitted as part of the URL path and must be escaped accordingly.

The identifer of the resource for which access is being checked. May be either a PID or a SID. Will use the HEAD PID when given a SID value. Transmitted as part of the URL path and must be escaped accordingly.

action (Types.Permission) -- The type of operation which is being requested for the given pid. Transmitted as a URL query parameter, and so must be escaped accordingly.

The type of operation which is being requested for the given pid. Transmitted as a URL query parameter, and so must be escaped accordingly.

This method is called by clients to update objects on Member Nodes.

Updates an existing object by creating a new object identified by *newPid* on the Member Node which explicitly obsoletes the object identified by *pid* through appropriate changes to the SystemMetadata of *pid* and *newPid*.

Updates an existing object by creating a new object identified by 

 on the Member Node which explicitly obsoletes the object identified by 

The Member Node sets :attr:`Types.SystemMetadata.obsoletedBy` on the object being obsoleted to the *pid* of the new object. It then updates :attr:`Types.SystemMetadata.dateSysMetadataModified` on both the new and old objects. The modified system metadata entries then become available in :func:`MNRead.listObjects`. This ensures that a Coordinating Node will pick up the changes when filtering on :attr:`Types.SystemMetadata.dateSysMetadataModified`.

 on both the new and old objects. The modified system metadata entries then become available in 

. This ensures that a Coordinating Node will pick up the changes when filtering on 

The update operation MUST fail with :exc:`Exceptions.InvalidRequest` on objects that have the :attr:`Types.SystemMetadata.archived` property set to true.

A new, unique :attr:`Types.SystemMetadata.seriesId` may be included when beginning a series, or a series may be extended if the newPid obsoletes the existing pid.

 may be included when beginning a series, or a series may be extended if the newPid obsoletes the existing pid.

pid (Types.Identifier) -- The identifier of the object that is being updated. If this identifier does not exist in the system, an error is raised and the operation does not cause any changes to the objects or their metadata.  Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object that is being updated. If this identifier does not exist in the system, an error is raised and the operation does not cause any changes to the objects or their metadata.  Transmitted as part of the URL path and must be escaped accordingly.

object (bytes) -- The bytes of the data or science metadata object that will be deprecating the exsting object.

The bytes of the data or science metadata object that will be deprecating the exsting object.

newPid (Types.Identifier) -- The identifier that will become the replacement identifier for the existing object after the update. This identifier must have been previously reserved.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

The identifier that will become the replacement identifier for the existing object after the update. This identifier must have been previously reserved.  Transmitted as a UTF-8 String as a 

sysmeta (Types.SystemMetadata) -- A System Metadata document describing the new object. The SystemMetadata.obsoletes field must contain the identifier of the object being obsoleted. Other required client provided fields as described for Types.SystemMetadata must be filled. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

 field must contain the identifier of the object being obsoleted. Other required client provided fields as described for 

 must be filled. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

The identifier of the document that is replacing the original, which should be the same as newPid.

The identifier of the document that is replacing the original, which should be the same as 

Exceptions.IdentifierNotUnique -- (errorCode=409, detailCode=1220)The requested identifier is already used by another object and therefore can not be used for this object.  Clients should choose a new identifier that is unique and retry the operation.

The requested identifier is already used by another object and therefore can not be used for this object.  Clients should choose a new identifier that is unique and retry the operation.

Exceptions.UnsupportedType -- (errorCode=400, detailCode=1240)The MN can not deal with the object provided.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=1260)The MN is unable to execute the transfer because it does not have sufficient storage space for example.

The MN is unable to execute the transfer because it does not have sufficient storage space for example.

Exceptions.NotFound -- (errorCode=404, detailCode=1280)The update operation failed because the object which was supposed to be updated in the system (indicated via the obsoletedPid parameter) is not present in the DataONE system, so update is an illegal operation.

The update operation failed because the object which was supposed to be updated in the system (indicated via the *obsoletedPid* parameter) is not present in the DataONE system, so update is an illegal operation.

The update operation failed because the object which was supposed to be updated in the system (indicated via the 

 parameter) is not present in the DataONE system, so update is an illegal operation.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=1300)One or more required fields are not set, the metadata document is malformed or the value of some field is not valid. SystemMetadata.obsoletes is set by the client and does not match the pid of the object being obsoleted. SystemMetadata.obsoletedBy is set on the SystemMetadata of the new object provided by the client (a new object cannot be created in an obsoleted state). SystemMetadata.obsoletedBy is already set on the object being obsoleted (no branching is allowed in the obsolescence chain).

One or more required fields are not set, the metadata document is malformed or the value of some field is not valid. :attr:`SystemMetadata.obsoletes` is set by the client and does not match the *pid* of the object being obsoleted. :attr:`SystemMetadata.obsoletedBy` is set on the SystemMetadata of the new object provided by the client (a new object cannot be created in an obsoleted state). :attr:`SystemMetadata.obsoletedBy` is already set on the object being obsoleted (no branching is allowed in the obsolescence chain).

One or more required fields are not set, the metadata document is malformed or the value of some field is not valid. 

 is set on the SystemMetadata of the new object provided by the client (a new object cannot be created in an obsoleted state). 

 is already set on the object being obsoleted (no branching is allowed in the obsolescence chain).

Exceptions.InvalidRequest -- (errorCode=400, detailCode=1202)Raised when the request parameters are incorrect or the operation is not applicable to the current state of the object (e.g. an archived object can not be updated)

Raised when the request parameters are incorrect or the operation is not applicable to the current state of the object (e.g. an archived object can not be updated)

The *MNPackage* API is an optional API that may be implemented by Member Nodes
that intend to support downloading all of the contents of a data package in a single API call. Without this service, a client application must individually retrieve each of the metadata and data components of a package as they are listed in the ORE document that describes the package.  Using the MNPackage service, a caller can instead request a serialized form of all of the data in a package, which is returned in the format requested.  All implementations must support the BagIt format specification, but may also support additional well-defined packaging standards and specifications.

 API is an optional API that may be implemented by Member Nodes

(:class:`session<Types.Session>`, :class:`packageType<Types.ObjectFormatIdentifier>`, :class:`id<Types.Identifier>`) ``->`` :class:`Types.OctetStream`

internal padding after source/apis/generated/generated_MNPackage.txt

Provides all of the content of a DataONE data package as defined by an OAI-ORE document in DataONE, in one of several possible package serialization formats.  The serialized package will contain all of the data described in the ORE aggregation. The default implementation will include packages in the BagIt format.  The packageType formats must be specified using the associated ObjectFormat formatId for that package serialization format.

The {id} parameter must be the identifier of an ORE package object. If it is the identifier of one of the science metadata documents or data files contained within the package,  the Member Node should throw an InvalidRequest exception. Identifiers may be either PIDss or SIDs.

packageType (Types.ObjectFormatIdentifier) -- Indicates which package format will be used to serialize the package. All implementations must support a default BagIt package serialization, but are free to implement additional package serialization formats. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which package format will be used to serialize the package. All implementations must support a default BagIt package serialization, but are free to implement additional package serialization formats. Transmitted as part of the URL path and must be escaped accordingly.

id (Types.Identifier) -- The identifier of the package or object in a package to be returned as a serialized package. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the package or object in a package to be returned as a serialized package. Transmitted as part of the URL path and must be escaped accordingly.

Any return type is allowed, including application/octet-stream, but the format of the response should be specialized by the requested packageType.

Exceptions.NotFound -- (errorCode=404, detailCode=2875)The specified pid does not exist.

The *MNQuery* API is an optional API that may be implemented by Member Nodes
that intend to support querying the local repository. The actual form of the
query is undefined, and t is expected that a small set of well known query
engine types will be supported.

The :func:`MNQuery.query` operation may be implemented by more than one type of search engine and the *queryEngine* parameter indicates which search engine is targeted. The value and form of *query* is determined by the specific query engine.

For example, the SOLR search engine will accept many of the standard parameters of SOLR, including field restrictions and faceting.

This method is optional for Member Nodes, but if implemented, both getQueryEngineDescription and listQueryEngines must also be implemented.

queryEngine (string) -- Indicates which search engine will be used to handle the query. Supported search engines can be determined through the MNQuery.listQueryEngines API call. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which search engine will be used to handle the query. Supported search engines can be determined through the MNQuery.listQueryEngines API call. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2825)The specified queryEngine does not exist

queryEngine (string) -- Indicates which query engine for which to provide descriptive metadata. Currently supported search engines can be determined through MNQuery.listQueryEngines. Transmitted as part of the URL path and must be escaped accordingly.

Indicates which query engine for which to provide descriptive metadata. Currently supported search engines can be determined through MNQuery.listQueryEngines. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2814)The specified queryEngine does not exist

internal padding after source/apis/generated/generated_MNQuery.txt

internal padding after source/apis/generated/generated_MNStorage.txt

Provides a mechanism for updating system metadata for any objects held on the Member Node where that Member Node is the authoritative Member Node. Coordinating Node can call this method on the non-authoritative Member Node. However, this is not a normal operation and is for the special case - the authoritative Member Node doesn't exist any more. Coordinating Node calling the method on the non-authoriative Memember Node in the normal operation can cause an unexpected consequence.

This method is typically used by Authoritative Member Node or rights holder[s] to ensure system metadata quality.

The *MNView* API is an optional API that may be implemented by Member Nodes
that intend to support providing rendered views of content on their repository. Each repository can implement multipe themed views of their content, each accesed using the name of the theme and the identifier of the content to be viewed.  Unlike the MNRead service, which returns the exact bytes of content, the MNView service provides a rendered view of the content which can transform the content into different formats.  The most common use of the view service will likely be to provide a rendered HTML landing page at a well-known URL that can be used to provide a human-readable view of metadata and data.  Other potential uses include providing alternative formats for metadata and data.  Each Member Node that implements the MNView service must implement at least one theme named 'default' which provides the default view of all content.  Other themes can be provided for use by various clients.

If this service is implemented, the :func:`MNView.view` operation must implement at least one {theme} named 'default' to provide a standard (possibly minimalistic) view of the content in HTML format.

 operation must implement at least one {theme} named 'default' to provide a standard (possibly minimalistic) view of the content in HTML format.

This method is optional for Member Nodes, but if implemented,  MNView.listViews must also be implemented.

id (Types.Identifier) -- The identifier of the object to render in a view. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to render in a view. May be either a PID or a SID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2835)The specified pid does not exist.

internal padding after source/apis/generated/generated_MNView.txt

This method is optional for Member Nodes, but if implemented,  MNView.view must also be implemented.

(:class:`session<Types.Session>`, :class:`pid<Types.Identifier>`, ``object``, :class:`newPid<Types.Identifier>`, :class:`sysmeta<Types.SystemMetadata>`) ``->`` :class:`Types.Identifier`

The *pid* must not exist in the DataONE system or should have been previously reserved using  :func:`CNCore.reserveIdentifier`. A new, unique :attr:`Types.SystemMetadata.seriesId` may be included.

 must not exist in the DataONE system or should have been previously reserved using  

The caller MUST have authorization to write or create content on the Member Node.

pid (Types.Identifier) -- The identifier that should be used in DataONE to identify and access the object. This is an Unicode string that follows the constraints on identifiers described in /design/PIDs. If the identifier is already in use, Exceptions.IdentifierNotUnique will be raised and the client SHOULD try again with a different, unique identifier.  Transmitted as a UTF-8 String as a Param part of the MIME multipart/mixed message.

 will be raised and the client SHOULD try again with a different, unique identifier.  Transmitted as a UTF-8 String as a 

object (bytes) -- The data bytes that are to be added to the Member Node.

sysmeta (Types.SystemMetadata) -- The system metadata document that provides basic information about the object, including a reference to its identifier, access control information, etc. Attributes of the sysmeta that are the responsibility of the client MUST be set. Note that the obsoletes and obsoletedBy elements MUST not be set. It is the role of the update() method to ensure these are properly updated to ensure object lineage is as expected. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a File part of the MIME multipart/mixed message.

The system metadata document that provides basic information about the object, including a reference to its identifier, access control information, etc. Attributes of the sysmeta that are the responsibility of the client MUST be set. Note that the obsoletes and obsoletedBy elements MUST not be set. It is the role of the update() method to ensure these are properly updated to ensure object lineage is as expected. Transmitted as an UTF-8 encoded XML structure for the respective type as defined in the DataONE types schema, as a 

The identifier that was used to insert the document into the system.

Exceptions.UnsupportedType -- (errorCode=400, detailCode=1140)The MN can not deal with the content specified in the data package.

The MN can not deal with the content specified in the data package.

Exceptions.InsufficientResources -- (errorCode=413, detailCode=1160)The MN is unable to execute the transfer because it does not have sufficient storage space for example.

Exceptions.InvalidSystemMetadata -- (errorCode=400, detailCode=1180)The supplied system metadata is invalid. This could be because some required field is not set, the metadata document is malformed, or the value of some field is not valid.

Given a scheme and optional fragment, generates an identifier with that scheme and fragment that is unique. Maybe be used for generating either PIDs or SIDs.

Exceptions.InvalidToken -- (errorCode=401, detailCode=2190)The supplied authentication token is not a proper certificate, or missing required fields, or otherwise proves invalid.

Exceptions.NotAuthorized -- (errorCode=401, detailCode=2192)Supplied credentials does not have WRITE permission

Exceptions.InvalidRequest -- (errorCode=400, detailCode=2193)The scheme requested is not a valid scheme accepted by this service

The scheme requested is not a valid scheme accepted by this service

Member Nodes MUST check that the caller is authorized to perform this function.

If the object does not exist on the node servicing the request, then an :exc:`Exceptions.NotFound` exception is raised. The message body of the exception SHOULD contain a hint as to the location of the :func:`CNRead.resolve` method.

If the object does not exist on the node servicing the request, then an 

 exception is raised. The message body of the exception SHOULD contain a hint as to the location of the 

id (Types.Identifier) -- The identifier of the object to be archived. May be either a PID or a SID. Will archive the HEAD PID when called with a SID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to be archived. May be either a PID or a SID. Will archive the HEAD PID when called with a SID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2911)The archive operation failed because the object is not present on the node servicing the request.

The archive operation failed because the object is not present on the node servicing the request.

Deletes an object managed by DataONE from the Member Node. Member Nodes MUST check that the caller (typically a Coordinating Node) is authorized to perform this function.

The delete operation will be used primarily by Coordinating Nodes to help manage the number of replicas of an object that are present in the entire system.

The operation removes the object from further interaction with DataONE services. The implementation may delete the object bytes, and in general should do so since a delete operation may be in response to a problem with the object (e.g. it contains malicious content, is innappropriate, or is the subject of a legal request).

id (Types.Identifier) -- The identifier of the object to be deleted. May be either a PID or a SID. Will delete the HEAD PID when called with a SID. Transmitted as part of the URL path and must be escaped accordingly.

The identifier of the object to be deleted. May be either a PID or a SID. Will delete the HEAD PID when called with a SID. Transmitted as part of the URL path and must be escaped accordingly.

Exceptions.NotFound -- (errorCode=404, detailCode=2901)The delete operation failed because the object is not present on the node servicing the request.

The delete operation failed because the object is not present on the node servicing the request.

The service interfaces described here are exposed through the Member
Node REST interface to support interactions with Coordinating Nodes and DataONE
clients.

The following table provides a list of API methods exposed by Member
Nodes.

@startuml images/class_mn.png
package MemberNode {
package Tier1 {
   class MNCore {
     Boolean : ping()
     Log : getLogRecords()

   }
   class MNRead {

   }
   class MNQuery {

   }
 }
 package Tier2 {
   class MNAuthorization{

   }
 }
 package Tier3 {
   class MNStorage {

   }
 }
 package Tier4 {
   class MNReplication {

   }
 }
}
@enduml

(POST) Create a new object with a given identifier (*XYZ33256*)::

curl -E /tmp/x509up_u502 \
     -F "pid=XYZ33256" \
     -F "object=@sciencemetadata.xml" \
     -F "sysmeta=@sysmeta.xml" \
     https://m1.dataone.org/mn/v1/object

HTTP/1.1 200 Success
Content-Type:
Date: Wed, 16 Dec 2009 13:58:34 GMT
Content-Length: 355

XYZ33256

For example, the system metadata document for the example above was generated
using the sequence of commands::

using the sequence of commands:

<<log on to cilogon.org and download my certificate>>

MYSUBJECT=`python my_subject.py /tmp/x509up_u502`
echo $MYSUBJECT

CN=Dave Vieglais T799,O=Google,C=US,DC=cilogon,DC=org

python generate_sysmeta.py -f sciencemetadata.xml \
                           -i "XYZ33256" \
                           -s "$MYSUBJECT" \
                           -t "eml://ecoinformatics.org/eml-2.0.1" \
                            > sysmeta.xml

The generated system metadata document contains default information that indicates:

The submitter is ``CN=Dave Vieglais T799,O=Google,C=US,DC=cilogon,DC=org``

The access policy indicates public read, and write by the submitter

The replication policy indicates replication is allowed to any node

<?xml version='1.0' encoding='UTF-8'?>
<ns1:systemMetadata xmlns:ns1="http://ns.dataone.org/service/types/v1">
  <identifier>XYZ33256</identifier>
  <formatId>eml://ecoinformatics.org/eml-2.0.1</formatId>
  <size>22936</size>
  <checksum algorithm="MD5">2ec0084d1e11e0d5c9a46ba6a230aa85</checksum>
  <submitter>CN=Dave Vieglais T799,O=Google,C=US,DC=cilogon,DC=org</submitter>
  <rightsHolder>CN=Dave Vieglais T799,O=Google,C=US,DC=cilogon,DC=org</rightsHolder>
    <allow>
    </allow>
    <allow>
      <subject>CN=Dave Vieglais T799,O=Google,C=US,DC=cilogon,DC=org</subject>
    </allow>
  <replicationPolicy replicationAllowed="true"/>
  <dateUploaded>2012-02-20T20:39:19.664495</dateUploaded>
  <dateSysMetadataModified>2012-02-20T20:39:19.70598</dateSysMetadataModified>
</ns1:systemMetadata>

Link these methods to the generated methods in the API specifications,
eliminate redundancy of the description text between the two locations.

This section needs to be updated with the latest revisions to the
AccessPolicy section. Also need to update / regenerate the example of access
policy.

policy.

Several existing authorization policy languages were evaluated for use in the
DataONE architecture.  Given the simplicity of authorization rules that DataONE
needs to express, these specifications were deemed overly complex and would
impose too signification of a cost on Member Node implementations.

Survey for additional policy languages to evaluate before deciding on a
custom specification for DataONE.

DataONE has designed a simple access policy language that can be embedded in
several contexts and can be used to express access rules.  The definitions of
the elements in this AccessPolicy language are:

A set of rules that specifies as a whole the allowable permissions that a
given user or system has for accessing a resource, including both data and
metadata resources and service resources. An access policy consists of a
sequence of allow rules that grant permissions to principals, which can be
individual users, groups of users, symbolic users, or systems and services.

A rule that is used to allow a principal to perform an action (such as
read or write) on an object in DataONE. Rules are three-tuples (principal,
permission, resource) specifying which permissions are allowed for the
principal(s) for the resource(s) listed. Access control rules are specified
by the OriginMemberNode when the object is first registered in DataONE. If no
rules are specified at that time, then the object is deemed to be private and
the only user with access to the object (read, write, or otherwise) is the
RightsHolder.

The 'deny' directive has been removed for simplicity, and because a survey
of existing member nodes indicates it is not being used by the community.

The unique identifier representing a principal that is allowed or denied
access to a resource.  Principal identifiers are strings that are found
transported in the subject field of an identifying certificate produced from
the authentication system.  Users, groups, systems, and services can all be
represented as principals.

A string value indicating the set of actions that can be performed on a
resource as specified in an access policy.  The set of permissions include
the ability to read a resource, modify a resource (write), and to change the
set of access control policies for a resource (changePermission).  In
addition, there is a permission that controls ability to execute a service
(execute).

The unique identifier (pid) for a resource in the system to which the access
rules in this access policy apply.

Include file u'/var/lib/jenkins/jobs/API_Documentation_trunk/workspace/api-documentation/source/d1_schemas/accesspolicy-example.xml' not found or reading it failed

Users and services authenticate in DataONE to confirm their identity. The
identity is then used for controlling access to objects, systems, and services
within the DataONE framework. Requirements for Authorization are listed here:

Privacy and access control in DataONE are primarily for the protection and
integrity of user contributed data and metadata via Member Nodes. There are,
however, other entities in DataONE that also need protection, including DataONE
specific services and system resources, like system metadata and components of
the general software stack (e.g., databases, web servers) for Coordinating and
Member Nodes. For this reason, all resources in DataONE, from data and metadata
objects to system services, have an access policy (:class:`Types.AccessPolicy`),
made up of one or more *access control rules* (:class:`Types.AccessRule`),
that is used to determine who may access the resource. The process of confirming
whether a user has privileges to access a resource in DataONE is called
*authorization*. The act of authorization uses attribute information contained
in the security token obtained by the user when authenticating with their
identity provider, and compares such information to the resource access control
rule. If the rule permits access by the :term:`principal` requesting the
resource, then authorization succeeds and permission is granted to access the
resource. The algorithm used to evaluate authorization for a resource is
described in the section *Object Access Control* below.

objects to system services, have an access policy (

),

. The act of authorization uses attribute information contained
rule. If the rule permits access by the 

 requesting the
described in the section 

Because nodes that form the DataONE federation are managed by various
administrative domains and may cross multiple political boundaries, "trust"
relationships are crucial for DataONE to succeed in its security plan. In simple
terms, this means that access control rules that are defined by one member of
the federation are upheld be another member. It also means that trust may be
revoked if a particular member does not behave accordingly within the
federation. Access control rules may be dynamic and must be propagated with
the resource they are designated to protect, such as when data or metadata
objects are replicated to another Member Node.

The language that specifies the policy for a given access control rule
dictates only whether a user is allowed access to a given resource; to include
the ability to explicitly deny access to a resource overly complicates
management of the authorization process and is seldom used in practice. Access
rules (:class:`Types.AccessRule`) consist of the system identity of the user,
also known as the :term:`Subject`, the type of permission granted (e.g.,
*read*, *write*, or *changePermission*), and the :term:`identifier` of the
resource being requested.

rules (

) consist of the system identity of the user,
also known as the 

An access policy is an optional element of the :term:`System Metadata`
associated with an object. The default access policy is to deny access to the
object to all users except the *subject* identified as the
:attr:`Types.SystemMetadata.rightsHolder` in the System Metadata.

object to all users except the 

DataONE will provide, where reasonable, a conversion of the internal access
control rule to a subset of one or more industry standard policy languages to
support interoperability between different organizations.

DataONE services are accessed through HTTPS connections. Restrictions on
agents (i.e. clients) that may call the services may be imposed through
network configuration (e.g. restricting IP addresses that may call the
service) or preferably through the *restriction* property of the
:class:`Types.Service` entry in the *services* property of the
:class:`Types.Node` entry describing the registered Member or Coordinating
Node.

service) or preferably through the 

The optional *restriction* property of the *service* lists subjects that have
permission to invoke the service. If a *restriction* is not included with the
service description, then any agent may call that service endpoint.

 lists subjects that have
permission to invoke the service. If a 

 is not included with the

NOTE: It is at the discretion of individual Node implementations as to whether these
defined service restrictions will be enforced for the method in question. The service method
restriction is meant only as a mechanism for node operators to record/manage
restrictions to be enforced in a transparent manner.

Adjustments to access control for services, or more accurately, the methods
exposed within a service, are made by altering the contents of the
*restriction* property of the :class:`Types.Service` entry for the
:class:`Types.Node` registration document for the node. These adjustments are
made through the :func:`CNRegistration.updateNodeCapabilities` method by
specifying a replacement node document. A current version of the node document
should be retrieved from the Coordinating Node through the
:func:`CNCore.listNodes` method.

 registration document for the node. These adjustments are
made through the 

 method by

Changes to node registration information can only be performed by subjects
listed in the *subject* property of the :class:`Types.Node` document for the
node.

listed in the 

Any authorization system in a federation requires trust among participants.
For DataONE, there are five types of trust relationships among nodes in the
federation:

**MN to CN**: Member Nodes need to have trust that Coordinating Nodes will
respect and enforce their authorization policies, including any restrictions
placed on where and when to create replicas of objects, and on the
presentation of search results for restricted content.

: Member Nodes need to have trust that Coordinating Nodes will

**CN to MN**: Coordinating Nodes rely upon Member Nodes for limited services,
and mainly expect Member Nodes to accurately implement the DataONE Service
API, including replication services.

: Coordinating Nodes rely upon Member Nodes for limited services,

**CN to CN**: Each Coordinating Node contains a replica of the content of the
others, and are configured to provide seamless failover and load-balancing
for all incoming requests across the three nodes. Consequently, the
Coordinating Nodes inherently trust one another fully. As the suite of
Coordinating Node instances expands to other continents, this relationship
may need to be re-examined.

: Each Coordinating Node contains a replica of the content of the

**MN to MN**: Member Node to Member Node trust relies on one Member Node
believing that another Member Node will respect the authorization policies
that they publish for their objects and services. In the case of restricted
access content, Member Nodes that house replicas of an object would need to
faithfully enforce authorization policies that were expressed by the data
owner. Because of this, Member Nodes can express replication policies for
objects that indicate which other Member Nodes are acceptable targets for
replication, and for which nodes they are willing to serve as replica stores.

: Member Node to Member Node trust relies on one Member Node

**User to DataONE**: Users trust that the DataONE system, that is, the
combination of Member and Coordinating Nodes interacting to provide the
DataONE infrastructure and services, implements access control rules
consistently and in compliance with the specifications provided when content
was added to the system or subsequently modified. This implies minimal
latency in propagation of rules between components of the system.

: Users trust that the DataONE system, that is, the

Verification of proper technical implementation of these trust relationships is
achieved through integration testing of the various components. This involves
exercising a wide array of combinations of users, groups, and access control
rules to ensure expected behavior as content moves around the DataONE
infrastructure.

The DataNet projects have a loosely defined requirement of interoperability
between their respective implementations. This also implies that content and
services *may* be shared between projects, and thus there will likely be
additional trust relationships that need to be taken into consideration as the
DataNet projects progress towards interoperability.

services 

 be shared between projects, and thus there will likely be

Need to update this section to cover the additional constraints beyond
subject authorization that will limit movement of content between components

Some nodes may also want to conditionally provide access to some services based
on a principal's current usage of a resource such as node storage or node
bandwidth.

Create/Update constraints

 * MaximumStorageQuota
 * MaximumNetworkTransferQuota

Embargoes

 * Add ability to specify an embargo period during which the access policies
   would not be in effect, and rather resources are only privately accessible

Add ability to specify an embargo period during which the access policies
would not be in effect, and rather resources are only privately accessible

Add constraints and embargoes to the AccessPolicy language described below

Located At CN or MN?

 * At CN requires global knowledge of ACLs
 * At CN requires a lot of network traffic for authorization on objects
 * At MN makes authorization of search results impossible

 * Compromise: Federated, each authoritative MN for an object keeps its ACL
   list, which gets synchronized to the CN at sync time

   * Assume most object write is at MN level, so best to not have to go to
     CN
   * Assume MN will want to control their own write access
   * Requires MN Authorization services

At CN requires a lot of network traffic for authorization on objects

Compromise: Federated, each authoritative MN for an object keeps its ACL
list, which gets synchronized to the CN at sync time

* Assume most object write is at MN level, so best to not have to go to
  CN
* Assume MN will want to control their own write access
* Requires MN Authorization services

Assume most object write is at MN level, so best to not have to go to
CN

Efficiency of search results authorization

 * Need to authorize large number of search results in each operation
 * Has implications for search results cacheing

Need to authorize large number of search results in each operation

Access control for content managed by DataONE (:term:`Data` objects,
:term:`Science Metadata` objects, and :term:`Resource Maps`) is determined by
the :class:`Types.AccessPolicy` entry in the :class:`Types.SystemMetadata`
associated with the object.

In addition, the :term:`rightsHolder` of the System Metadata holds all
permissions on the object, and the :term:`Authoritative Member Node` has
equivalent privileges as the *rightsHolder*.

 of the System Metadata holds all
permissions on the object, and the 

The *Authoritative Member Node* is identified by one or more :term:`Subjects`
listed in the Member Node :class:`Types.Node` record registered in the DataONE
:term:`node registry`. Thus, the :class:`Types.NodeReference` entry recorded
in the System Metadata *Authoritative Member Node* references the *Node* entry
in the node registry, which in turn contains a list of *Subjects* that, when
used in a request to access or manipulate an object, identify the user as the
*Authoritative Member Node*.

 that, when

The ability to change the content of this object via update services.
Permissions are hierarchical, so *write* permission also includes *read*
permission.

Permissions are hierarchical, so 

The ability to change the authorization policies for this object.
Includes both *read* and *write* permissions.

Includes both 

Conceptually, an :class:`Types.AccessRule` is a tuple with three components:
an *identifier* which indicates which object the rule applies to; a *subject*
which indicates who the rule applies to; and a *permission* which indicates
the level of access described by the rule.

In practice, the *access rule* is contained in the System Metadata, and so
each access rule contains a permission and list of subjects. A set of *access
rules* are contained in the :class:`Types.AccessPolicy`, and these together
with the *rights holder* and *authoritative member node* determine which
subjects may perform operations on an object.

 is contained in the System Metadata, and so
each access rule contains a permission and list of subjects. A set of 

Evaluation of a permission for an object is determined in a manner thus, where
SUBJECT is the *subject* making the request, and PERMISSION is the
*permission* being evaluated::

SUBJECT is the 

Is SUBJECT == rightsHolder?  Yes -> return True
Is SUBJECT IN authoritiveMemberNode.Subject?  Yes -> return True
for each accessRule in accessPolicy
  if PERMISSION is IN accessRule.Permission
    Is SUBJECT IN accessRule.Subject? Yes -> return True
return False

DataONE supports *equivalent identities*, where a single principal may have
multiple subjects associated with them. As such, the ``SUBJECT`` in algorithm
described above is actually a list of 1 or more subjects. The list of subjects
to be used for comparison is determined from the *Session* parameter of an API
call as follows::

, where a single principal may have
multiple subjects associated with them. As such, the 

 in algorithm
to be used for comparison is determined from the 

Adjustments to access control for objects is made by altering the
*accessPolicy* of the :class:`Types.SystemMetadata` for the object. The
process is to retrieve a current copy of the system metadata from a
Coordinating Node using the :func:`CNRead.getSystemMetadata` method, edit the
:class:`Types.AccessPolicy` entry as necessary, then send the updated
*AccessPolicy* structure to a Coordinating Node using the
:func:`CNAuthorization.setAccessPolicy` method.

 for the object. The
Coordinating Node using the 

Changes to *accessPolicy* are then propagated to other Coordinating Nodes
through the Coordinating Node replication process (and hence to the search
index), then to the Member Nodes that hold a copy of the object. Member Nodes
are informed of a change to *accessPolicy* through the
:func:`MNStorage.systemMetadataChanged` method which is called by a
Coordinating Node. Member Nodes are expected to update the *accessPolicy* for
an object as soon as possible after being informed of an update.

 are then propagated to other Coordinating Nodes
are informed of a change to 

 method which is called by a
Coordinating Node. Member Nodes are expected to update the 

 for

Need to update authorization use cases and include references to them.

Implementation phases
---------------------

During the first DataONE Federated Security workshop, four phases for
development were identified that involve increasingly sophisticated
authorization and access control mechanisms. The four phases are:

- **Phase 1: Mostly public access (target date: January 2011)**: Only publicly
  readable content is replicated. Only publicly readable content is indexed
  for search and retrieval. Access to restricted content is through origin
  member node only. No authentication is required to search and retrieve
  public content. Authentication is required to upload (create) content.

- **Phase 2: Access control supported for search and retrieval**: ACLs
  respected by coordinating nodes. Authenticated users can discover content
  that is restricted to them or their groups. Restricted access content is not
  replicated.

- **Phase 3: Access control supported for content replication**: Restricted
  access content is replicated to member nodes with compatible ACLs and
  pre-arranged trust agreements.

- **Phase 4: Consistent semantic and functional interoperability for identity
  and security**: Restricted access content is replicated to any member node.
  Authentication by long-running workflows is supported.

*Figure 3.* Trust relationships between components during phase 2 of
Authn/Authz. Triangle = CN, Rectangle = MN, open circle = public data, filled
circle = private data, dashed line = untrusted connection, solid line =
trusted connection, user with hat = authenticated user. Coordinating Nodes
synchronize public and private content (A). Authenticated users can retrieve
private data from the origin Member Node (B) and can discover and retrieve
metadata from the Coordinating Nodes (C). Public content is replicated between
Member Nodes (D) under the direction of a trusted connection from the
Coordinating Nodes (E).

 Trust relationships between components during phase 2 of

*Figure 4.* Trust relationships between components during phase 3 of
trusted connection, user with hat = authenticated user. Member Nodes of
compatible technology (D) have a trust relationship that enables transfer of
protected content from one member node to another (A). An authenticated user
has the same access to private content replicated to other Member Nodes (B).
Member Nodes with incompatible technology (i.e. unable to create a trusted
relationship) are only able to replicate public content (C). Coordinating
Nodes must have trusted relationships to all Member Nodes (E) to direct
replication.

 Trust relationships between components during phase 3 of

@startuml images/authorization_seq.png
actor User
participant MN1
participant MN2
participant CN
User -> CN: login(D1.username, password)
activate CN
CN --> MN1: token
deactivate CN
User -> MN1: create(token, pid, object, sysmeta)
activate MN1
MN1 -> MN1: verify(token)
MN1 -> MN1: isAuthorized(token, pid, OP_CREATE)
MN1 --> User: pid
deactivate MN1
@enduml

*Figure 1.* Only public objects are searchable and replicated in the system.
Create, Read, Update, and Delete operations are controlled by member nodes
for private objects, but read for public resources can be handled by any
replicating member node, or a coordinating node in the case of metadata
resources.

 Only public objects are searchable and replicated in the system.

*Figure 2.* Trust relationships between components during phase 1 of
Authz/Authn. Triangle = CN, Rectangle = MN, open circle = public data, filled
circle = private data, dashed line = untrusted connection. A Coordinating Node
retrieves only public content from a Member Node (A), and only publicly
readable content is available to users through the Coordinating Nodes (B) and
Member Nodes (C). A Coordinating Node must have a trusted relationship with
Member Nodes to request replication operations (E) even though the content
being replicated is publicly readable and does not require a trusted
connection (D).

 Trust relationships between components during phase 1 of

Access to log records is evaluated in the same manner as access to objects. If
the requesting *subject* does not have *read* permission for the *identifier*
recorded in the log record, then they will be denied access to the log entry.

the requesting 

Adjustments to access control for log records are made indirectly by adjusting
access control for the referenced object(s).

EDIT: Log records are now completely restricted to administrative users so as
not to expose raw usage patterns for any/all public objects.

This document outlines the mechanism for specifying authorization policies for
objects and service in DataONE and a set of services for controlling access to
those objects on Member Nodes and Coordinating Nodes.

Update this section to include the latest revisions to the methods defined
for managing and working with the access control for objects.

In this section, define a set of Authorization services to be implemented at
CN and MN. The current Authorization Service is defined as a standalone
service.

Determine if the user authenticated by the token can take the action
specified (read, write, changePermission, execute) on the resource named by
the identifier pid.

Set the access policy for a series of resources as specified by the
provided AccessPolicy document.  The user identified by the authentication
token must have changePermission permission on all resources named in the
AccessPolicy.  If so, then the policies for those resources will be
replaced (or created as needed) by the policies specified in AccessPolicy.
If the user does not have sufficient permission, then the NotAuthorized
exception must be thrown, and none of the policies should be applied (it is
not sufficient to have appropriate permissions on just one resource -- if
permission is not present for all listed resources, then implementations
must roll back any changes and return NotAuthorized.

guidance on RESOURCE_MAPS - initial thoughts: depends on references to DATA
objects, whether they be SIDs or PIDs

reassignment of authoritativemembernode field for unhosted versions

In a perfect world, all obsolescence chains will have be complete,
bi-directional links, and so determining the latest version of an object is
determined simply by examining the set of all objects with the same SID, and
selecting the object that is not ``obsoletedBy`` anything else. Obsolescence
chains may be incomplete for various reasons and in such situations, resolution
of series identifiers should still operate consistently.

selecting the object that is not 

 anything else. Obsolescence

The following series of scenarios demonstrate the behavior of the DataONE
system when resolving a seriesId to a specific object. The behavior of
resolution is to rely primarily on the obsoletes and obsoletedBy entities,
falling back to the date when an object is added to a Member Node
(:attr:`~Types.SystemMetadata.dateUploaded`) to determine the newer version.

(

The value of :attr:`~Types.SystemMetadata.dateUploaded` for
an object

an object

An object with
:attr:`~Types.SystemMetadata.identifier` (PID) |Pi|, a
:attr:`~v2_0.Types.SystemMetadata.seriesId` (SID)
of |Sj|, and a :attr:`~Types.SystemMetadata.dateUploaded`
of |tk|.

of |tk|.

|Pi| has an :attr:`~Types.SystemMetadata.obsoletedBy`
entry that contains the value |Pj|

|Pj| has an :attr:`~Types.SystemMetadata.obsoletes`
entry that contains the value |Pi|

|Pi| has an
:attr:`~Types.SystemMetadata.obsoletedBy` entry that
contains the value |Pj| and |Pj| has an
:attr:`~Types.SystemMetadata.obsoletes` entry that
contains the value |Pi|.

|Pi| has an

Neither :attr:`~Types.SystemMetadata.obsoletedBy`
nor :attr:`~Types.SystemMetadata.obsoletes` is set by
|Pi|  or |Pj|.

P_1\binom{S_1}{t_1} & \leftrightarrows P_2\binom{S_1}{t_2} \\
t_1 & < t_2 \\
resolve(S_1) & \Rrightarrow P_2

A set of objects :math:`O = \lbrace P_1, P_2 \rbrace` have the same series
identifier, |S1|.

All objects in :math:`O` are participants in an obsolescence chain since |P2|
:attr:`~Types.SystemMetadata.obsoletes` |P1| and |P1| is
:attr:`~Types.SystemMetadata.obsoletedBy` |P2|.

All elements of the obsolescence chain :math:`P_1 \leftrightarrows P_2` have the
same series identifier, |S1|.

The :attr:`~Types.SystemMetadata.dateUploaded` of |P1| is older than that of
|P2|.

This is a perfect obsolescence chain and resolving |S1| will result in the
object identified by |P2|.

P_1\binom{S_1}{t_1}\; & \square \; P_2\binom{S_1}{t_2} \\

No obsolescence assertions are made, so resolution is inferred by the most
recent value of :attr:`~Types.SystemMetadata.dateUploaded`.

recent value of 

P_1\binom{S_1}{t_1}\; & \leftarrow \; P_2\binom{S_1}{t_2} \\

:attr:`~Types.SystemMetadata.obsoletes` |P1| even though |P1| does not assert
it is :attr:`~Types.SystemMetadata.obsoletedBy` |P2|.

All elements of the obsolescence chain :math:`P_1 \leftarrow P_2` have the

This is a damaged, but consistent obsolescence chain and resolving |S1| will
result in the object identified by |P2|.

This is a damaged, but consistent obsolescence chain and resolving 

P_1\binom{S_1}{t_1} \leftrightarrows
P_2\binom{S_1}{t_2}& \leftrightarrows
P_3\binom{S_2}{t_3}\\
t_1 < t_2 & < t_3 \\
resolve(S_1) &\Rrightarrow P_2 \\
resolve(S_2) &\Rrightarrow P_3 \\

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2 \rbrace` have the same series

A set of objects :math:`O_{S_2} = \lbrace P_3 \rbrace` has the series
identifier, |S2|.

Objects :math:`O = O_{S_1} \cup O_{S_2}` all participate in a full,
bi-directional obsolescence chain.

In this case resolving |S1| will result in |P2| which is not the most recent
object in the obsolescence chain, however it is the newest version in the
obsolescence chain identified by |S1|.

 which is not the most recent
obsolescence chain identified by 

P_1\binom{S_1}{t_1} \leftarrow
P_2\binom{S_1}{t_2}& \leftarrow

Objects :math:`O = O_{S_1} \cup O_{S_2}` all participate in a damaged, though
consistent obsolescence chain.

 all participate in a damaged, though

P_3\binom{}{t_3}\\

Objects :math:`O = O_{S_1} \cup P_3` all participate in an obsolescence chain.

P_3\binom{}{t_3} \leftrightarrows
P_4\binom{S_2}{t_4} \\
t_1 < t_2 & < t_3 < t_4\\
resolve(S_2) &\Rrightarrow P_4

A set of objects :math:`O_{S_2} = \lbrace P_4 \rbrace` has the series

Objects :math:`O = O_{S_1} \cup P_3 \cup O_{S_2}` all participate in an
obsolescence chain.

P_2\binom{S_1}{t_2}& \rightarrow
?? \leftarrow
P_4\binom{S_1}{t_4} \\
t_1 < t_2 & < t_4\\
resolve(S_1) &\Rrightarrow P_4 \\

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2, P_4 \rbrace` have the same series

Objects :math:`O_{S_1}` all participate in an obsolescence chain, however the
chain is broken with no way to traverse between |P2| and |P4| because the
object that |P2| indicates it is ``obsoletedBy``, and the object that |P4|
indicates it ``obsoletes`` is not recorded by the DataONE Coordinating Nodes
(does not resolve).

 all participate in an obsolescence chain, however the
chain is broken with no way to traverse between 

 is not recorded by the DataONE Coordinating Nodes

In this case resolving |S1| will result in |P4| since that is the most recent
object in the set of objects :math:`O_{S_1}`.

P_2\binom{S_1}{t_2}& \;\square\;

chain is broken with no way to traverse between |P2| and |P4| because the the
object that |P4| indicates it ``obsoletes`` is not recorded by the DataONE
Coordinating Nodes (does not resolve).

 is not recorded by the DataONE

P_{del}\binom{}{} \leftarrow

The object :math:`P_{del}` was deleted from the system, so the identifier is
known, but the object and associated system metadata are no longer available.

 was deleted from the system, so the identifier is

archived\biggl[P_3\binom{S_1}{t_3}\biggr] \\
resolve(S_1) &\Rrightarrow P_3 \\

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2, P_3 \rbrace` have the same series

Objects :math:`O_{S_1}` all participate in an obsolescence chain.

In this case resolving |S1| will result in |P3| which is the most recent
object in the obsolescence chain even though it is archived.

 which is the most recent

P_1\binom{S_1}{t_1} & \leftrightarrows
P_2\binom{S_1}{t_2} \rightarrow
?? \\

Objects :math:`O_{S_1}` participate in an obsolescence chain which is damaged
by |P2| indicating it is ``obsoletedBy`` some object that is not resolvable.

In this case resolving |S1| will result in |P2| which is the most recent
resolvable object in the obsolescence chain.

P_1\binom{S_1}{t_1} & \leftarrow
?? \\

Objects :math:`O_{S_1}` participate in a damaged obsolescence chain since |P2|
indicates it is ``obsoletedBy`` some object that is not resolvable, and |P1|
does not assert it is ``obsoletedBy`` |P2|.

resolve(S_2) &\Rrightarrow P_3

Objects :math:`O = O_{S_1} \cup O_{S_2}` all participate in a damaged
obsolescence chain, with |P1| not indicating it is obsoleted by |P2|, and |P3|
not indicating that it obsoletes |P2|.

P_2\binom{S_1}{t_2} \; & \square \;
P_4\binom{S_1}{t_4} \leftrightarrows
P_5\binom{S_2}{t_5}\\
t_1 < t_2 & < t_4 < t_5\\
resolve(S_2) &\Rrightarrow P_5

A set of objects :math:`O_{S_2} = \lbrace P_5 \rbrace` has the series

Objects :math:`O = O_{S_1} \cup P_3 \cup O_{S_2}` all participate in a damaged
obsolescence chain with no assertion of the relationship between |P2| and |P4|.

 all participate in a damaged
obsolescence chain with no assertion of the relationship between 

In this case resolving |S1| will result in |P4| which is not the most recent

P_2\binom{S_1}{t_2} & \rightarrow

Objects :math:`O_{S_1}` and :math:`O_{S_2}` are both damaged obsolescence
chains though the Coordinating Nodes may infer association between
:math:`O_{S_1}` and :math:`O_{S_2}` since even though the object that |P2| is
``obsoletedBy`` and the object that |P4| ``obsoletes`` can not be resolved,
:math:`P_2.obsoletedBy` and :math:`P_4.obsoletes` are be the same value.

 are both damaged obsolescence

resolve(S_1) &\Rrightarrow P_4

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2, P_4 \rbrace` have the same
series identifier, |S1|.

Objects :math:`O_{S_1}` form a damaged obsolescence chain though it can be
inferred that |P2| is ``obsoletedBy`` and |P4| ``obsoletes`` the same object
even though it can not be resolved, :math:`P_2.obsoletedBy` and
:math:`P_4.obsoletes` are be the same value.

 form a damaged obsolescence chain though it can be
inferred that 

?? \; \square \; ?? \leftarrow
P_5\binom{S_1}{t_5} \\
t_1 < t_2 & < t_5\\
resolve(S_1) &\Rrightarrow P_5

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2, P_5 \rbrace` have the same

The obsolescence chain :math:`O_{S_1}` is broken, with no way to traverse from
|P2| to :math:`P_5`.

The :attr:`~Types.SystemMetadata.dateUploaded` places :math:`P_5` as the newest
object with the series Id of |S1|.

P_2\binom{S_1}{t_2} & \leftarrow
P_3\binom{S_1}{t_3} \\
t_1 > t_2 & > t_3\\
resolve(S_1) &\Rrightarrow P_3

A set of objects :math:`O_{S_1} = \lbrace P_1, P_2, P_3 \rbrace` have the same

Objects :math:`O_{S_1}` form a damaged obsolescence chain since only
``obsoletes`` values are specified.

The :attr:`~Types.SystemMetadata.dateUploaded` of |P1| is newer than |P2|,
which in turn is newer than |P3|.

In this case resolving |S1| will result in |P3| even though |P1| is the most
recent object since the obsolescence chain overrides the times.

 is the most

duplicate label of equation c16, other instance in /var/lib/jenkins/jobs/API_Documentation_trunk/workspace/api-documentation/source/design/ContentMutability.txt

duplicate label of equation c17, other instance in /var/lib/jenkins/jobs/API_Documentation_trunk/workspace/api-documentation/source/design/ContentMutability.txt

The proposed solution is to model and implement a "series identifier" (SID)
along with modified services that would work with both SIDs and PIDs.  From a
DataONE perspective, the series identifiers would be assigned to all versions of
an object, be unique in DataONE (assigned to only one version chain), and would
be reserved just as PIDs - from the same namespace.  The series identifier, once
assigned to the version chain, would similarly be immutable, and could apply to
all new versions of the item.   It is also assumed that in order to coordinate
users to use one identifier for citations, that the cardinality for the citation
identifier would be 0..1.  The semantics for making API calls with a SID would,
in general, be to return responses as if the call were made with the most
current PID.

Member Nodes that only maintain the latest version of an item would be required
to use a new PID for any updated content, and modify the System Metadata
appropriately so that the new version can be synchronized with the network. The
same SID would typically be used for the updated object, although we would allow
the revision chain to shift to a new SID as desired by the client and/or member
node.

It cannot be assumed that a user with an identifier in hand knows whether it is
a SID or a PID, so DataONE expects the user to refer to the System Metadata once
it has the item to determine if the identifier used in the call matches the PID
or the SID.  Similarly, they could interrogate search results for the same
information.  For high-level interfaces, like D1Client.getD1Object(id), the PID
of the object returned may or may not match the passed in 'id'.  So, high-level
functions or applications that use resolve will have to make sure they handle
the new resolving semantics.

It is recommended that search indexes include a search field for the
series identifier that can also be returned in the results.

Type 1: An object on the SID chain doesn't have the "obsoletedBy" field.

Type 2: An object on the SID chain does have the "obsoletedBy" field, but the
PID in the "obsoletedBy" field has a different SID (including no SID value).

It is tricky to determine a type 2 end if the object in the "obsoletedBy" field
is missing. For example, ``P1(S1) ⟺ P2(S1) ⟹ ??``. We don't have the
knowledge of the series id of the object "??". So we generally consider it a
type 2 end except we are sure it is not an end - there is another object in the
chain (has the same series id) that obsoletes the missing object.

is missing. For example, 

. We don't have the

In previous example [P1(S1) ⟺ P2(S1) ⟹ ??], P2 is a type 2 end (case 12).

However,  P1(S1) ⟺ P2(S1) ⟹  ??  ⟸ P4(S1), P2 is not an end (case 8) since "??" is in the obsoletes field of P4 that has the same series id - S1 (We are sure that the "??" has the series id S1 as well, so P2 is not an end).

In P1(S1) ⟺ P2(S1) ⟹  ??  ⟸ P4(S2), P2 is a type 2 end even though "??" is in the  obsoletes field of P4. But P4 has a different series id - S2 (so we are not sure "??" has the S1 or S2).

Ideally, if there is one and only one end on a SID chain, this end will be the HEAD (current) version. This kind of chains are called ideal chains.

If there are more than one end on a SID chain because of the incompleteness of the system metadata, It is hard to determine which one is the real end. This kind of chain is not a ideal chain and we have to use this mechanism to determine the HEAD version:
1. Choose the end with latest dateUploaded in the chain as the temporary HEAD version. This rule works if the uploaded time stamps of objects aren't messed up.
2. If the time stamps are messed up, we need to test if any object obsoletes the temporary HEAD on the obsolete chain with the SAME SID. If nothing obsoletes the temporary HEAD, the temporary HEAD is the final HEAD; otherwise, the end of obsolete chain is the final HEAD.

Choose the end with latest dateUploaded in the chain as the temporary HEAD version. This rule works if the uploaded time stamps of objects aren't messed up.

If the time stamps are messed up, we need to test if any object obsoletes the temporary HEAD on the obsolete chain with the SAME SID. If nothing obsoletes the temporary HEAD, the temporary HEAD is the final HEAD; otherwise, the end of obsolete chain is the final HEAD.

Take this example P1[S1, t1] ⟸P2[S1, t2] ⟸P3(S1, t3) (case 19) (The t1, t2 and t3 are time stamps and t1 > t2 > t3. This means the time stamps are messing up - the newest version P3 was uploaded the earliest while the oldest version p1 was uploaded the latest)
1. This chain has three type 1 ends - P1, P2 and P3. It is not an ideal chain.
2. Choose P1 which has the latest date of uploaded as the temporary HEAD.
3. P2 obsoletes P1 and also P3 obsoletes P2 on the obsolete chain P1 ⟸ P2 ⟸ P3. So we choose the end of the whole chain - P3 as the final HEAD.

This chain has three type 1 ends - P1, P2 and P3. It is not an ideal chain.

Choose P1 which has the latest date of uploaded as the temporary HEAD.

P2 obsoletes P1 and also P3 obsoletes P2 on the obsolete chain P1 ⟸ P2 ⟸ P3. So we choose the end of the whole chain - P3 as the final HEAD.

Mutable content implies that back-versions of content may not be readily available
on the nodes that originally produce the content. For metadata and resource maps,
the coordinating nodes will store previous versions of objects during the synchronization process,
but any data updates will result in only the latest version being available at the originating node.
If the data objects were replicated (as is the hope), it is likely that previous versions of
the data can still be resolved from replica target nodes, though this is dependent on replication policies,
synchronization schedules and the availability of replica storage across the federation.

The current DataONE storage model, through the MN_Storage.update method, places
responsibility for storing versions squarely on the submitter. Each update to the object requires
a new unique identifier (PID) and must state which PID the new version is obsoleting.
We will continue to require that unique PIDs are provided for each
and every version of an object, but the member node will not be required to maintain a copy of previous revisions
if it chooses not to. An optional series identifier (SID) can be provided with object SystemMetadata
to group revisions together and to provide a convenient way to refer to the latest version of the object.

DataONE essentially considers member nodes as the originators of *selected* versions of
content.  That is, not every intermediate revision on the way to a final
product should neccessarily be saved for future reference.  Organizations following the mutable
content model for storage may wish to limit the objects returned by listObjects() to those that
are considered in their publishable form. Certainly theses objects can later be updated as needed,
but minimizing draft-status objects will reduce the amount of [possibly irretrievable] draft content floating around
the federated network.

DataONE essentially considers member nodes as the originators of 

 versions of

As illustrated in the optional use cases, the rate and regularity of change of
objects can be widely variable. The more frequent the change, the less likely
that all versions would need to be reproduced, and the utility of complete version
history diminishes.  One can imagine a member node serving up an unrecorded data
stream, such as a web-cam, delaying creating a version until a user calls MN.get()
on the item, by tee'ing the output stream to file while returning the object.

Additionally the need to keep past versions may be less important for metadata
objects (correcting typos that do not change the meaning or interpretation of the data)
than data objects or resource maps.

The use case of mutable data objects that grow with new records appended to the
end of a table, for example, was given as a common practice for some groups, and
one that would produce progressively redundant information with each persisted
version.  The motivation for rolling up records accumulated over time instead
of new data files for each is the ease of use for end users. Using a SID to access
the data object will always give the latest snapshot of the data records where old revisions
may or may not also be accessible.

Objects like NetCDF files that include both metadata and data in the same object
will be managed with the same PID and SID considerations. If only the metadata portion of
the file is modified, the SID may remain the same, but a new PID and checksum must be created and
made available for synchronization. The old revision may immediately become inaccessible using the PID
and that is allowable under the proposal.

Implicit in the support for versioned content is support for retrieval of, or possibly
just resolution to, the current object bytes by the identifier assigned in the
originating system.  At a minimum CNs will be required to support calculating
which is the current version of series of versions and returning it or its
identifier. This will be accomplished using the series identifier (SID) associated with object[s]
in a revision chain. The "current" version of an object is defined as the non-obsoleted object with
a SID that matches the requested identifier. Objects that are marked as "archived" may be returned as the
most current version, but they should not be seen in default search interfaces.
Since DataONE identifiers have no special formating semantics, those following a citation
will not know by looking at the identifier whether it is referring to a specific
version (PID) or the latest version of the item (SID), so services may be provided to easily investigate
an entire version series. Existing services allow clients to deduce this information by inspecting the system
metadata for the identifier and following any obsolescence properties as needed.

Because the content of an object is retrieved in a separate call from its system
metadata, use of the SID for MN Read API calls is troublesome because the content may be updated
between the two calls.
It would be impossible to tell if the bytes retrieved were incorrect (bit rot) or correct (newer version)
when comparing checksums in this case. If data consistency is important to the caller, the PID should be used
to guarantee that only the expected bytes (or a NotFound exception) are returned by any MN.get calls.

Those making a citation may wish to cite a specific version, or the latest current
version.  Followers of citations may wish to, if given an identifier representing
a specific version (PID), find out what is the latest version (another, newer PID, or the SID).
Conversely, if given a series identifier that navigates to the latest version, they may wish to find
out what the content was at some previous point in time (e.g., the time of the citation) by following
the obsolescence chain backward.

DataONE will be providing CN services for navigating to the latest version of an object, since the only way
to do it currently is for the clients to serially retrieve the system metadata for
versions in the chain until they reach the head version, which is can be inefficient.
A new method to retrieve the entire version history is also under consideration.

MN.get(Identifier id):
Identifier can be either a PID or SID, and if a SID, return the bytes of the HEAD PID in the series.

MN.getSystemMetadata(Identifier id):
If PID, return SystemMetadata of PID.
If SID, return HEAD PID SystemMetadata.

MN.describe(Identifier id):
If PID, return header for PID.
If SID, return header for HEAD PID.

MN.getChecksum(Identifier pid):
Requires PID to effectively verify data integrity.

MN.create(Identifier pid, object, SystemMetadata):
Identifier must be PID and included in accompanying systemMetadata.
SID may be included in accompanying systemMetadata if known at time of creation. The SID mustn't exist in the system.

MN.update(Identifier id, Identifier newPid, SystemMetadata):
Identifier id may be a PID or SID -- in the case of a SID, the method works with the HEAD PID of the chain.
The new Identifier must be a PID and must match the accompanying SystemMetadata.
The new SID can match the old SID in previous SystemMetadata (objects are in the same series),
or it can be any unique SID that does not already exist in the system (newly assigning a SID or shifting the SID because of a "scientifically meaningful change").
Moreover, the new system metadata may not have a SID no matter the previous version has a SID or not.

MN.getLogRecords(?idFilter):
Filter can be PID or SID. The MN should resolve the SID to the HEAD PID, and return the log records for that PID.
If a client wishes to retrieve log records for the entire family of objects referenced by a SID, then the client should retrieve a list of PIDs for the SID, the call getLogRecords for each PID to retrieve the entire set of log records.
The Log.identifier field will only contain PID values, no SIDs.

MN.delete(Identifier id):
Identifier can be PID or SID.
If PID, delete that specific version;
If a SID, delete the HEAD PID version.

MN.archive(Identifier id):
If PID, archive that specific version.
If a SID, archive the HEAD PID.

MN.isAuthorized(Identifier id):
Can accept either PID or SID, but in the case of a SID parameter only reports on the the accessPolicy for HEAD PID.

MN.synchronizationFailed(Identifier pid):
Inter-node communication should only use PIDs for identifying objects.

MN.replicate(Identifier id):
No changes in behavior. SystemMetadata object has changed structure so there is a change in signature.
Replication is based on the PID so that we can ensure content has not been corrupted.

MN.getReplica(Identifier id):
Can only make requests for PIDs so that checksum integrity can be verified.

MN.systemMetadataChanged(Identifier, serialVersion, dateSysMetaModified):
May be called on the MN if the CN infers an obsoletes relationship for a new PID based on a shared SID.
Identifier can be either a PID or a SID.
If a SID, the MN will fetch SystemMetadata from the CN using SID (which will return the HEAD PID SystemMetadata).

MN.listObjects(?identifier=XXX):
Returns an ObjectList like normal, but can be filtered by identifer (SID or PID).
If the Identifier is a PID, it returns just the single entry for that PID.
If the Identifier is a SID, it returns the objects (PIDs) of all objects that have that SID.

MN.view(Identifier id):
Can accept either PID or SID. If a PID, get the formated view for the specified version. If a SID, get the view for the HEAD PID.

MN.getPackage(Identifier id):
Can accept either PID or SID. If a PID, get the package of the specified version. If a SID, get the package of the HEAD PID.

MN.updateSystemMetadata(Identifier id, SystemMetadata newSysmeta):
Requires a PID. The SID can exist in the newSysmeta object. Since SID is immutable, the SID in the newSysmeta should match the current SID if it exists. If current system metadata doesn't have a SID, the new SID can be one of the following cases:
  1. The new SID is null (without a SID).
  2. The new SID is a unique identifier which doesn't exist in the system.
  3. The new SID matches the SID in the system metadata of the object in the "obsoletes" value.
  4. The new SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

1. The new SID is null (without a SID).
2. The new SID is a unique identifier which doesn't exist in the system.
3. The new SID matches the SID in the system metadata of the object in the "obsoletes" value.
4. The new SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

The new SID is a unique identifier which doesn't exist in the system.

The new SID matches the SID in the system metadata of the object in the "obsoletes" value.

The new SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

CN.getSystemMetadata(Identifier id):
Behaves the same as MN.
N.B. This method can be used with a SID to locate the PID of the latest version which may be sufficient without implementing a
getHead() method.

CN.create(Identifier pid, object, SystemMetadata):
Identifier must be PID and included in accompanying systemMetadata. SID may be included in accompanying systemMetadata if known at time of creation. The SID can be one of the following cases:
  1. The SID is a unique identifier which doesn't exist in the system.
  2. The SID matches the SID in the system metadata of the object in the "obsoletes" value.
  3. The SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

1. The SID is a unique identifier which doesn't exist in the system.
2. The SID matches the SID in the system metadata of the object in the "obsoletes" value.
3. The SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

The SID is a unique identifier which doesn't exist in the system.

The SID matches the SID in the system metadata of the object in the "obsoletes" value.

The SID matches the SID in the system metadata of the object in the "obsoletedBy" value.

CN.registerSystemMetadata(Identifier pid, SystemMetadata sysmeta):
Requires a PID, allows a SID in the SystemMetadata. The SID can be one of the following cases:

CN.updateSystemMetadata(Identifier id, SystemMetadata newSysmeta):
Behaves the same as MN

CN.reserveIdentifier(Identifier id):
Accepts PID or SID values and treats them exactly the same.

CN.hasReservation(Identifier id):

CN.resolve(Identifier):
If PID, resolve it.
If a SID, then resolve the HEAD PID.

CN.isNodeAuthorized(Identifier id):
Only accept PID since it is a replication related method. No behavior change.

CN.updateReplicationMetadata(Identifier id):

CN.deleteReplicationMetadata(Identifier id):

CN.setReplicationStatus(Identifier id):

CN.setReplicationPolicy():

CN.setRightsHolder():
Ownerships apply to particular revisions, not the entire chain.
If a SID is passed in to a method that affects one of these policies, the change is applied to the HEAD PID for that series.

CN.setAccessPolicy():
Policies apply to particular revisions, not the entire chain.

CN.setObsoletedBy(Identifier id, Identifier obsoletedByPid):
Only PIDs can be used when expressing obsolescence chain.

As is currently the case, the member node should maintain all versions of content using
unique identifiers (PID) and synchronization will harvest each new revision to the network.
While there will be no requirement that the Member node continue to make available the object identified
by the obsoleted PID, the hope is that they will persist the data history as best they can.
If the objects in the revision chain have a SID assigned, the new PID will be considered the latest
version of this series.

The member node can allow access to the current version of the object using MN_Read.get(sid) as a convenience and
any reference to the SID would resolve to the latest version of the object with a potentially different checksum and PID
from what was originally present when the citation was distributed.

The member node must [minimally] maintain system metadata for the current revision of the object.
Any updated object is still required to be identified by a new unique PID, but would include the same SID used
in the previous version. The obsoletes field should indicate that the new PID replaces the previous PID.
The coordinating node learns about the updated content during synchronization because there is:

N.B. Multiple revisions between synchronization periods would not
result in multiple versions recorded in the federation - just the revision[s] that happened to be
synchronized would be persisted in DataONE. This leaves open the possibility of an end user retrieving a version from the MN that
will ultimately not be persisted in perpetuity.

The use cases below organize the identified requirements related to mutable
content, with the most relevant use cases listed first.

Defined as activities that help ensure continued discoverability and usefulness
and usually in reference to metadata, not data.

absent a "push" notification, users should be able to easily determine if they
have the most current version of something, and easily and quickly get it.

Minimize the burden of adaptation to working with versioned content.

Allow use of their identifiers in DataONE in the context they are familiar with
(if their identifier always points to the latest, in DataONE it should too)

Differentiating between incremental internal saves, vs. new revision.

avoid unnecessary costs associated with obtaining resolvable (e.g., DOIs) for each version

coordinating citation by a common identifier for citation tracking

ensuring that the cited object is the same when accessed as when it was originally used

What is the best way to version mutable data that frequently changes but may or
not be used.  For example a "current time" object, replaced every minute, or
"current weather radar" that's replaced every 3 hours.

preserving every version could be very expensive for very little value

The underlying dynamic here is the the rate of mutation vs. the rate of synchronization

This means supporting data objects that add records over time, either:

within pre-defined bounds  e.g. "2013 year-to-date"  (the metadata could stay
the same, while data changes)

without pre-defined bounds e.g. "JGoodall primate observation log"?

Some formats combine data with metadata, for example netCDF, so allowing the
metadata to change without impacting the consistency assessment of the data itself.

changes in the file are treated like any other change; they will be versioned,
      but may be referenced using a seriesId

but may be referenced using a seriesId

Mutable content can theoretically include things that are live feeds from
sensors, but are otherwise not captured.

This proposal does not accommodate streams unless they have discrete snapshots that can be referenced as part of
a seriesId.

Should we allow identifiers to resolve to a URL that returns an input stream?

Can we mark it as the user's responsibility to do the mn.create?

Reassignment of AuthoritativeMemberNode field for unhosted versions

Current DataONE replication processes and fixity checks depend on content
identified by a PID that does not change. If this were not enforced, mutable
content from a member node would not be differentiated from corrupt copies of
the object and our replication and recovery features would attempt to correct
the byte inconsistency. The immutability requirement helps to ensure
reproducible results of any use of an object. Any analysis on a data set
repeated sometime in the future should yield identical results (within the
limits of precision of the analytical tools) and this is one of the major
guiding principles in creating DataONE as a long term data repository
federation. By simply overwriting existing content using the same identifier,
nodes cannot be relied upon for repeatable retrieval of content.

At its core, DataONE is in the business of preserving definite versions of
content through centrally coordinated per-to-peer replication.  That is, DataONE
Coordinating Nodes direct certain Member Nodes to replicate newly synchronized
objects from the originating Member Node to better preserve it.  New versions of
objects appear as first class immutable objects with unique PIDs, even if
originating from mutable Member Nodes.

From the DataONE perspective the only difference between objects from mutable
Member Nodes and immutable Member Nodes is the completeness of the series of
versions it is able to synchronize and replicate.

For Member Nodes that employ a mutable content storage model, the only
additional DataONE requirement is that the Member Node generate a SystemMetadata
document for the updated content, containing:

Ideally, the SystemMetadata of now unavailable versions will be maintained, and
the ``obsoletedBy`` field is populated with the PID of the version that replaced
it.

the 

 field is populated with the PID of the version that replaced
it.

Some Member Nodes may opt to preserve recent back-versions to aid the complete
capture of versions by the DataONE network via synchronization.

All content synchronized by DataONE is immutable, and so resolution of a
:term:`persistent identifier` (PID) will always result in a pointer (URI) to a set of
bytes that are in all respects identical to the original. Version 2.0 of the
DataONE APIs introduced the ability to associate an optional series identifier
(SID) with an object. Unlike a PID, resolution of a SID will always result in a
pointer (URI) to a set of bytes that represent the latest revision of an object.

 (PID) will always result in a pointer (URI) to a set of

A revision or obsolescence chain is constructed by setting the obsoletes and
obsoletedBy properties of the new and old objects respectively. For example,
here PID_B represents the latest revision of object as it obsoletes PID_A
(object PID_A has a value of "PID_B" in its system metadata
:attr:`~Types.SystemMetadata.obsoletedBy` property, and object PID_B has a
value of "PID_A" in its system metadata :attr:`~Types.SystemMetadata.obsoletes`
property)::

 property, and object PID_B has a
value of "PID_A" in its system metadata 

+------------+                      +------------+
|            | ----- obsoletes ---> |            |
|   PID_B    |                      |   PID_A    |
|            | <--- obsoletedBy --- |            |

             resolve(PID_A) => PID_A
             resolve(PID_B) => PID_B

In version 1.x of DataONE, it was necessary to manually follow the obsolescence
chain in order to find the latest version of an object. This process is
simplified in version 2.x and later through the use of series identifiers. The
previous example can be augmented with series identifiers::

previous example can be augmented with series identifiers:

|   SID_1    |                      |   SID_1    |

             resolve(SID_1) => PID_B

Each object in the obsolescence chain has the same value for the series
identifier ("SID_1"), and calling :func:`~CNRead.resolve` with the value "SID_1"
will result in the URIs from which the object "PID_B" may be retrieved, since
that object is the latest revision in the obsolescence chain.

identifier ("SID_1"), and calling 

 with the value "SID_1"

The availability of PIDs and SIDs means users may now refer to objects using
either a PID when it is necessary or appropriate to refer to an exact set of
bytes that represent an object or through a SID when referring to the latest
version of an object. The former is important for repeatable analyses, since
the same content may be reliably referenced and retrieved. The latter is
important for referencing the most up to date revision of some object, and so
may be useful for example to perform anaysis with the latest information
available.

Unless indicated otherwise, the DataONE version 2.x and later APIs will accept
either a PID or a SID when an identifier is specified as a request parameter.

Depending on the Member Node used as the primary repository, content originators
may have some choice in assigning identifiers. For those that do, it is advised
that they assign PIDs and SIDs according to the typical usage pattern described
above.

Some Member Nodes may not preserve past versions of content, in which case the
PID is likely to be automatically generated, and the submitter only has to
determine the SID, and may not need to know the difference between the SID and
PID.  Other Member Nodes may still be at v1 of the DataONE APIs and only allow
assignment of the PID.

The proposal for supporting "mutable" content is to allow a series identifier
(SID) to facilitate the semantics of citing an object at the conceptual level,
instead of the version level. As content changes over time, new identifiers
(PIDs) will still be used to mark each change, but the conceptual object can
continue to be referred to with an unchanging identifier (SID). The member node
will be responsible for creating each version and assigning a unique PID to it
and these objects will be synchronized and replicated to other DataONE member
nodes as they are today. So instead of allowing content to be directly modified,
we are allowing strongly-versioned chains to be referenced by an identifier; and
relaxing the requirement that all revisions be resolvable forever.

The SID is used to conceptually represent an object that may vary modestly over
time, but remains conceptually the same. Content contributers should be careful
to apply reasonable limits on the scope of documents such that an entity does
not deviate too much from the original item.  In such cases, a new / different
series should be initiated.

DataONE will attempt to synchronize all versions it's made aware of through the
synchronization process, but may miss short-lived versions that are in existence
only between the Member Node's synchronization interval.    Please note, also,
that the synchronization schedule is not guaranteed.  Periods of DataONE
maintenance may suspend synchronization, or high CN load could prolong the
synchronization interval.

Member Nodes keen to make sure versions have the highest chance of
synchronization can choose to issue a :func:`CNCore.synchronize` command that
will put the item on the synchronization queue instead of waiting for the
harvest interval.

synchronization can choose to issue a 

 command that

Conversely, if the Member Node expressly doesn't want DataONE to preserve
back-versions, they can set systemMetadata.replicationPolicy.numberReplicas
field to 0.

The use of the PID or SID for either citation or analysis workflows is up to the
user and is context dependent. In general, DataONE anticipates ``DATA`` and
``RESOURCE_MAP``  objects will be referenced by PID, to ensure reproducibility;
and in general,  ``METADATA`` documents will be referenced by SID, to take
advantage of any data  curation / correction efforts that would not otherwise
affect scientific  reproducibility.  Additionally, clues for the content
submitter's preference can  be found in the format of the identifiers
themselves. For example, DOIs and EZIDs  take a recognizable format, and are
often encouraged in scientific communities for  citations, so an end-user might
take that into consideration when deciding which  identifier to choose.

user and is context dependent. In general, DataONE anticipates 

  objects will be referenced by PID, to ensure reproducibility;
and in general,  

 documents will be referenced by SID, to take

<locations identifier='1234'>
  <location node='mn1.dataone.org/mn' href='http://mn1.dataone.org/mn/object?id=1234' />
  <location node='mn2.dataone.org/some_base/mn' href='http://mn1.dataone.org/some_base/mn/object?id=1234' />
  <location node='cn.dataone.org/cn' href='http://cn.dataone.org/cn/object?id=1234' />
</locations>

...
<ol identifier='1234'>
  <a href='http://mn1.dataone.org/mn/object?id=1234'>mn1.dataone.org/mn</a>
  <a href='http://mn2.dataone.org/some_base/mn/object?id=1234'>mn1.dataone.org/some_base/mn</a>
  <a href='http://cn.dataone.org/cn/object?id=1234'>cn.dataone.org/cn</a>
</ol>
...

{
 'identifier':'1234',
 'locations': [
   ['mn1.dataone.org/mn', 'http://mn1.dataone.org/mn/object?id=1234'],
   ['mn2.dataone.org/some_base/mn', 'http://mn2.dataone.org/some_base/mn/object?id=1234'],
   ['cn.dataone.org/cn', 'http://cn.dataone.org/cn/object?id=1234'] ]
}

<d1:listObjects xmlns:d1="http://dataone.org/service/types/ListObjects/0.1"
  xmlns:common="http://dataone.org/service/types/common/0.1"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  count="_integer_"
  start="_integer_"
  total="_integer_">
    <identifier>_identifier_</identifier>
    <objectFormat>_object class_</objectFormat>
    <checksum algorithm='SHA-1 | SHA-224 | MD5 | ...'>_hash of object_</checksum>
    <dateSysMetadataModified>_date time last modified_</dateSysMetadataModified>
    <size>_byte size of object_</size>
  ...
</d1:listObjects

{
  'start': <integer>,
  'count': <integer>,
  'total': <integer>,
  'objectInfo':
  [
    {
      'identifier':<IdentifierType>,
      'objectFormat':<ObjectFormatType>,
      'checksum':
      {
        'algorithm':<checkSum Algorithm Type>,
        'value':<SHA1 hash of object>
      },
      'dateSysMetadataModified':<DateTime>,
      'size':<integer>
    },
    ...
  ]
}

When the response is simply a boolean True or False, the response returned is in
one of two forms:

For cases where false is a valid response: a single character "1" if *True*,
or "0" if *False*

For cases where false is a valid response: a single character "1" if 

For cases where false coincides with a DataONE exception: an arbitrary body
if *True* (ignored by caller), and an error-indicating HTTP status code and
serialized error object if *False*.

if 

 (ignored by caller), and an error-indicating HTTP status code and
serialized error object if 

Returned in response to :func:`MN_crud.describe` which is called using the
HTTP HEAD verb. As such, the response is entirely in the HTTP headers (no body).

 which is called using the

#<start>,<count>,<total>
identifier,objectFormat,algorithm,checksum,dateSysMetadataModified,size
"<identifier>","<object class>","<SHA-1 | SHA-224 | MD5 | ...>","<SHA1 hash of object>","<date time last modified>",<byte size of object>

roadmap to system metadata control changes (draft - to be reviewed)

The example instance document included here was auto-generated so does not
include useful values. It is included here to provide a general indication as
to the structure of a populated system metadata document.

Include file u'/var/lib/jenkins/jobs/API_Documentation_trunk/workspace/api-documentation/source/d1_schemas/instance-eg-v11.xml' not found or reading it failed

Roadmap to System Metadata Control Changes (DRAFT - to be reviewed)

The following outline describes the policy and technical steps needed to shift the majority of control of system metadata attributes to Member Nodes such that client operations are more responsive.  The changes would require a new DataONE API v2 that involve changes to the DataONE Types schema, changes to the Member Node APIs, changes to the Coordinating Node APIs, and changes to the various software stacks that implement these APIs.  It will also involve a release and deployment schedule that allows both v1 and v2 of the APIs to be in operation simultaneously.

In transferring control to the Member Nodes, they also adopt the responsibility of consistently managing the versions of the documents in a serial manner.  Use of the `serialVersion` attribute ensures that previous values are not overwritten by new values out of order (e.g. AccessPolicy)

In transferring control to the Member Nodes, they also adopt the responsibility of consistently managing the versions of the documents in a serial manner.  Use of the 

 attribute ensures that previous values are not overwritten by new values out of order (e.g. AccessPolicy)

The main use case involves access control.  When a scientist using an ITK client creates an object through MN.create(), control of the system metadata is currently transferred to the CN once synchronization happens. After that point, the ITK client (and scientist) has to make CN.setAccessPolicy() calls to make any changes. If the MN is set to sync once a week, this is problematic, since the scientist would naturally expect that the access control changes should take effect immediately.

@startuml images/sysmeta_cn_control.png
skinparam notebordercolor #AAAAAA
skinparam notefontcolor #222222
title Current Set Access Policy Sequence\n\n
participant "Client" as Client <<D1Client>>
participant "MN" as MN <<MNode>>
participant "CN" as CN <<CNode>>
Client -> MN : MNStorage.create(pid, object, sysmeta)
Client -> CN : CNAuthorization.setAccessPolicy(pid, accessPolicy, serialVersion)
activate CN #D74F57
CN --> Client :    NotFound
note right
  The client must handle NotFound exceptions
  that are unexpectedly returned.
end note
... <b>Potentially long delay due to sync schedule</b> ...
CN -> MN : MNRead.listObjects(fromDate, toDate)
activate MN #D74F57
MN --> CN : objectList
deactivate MN
CN -> MN : MNread.getSystemMetadata(pid)
MN --> CN : sysmeta
CN -> CN : CNCore.registerSystemMetadata(pid, sysmeta)

note right
  Success, only after CN is sync'd.
end note
@enduml

@startuml images/sysmeta_mn_control.png
title Proposed Set Access Policy Sequence\n\n
MN --> Client : pid
Client -> MN : MNStorage.updateSystemMetadata(pid, SystemMetadata)
note left
     SystemMetadata includes AccessPolicy
end note
MN --> Client :    true
note right
  Success. The client calls directly
  to the MN without delay.
end note
   deactivate MN

MN -> CN : CNCore.updateSystemMetadata(pid, sysmeta)
CN --> MN :    [true | false]
note right
  The MN calls the CN;
  best-faith effort to keep
  it synchronized, but fine if the object
  has not been harvested.
end note

note left
  Now the CN finally gets the
  SystemMetadata, with most recent values
  already set on the MN
end note

@enduml

This document describes the management of system metadata across nodes, and has been updated to reflect control of system metadata attributes by the MN rather than the CN, except for the Replicas listed per object.
Text changes are highlighted here_.

Text changes are highlighted 

Changes would be required for both the Member Node and Coordinating Node APIs, in both the architecture documentation and the `d1_common_java` and `d1_common_python` libraries:

Changes would be required for both the Member Node and Coordinating Node APIs, in both the architecture documentation and the 

As an alternative to individual MN APIs above, we might want to consider using a single MN call to update system metadata documents:

CJ and BRL discussed this and decided the single updateSystemMetadata method would suffice and implementations could determine which mutable fields from the SystemMetadata it would update.
TBD: do we reject updates if an immutable field differs from the original value  even if we never intend to save that new value anyway?

The DataONE Client Libraries (d1_libclient_java and d1_libclient_python) will need to be changed to support the above API changes in v2, as well as the existing v1 APIs.  This will help multiple MN software stacks in supporting both APIs.

The CN REST Service will need to be modified to add and deprecate the methods listed above. Likewise, the CN REST Proxy will also need to be adjusted.

With these changes, d1_synchronization classes will need to consult the node registry to determine if an MN implements v1 or v2 of the API, and act accordingly.  As the synchronization code adds in replica entries, it should notify the authoritative Member Node and all replica Member Nodes of the change using MNRead.systemMetadataChanged() calls.  It will also need to call CNReplication.setReplicaVersion() for new entries.

The CN ReplicationManager code will need to be adjusted to 1) Get authoritative copies of system metadata from the MN, 2) use CNReplication.getReplicaVersion() and CNReplication.setReplicaVersion() when processing replica tasks rather than setting the `serialVersion` of the system metadata document.

The CN ReplicationManager code will need to be adjusted to 1) Get authoritative copies of system metadata from the MN, 2) use CNReplication.getReplicaVersion() and CNReplication.setReplicaVersion() when processing replica tasks rather than setting the 

The MetacatCNodeService class will need to be modified to implement the above CN API calls.  Likewise, the database schema will need to change to store a new `version` column in the `smreplicationstatus` SQL table.  This will also affect other classes that manage the persistence of system metadata, namely IndetifierManager.  Upgrade classes and scripts will need to be written for existing installations.

The MetacatCNodeService class will need to be modified to implement the above CN API calls.  Likewise, the database schema will need to change to store a new 

 SQL table.  This will also affect other classes that manage the persistence of system metadata, namely IndetifierManager.  Upgrade classes and scripts will need to be written for existing installations.

Member node software stacks will need to implement the API methods listed above, and will need to ensure that other calls that affect system Metadata entries also update Coordinating Node system metadata copy.
For instance, a call to MNStorage.update() should also call CNCore.updateSystemMetadata() so that the CNs remain in sync with the MNs with regard to system metadata.

We will need to establish a release schedule and deploy software stacks, likely in this order:

Note that we plan on introducing other changes into the DataONE types schema to accommodate mutable content and other features.  Changes to the type schema should be consolidated to reduce the impact on software that depend on the types.

By adding an optional `version` attribute to the Replica Type, the Coordinating Nodes would no longer need to rely on the `serialVersion` attribute of the entire system metadata document to manage versions.
A Replica example, with the version line highlighted, would be:

 attribute to the Replica Type, the Coordinating Nodes would no longer need to rely on the 

 attribute of the entire system metadata document to manage versions.

<replica version="1">
  <replicaMemberNode>urn:node:PISCO</replicaMemberNode>
  <replicationStatus>completed</replicationStatus>
  <replicaVerified>2012-07-10T00:00:00.000+00:00</replicaVerified>
</replica>

By making the `version` attribute optional, this approach would be backwards-compatible with existing system metadata documents in the system.
However, the Replica list in System Metadata documents on the MN may be out of sync with the list on the CN during times of rapid change such as MN-to-MN replication operations.

 attribute optional, this approach would be backwards-compatible with existing system metadata documents in the system.

Another approach is to remove the Replica entry from the SystemMetadata Type entirely, and manage replicas separately.
This approach would be backwards-incompatible with existing system metadata documents, but once upgraded, all Replica information would be obtained through the CN services.

2.3 Leave the data types as is and let the CN have control over both the replica list and the serialVersion as it currently does.
We always hope and intend that the MN and CN will have the same consistent SystemMetadata eventually. In this scenario, the CN would ignore any values the MN provided for SM.serialVersion and SM.replica and the MN would accept
those values as provided by the CN's copy of SystemMetadata. This allows much of our processing on the CN to remain as is and the different types of nodes then choose which parts of SM to manage/ignore.
BRL: I believe we decided to pursue this course for now.

.. _`Types Schema`: http://mule1.dataone.org/ArchitectureDocs-current/apis/Types.html

The structure of :class:`~v2_0.Types.SystemMetadata` is defined in XMLSchema.

The current release of the System Metadata schema involves three parts. Version 1.0 provides a full schema which version 1.1 extends, which is in turn extended by version 2.0.

Release versions of the schema are available from their namespace URIs:

Development versions of the schema are maintained in a subversion repository at:

.. figure:: images/sysmeta_flow.png

   *Figure 2.* System metadata is provided by Member Nodes (1) in response to a
   :func:`~MNRead.getSystemMetadata` call by the Coordinating Nodes during the
   Member Node synchronization process (2). The Coordinating Node updates the
   replica location information (3). The copy of system metadata is replicated
   between the Coordinating Nodes, and the Member Node is informed of the
   system metadata change (4). A user will typically retrieve system metadata
   from the Member Node (5) using the :func:`~MNRead.getSystemMetadata` call
   since that is the authoritative source for the information.

*Figure 2.* System metadata is provided by Member Nodes (1) in response to a
:func:`~MNRead.getSystemMetadata` call by the Coordinating Nodes during the
Member Node synchronization process (2). The Coordinating Node updates the
replica location information (3). The copy of system metadata is replicated
between the Coordinating Nodes, and the Member Node is informed of the
system metadata change (4). A user will typically retrieve system metadata
from the Member Node (5) using the :func:`~MNRead.getSystemMetadata` call
since that is the authoritative source for the information.

 System metadata is provided by Member Nodes (1) in response to a

 call by the Coordinating Nodes during the
from the Member Node (5) using the 

 call

.. figure:: images/sysmeta.png

   *Figure 1.* All managed content (science metadata, science data, and resource
   maps) in DataONE is accompanied by *system metadata* (1, 2, 3 respectively).
   The relationships between science metadata and data object, and thus the
   structure of a data package, is described by resource maps. In this simple
   data package, the resource map indicates that the science metadata document
   *documents* the science data (4), and that the science data *isDocumentedBy*
   the science metadata (5).

*Figure 1.* All managed content (science metadata, science data, and resource
maps) in DataONE is accompanied by *system metadata* (1, 2, 3 respectively).
The relationships between science metadata and data object, and thus the
structure of a data package, is described by resource maps. In this simple
data package, the resource map indicates that the science metadata document
*documents* the science data (4), and that the science data *isDocumentedBy*
the science metadata (5).

 All managed content (science metadata, science data, and resource
maps) in DataONE is accompanied by 

 (1, 2, 3 respectively).

System metadata elements are partitioned into two classes: metadata elements
that must be provided by client software to the DataONE system, and elements
that are generated by DataONE itself in the course of object management.

The mutability of system metadata elements is described in Table 1.

**Table 1.** Mutability of system metadata. Values are initialized by
different components during creation, and those values are vetted by
(controlled by) downstream, authoritative components. Mutable properties are
edited through the specified edit method.

 Mutability of system metadata. Values are initialized by

The identifier is assigned during creation and can not be changed. The
CNs will not allow reuse of an identifier, so content created with an
identifier that is not unique will be rejected by CNs.

The object checksum is determined during creation and can not be changed.

The object submitter is set during creation and can not be changed.

Upload to a MN occurs once, and the MN MUST set this value to indicate
when the content was added to the repository (may be before the
repository started operating as a MN).

The seriesId can be set if it has no value, but can not be changed once
set. Use :func:`MNStorage.updateSystemMetadata`

set. Use 

The ``obsoletes`` property may be set once, to indicate that the current
object obsoletes or supercedes the PID that is the value of this field.
Care must be taken when setting this value to ensure that no branching
or circular lists are created.

 property may be set once, to indicate that the current

The ``obsoletedBy`` property may be set once, to indicate that the current
object is obsoleted superceded by the PID that is the value of this field.

v1: Set by CNs in response to any operation that alters System Metadata.

Change is discouraged, values limited to the formats available from
:func:`CNCore.listFormats`.

Manual update process that must be performed by a DataONE administrator.

:func:`CNReplication.updateReplicationMetadata` (internal to
Coordinating Nodes)

In version 2.x, properties of System Metadata are set via the Member Node using
the :func:`MNStorage.updateSystemMetadata` implemented by version 2.x Member
Nodes. This method is implemented as a HTTP PUT agains the /meta endpoint of the
Member Node with the identifier included as a URL parameter and the new System
Metadata serialized as XML included in the MIME-Multipart payload.

the 

 implemented by version 2.x Member

Every object (science metadata document, data object, or resource map) managed
by DataONE has number of properties that are used to faciliate access and
mangement of the object (Figure 1). These properties are collectively called "system
metadata" (aka. :term:`sysmeta`). This document describes which properties of
system metadata may be edited and why. More detailed information about the
purpose for each property and its range of values can be found in the schema
and associated documentation available at :class:`Types.SystemMetadata` and
:class:`v2_0.Types.SystemMetadata`.

metadata" (aka. 

). This document describes which properties of
and associated documentation available at 

System metadata is maintained dynamically by each Member Node and synchronized
with Coordinating Nodes through the synchronization process or more directly
through the :func:`CNRead.synchronize` API. The Member Node copy is
authoritative except for replica information which the Coordinating Nodes
control (Figure 2). Collation of system metadata properties starts with a
client adding content to a Member Node using the :func:`MNStorage.create` API.
System metadata contains properties that are immutable, may be set once, or may
be altered at any time by an authorized subject.

through the 

 API. The Member Node copy is
client adding content to a Member Node using the 

 API.

Table 1. lists the properties of System Metadata that may be altered by
authorized users. This section describes how those changes are performed.

In all cases it is assumed that the user is appropriately authenticated and is
authorized to perform the operation. Users must have ``CHANGE_PERMISSION``
permission in order to alter system metadata. The object ``rightsHolder``, the
identity of the ``authoritativeMemberNode``, and the identity of a Coordinating
Node will always have permission to update system metadata for an object.

authorized to perform the operation. Users must have 

, and the identity of a Coordinating

All System Metadata, science metadata, and resoruce map documents from a MN
are also present on the CN

Science metadata records, resource maps, and system metadata for all objects
available on a Member Node are synchronized to a Coordinating Node.

Signal to CN issued by a MN that is requesting synchronization

The Member Node synchronization process will operate in an asynchronous
manner, with a task queue on the CN service containing a list of objects that
need to be added to the coordinating node populated through two mechanisms:

through a new notification API that is called by a Member Node after a
successful create operation.

The task queue is processed by a pool of workers which retrieve the content
using :func:`MNRead.get` and :func:`MNRead.getSystemMetadata`, and store the
content in the Coordinating Node using the internal :func:`CNStorage.create`
method.

using 

, and store the
content in the Coordinating Node using the internal 

The initial implementation will focus on using only scheduled calls to
:func:`MNRead.listObjects` to populate the task queue. The scheduling
information shall be set during the MN registration process.

 to populate the task queue. The scheduling

Multiple threads should be able to process the task queue, and the number of
threads should be adjustable by an administrator. The total number of threads
hitting on a particular MN should be limited, the limit should be specified in
the MN capabilities recorded during the registration process.

The tasks queue will need to be thread safe. In the future it may be feasible
to use a distributed queue (based on Apache MQ or Zookeeper for example;
shared across the CNs) so that processes on any CN can issue the
:func:`MN_get` and :func:`MN_getSystemMetadata` requests.

There will likely be a significant load on the system for indexing as new
content is being added - this is the case for both Mercury and Metacat.
Options for disabling indexing for bulk load should be considered.

The synchronization process will certainly be more sophisticated down the road,
but for now (i.e. prototype implementation) simplicity should be a priority:

MNs are only scanned by the CN where they are registered (restriction to be
dropped down the road).

listObjects is polled (with max objects set to some limit), with results
stored in a queue on the CN

as the queue is processed and shrinks to some size, then listObjects process
is polled again

the queue is processed by n worker threads, with each task being to pop an
entry from the queue, pull the content from the target, and store it in the
object store.

when there is some number of new entries in the object store, then the
mercury indexing process needs to kick in. Ideally this should be updating
rather than re-creating the index at each pass. This may be initiated by a
cron job for the prototype implementation.

This fairly simplistic approach should be enough to get things started. Not
ideal, but should suffice to get some data moving around. To implement, there
is need for a few new components - a queue, a place to store state
information, the code that does the polling, the code that does the object
retrieval, the worker thread code, and an overall controller service.

Retrieve objects from Member Nodes. getNextMemberNode() gets the next Member
Node to work on - this could be serial selection from the list of registered
MNs, could be based on the last update, perhaps from some hint returned in
:func:`MN_health.ping`.

MemberNodeState
  String url;  //URL of the member node
  DateTime lastCheckTime;  // time stamp for when the MN was last polled for listOjects

RetrievalTask
  String url;
  ObjectInfo objectInfo;

void populateTaskQueue ()
{
  ThreadSafeQueue taskQueue = getSharedTaskQueue()
  while ( keepRunning() )
  {
    if (taskQueue.length < MIN_TASKS)
    {
      MemberNodeState = getNextMemberNode();

      // This could be executed in a different thread, which would
      // enable multiple MNs to be polled at a tme if necessary
      int numNewEntries = loadMNTasks(taskQueue, mnState);
    }
  }
}

int loadMNTasks(ThreadSafeQueue taskQueue, MemberNodeState mnState)
{
  AuthToken token = getSystemToken();
  D1cient client = D1Client( mnState.url );

  // get a list of all the new entries since lastCheckTime
  DateTime tstamp = now();

  // Will actually need to loop here to page through results
  ObjectList objectList = client.listObjects(token, mnState.lastCheckTime);
  for (ObjectInfo objinfo : objectList)
  {
    RetrievalTask task = RetrievalTask(mnState.url, objinfo)
    taskQueue.push(task)
  }
  mnState.lastCheckTime = tstamp;
  setMemberNodeState(mnState);
  return objectList.length;
}

Worker threads are responsible for retrieving each object listed in the
taskQueue. The number of workers should be adjustable. An obvious target for
improving efficiency is to enable requesting multiple objects in a single call
to a MN (later enhancement)::

to a MN (later enhancement):

// object is the next entry from the taskQueue
bool retrieveObject(String mnurl, ObjectInfo object)
{
  Authtoken token = getSystemToken();
  D1Client client = D1Client(mnurl);
  SystemMetadata sysmeta = client.getSystemMetadata( object.identifier );
  if ( isScienceMetadata(sysmeta.objectClass) )
  {
    scimeta = client.get( object.identifier );
    CN_SERVICE.create( object.identifier, sysmeta, scimeta );
  }
  else
  {
    CN_SERVICE.create( object.identifier, sysmeta );
  }
}

As content is added to Member Nodes, the associated system metadata, science
metadata, and resource map documents are retrieved by a Coordinating Node.
Once the objects are registered with the Coordinating Node, they are
considered to be part of the DataONE system, and so will be managed
accordingly - the metadata and resource map object will be replicated between
Coordinating Nodes, metadata will be indexed, and the objects will be
replicated to other Member Nodes as required.

Content replication as managed by Coordinating Nodes is described in Use Case
09.

In version 1.x of the DataONE infrastructure, synchronization is through polling
only. A Coordinating Node will periodically request a list of objects that are
new or have been altered for some specified time period using the listObjects
method.

In version 2.x of the DataONE infrastructure, a new mechanism to enable clients
or Member Nodes to request synchronization of an object was added. This helps
ensure a more rapid update of object properties such as access control rules are
propogated through the infrastructure.

.. uml::

  @startuml images/06_uc_a.png
  autonumber "[0] "
  participant MN <<Member Node>>
  participant CN <<Coordinating Node>>

  CN -> MN: listObjects( timePeriod )
  activate CN
  MN -> CN: objectList
  loop "for each PID"
    CN -> CN: queue PID for synchronization
  end
    deactivate CN

  ... <font color=red>**Possibly Lengthy Delay** ...

  CN -> MN: getSystemMetadata(PID)
  activate MN
  MN -> CN: SystemMetadata
  deactivate MN
  alt New Objec
    activate CN
    CN -> CN: store System Metadata
    CN ->o]: Notify index of new System Metadata
    alt "Is Science Metadata or Resource Map"
      CN -> MN: get(PID)
      activate CN
      activate MN
      MN -> CN: object
      deactivate MN
      CN -> CN: store object
      CN ->o]: Notify index of new content
      deactivate CN
    end
  else "Existing Object"

    CN -> CN: update system metadata properties
    CN ->o]: Notify index of modified content
  end
  deactivate CN
  @enduml

@startuml images/06_uc_a.png
autonumber "[0] "
participant MN <<Member Node>>
participant CN <<Coordinating Node>>

CN -> MN: listObjects( timePeriod )
MN -> CN: objectList
loop "for each PID"
  CN -> CN: queue PID for synchronization
end

... <font color=red>**Possibly Lengthy Delay** ...

CN -> MN: getSystemMetadata(PID)
activate MN
MN -> CN: SystemMetadata
alt New Objec
  CN -> CN: store System Metadata
  CN ->o]: Notify index of new System Metadata
  alt "Is Science Metadata or Resource Map"
    CN -> MN: get(PID)
    activate MN
    MN -> CN: object
    deactivate MN
    CN -> CN: store object
    CN ->o]: Notify index of new content
  end
else "Existing Object"

  CN -> CN: update system metadata properties
  CN ->o]: Notify index of modified content
end
@enduml

**Figure 2.** Overview of synchronization process using a polling process to
periodically request information about objects that have changed within a time
period.

 Overview of synchronization process using a polling process to
period.

.. uml::

  @startuml images/06_seq_a

  MN -> CN: synchronize(PID)
  CN -> MN: ack

    end

  end
  @enduml

@startuml images/06_seq_a

MN -> CN: synchronize(PID)
CN -> CN: queue PID for synchronization
CN -> MN: ack

  end

end
@enduml

**Figure 3.** Overview of synchronization process with an external request to
synchronize an object rather than relying on periodic polling. Note this is a
Version 2.x feature.

 Overview of synchronization process with an external request to

.. uml::

   @startuml images/06_uc.png
   actor "Coordinating Node" as CN
   actor "Member Node" as MN

   usecase "13. Authorization" as author
   usecase "06. Synchronize Metadata" as SYNC
   usecase "43. Notify Indexer" as NOTIFY
   CN -- SYNC
   MN -- SYNC
   SYNC ..> author: <<includes>>
   SYNC ..> NOTIFY: <<includes>>
   @enduml

@startuml images/06_uc.png
actor "Coordinating Node" as CN
actor "Member Node" as MN

usecase "13. Authorization" as author
usecase "06. Synchronize Metadata" as SYNC
usecase "43. Notify Indexer" as NOTIFY
CN -- SYNC
MN -- SYNC
SYNC ..> author: <<includes>>
SYNC ..> NOTIFY: <<includes>>
@enduml

**Figure 1.** Use case diagram indicating the components and other use cases
involved in this use case.

 Use case diagram indicating the components and other use cases

*Figure 4.* Sequence diagram for use case 06. Synchronize content between
Member and Coordinating Node. Detail for the *processObject()* step provided
in the activity diagram below.

 Sequence diagram for use case 06. Synchronize content between
Member and Coordinating Node. Detail for the 

*Figure 5.* Activity diagram indicating the execution flow after attempting to
retrieve the system metadata for the object from the CN.

 Activity diagram indicating the execution flow after attempting to

@startuml images/06_seq.png

participant "Repl Task Queue" as cn_repl << Cluster >>
participant "Replication" as cn_replication << CN >>
participant "Index Task Queue" as cn_index << Cluster >>
participant "Indexer" as cn_indexer << CN >>
participant "Object Store" as cn_objs << CN >>
participant "System Metadata Map" as cn_sysmeta << Cluster >>
participant "Sync Task Queue" as cn_queue << CN >>
participant "Node Map" as cn_nodes << Cluster >>
participant "Synchronization" as cn_sync << CN >>
participant "Read API" as mn_read << MN >>

'm_crud -> c_notify: notify(session, PID, OBJECT_CREATED)
'c_notify -> cn_queue: addTask(SyncTask, node, PID)
'note right
'  notification triggered by successful
'  create operation on MN.
'end note

'm_rep -> c_notify: setReplicationStatus(token, PID, COMPLETE)
'  notification triggered by completed
'  replication operation on MN.
'end note

group populateSynchronizationQueue
  cn_sync -> cn_nodes: lock(node_id)
  activate cn_sync #D74F57
  note right
    Start of synchronization
    process triggered by quartz
  end note
  activate cn_nodes #D74F57
  cn_nodes --> cn_sync: OK
  cn_sync -> cn_nodes: getLastUpdateTime(node_id)
  note right
    The Node Map is a hash of
    (node_id, Node), use Hazelcast query
    Map.values() passing in SqlPredicate
  end note
  cn_nodes --> cn_sync: startTime
  deactivate cn_nodes

  cn_sync -> mn_read: listObjects(session, startTime, ...)
  activate mn_read #D74F57
  cn_sync <-- mn_read: ObjectList
  deactivate mn_read
  loop for each PID
    cn_sync -> cn_sync: createTask(PID)
  note right
    Each SyncTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on a CN
  end note

    cn_sync -> cn_queue: offer(taskid, SyncTask)
    activate cn_queue #D74F57
    cn_queue --> cn_sync: OK
    deactivate cn_queue
  end
  note right
    adding SyncTasks should fail if PID is already
    in the list and the PID is NOT locked. This enables
    very recent updates to a PID to occur and be
    correctly managed by the overall synchronization
    process.
  end note
  cn_sync -> cn_nodes: setLastUpdateTime(node_id, startTime)

  cn_sync -> cn_nodes: unlock(node_id)
  deactivate cn_sync
end

group processSynchronizationQueue
  cn_sync -> cn_sync: entryAdded(EntryEvent<PID, SyncTask>)
  note left
    Synchronization implements EntryListener,
    monitors the Sync Queue for changes.
  end note
  cn_sync -> cn_queue: poll(timeout)
  activate cn_queue #D74F57
  note right
    Only one CN will win the poll and
    process the SyncTask
  end note
  cn_queue --> cn_sync: SyncTask
  deactivate cn_queue
  cn_sync -> cn_sync: ExecutorService.submit(SyncTask)
  cn_sync -> cn_sysmeta: lock(PID)
  activate cn_sysmeta #D74F57
  group ProcessPID( PID )
    cn_sync -> mn_read: getSystemMetadata(PID)
    mn_read -> cn_sync: SystemMetadata
    cn_sync -> mn_read: get( PID )
    mn_read -> cn_sync: object
    cn_sync -> cn_sync: work
    note right
      Check for new object,
      updates to properties
    end note
  end
  cn_sync -> cn_objs: createOrUpdate(session, PID, object, SystemMetadata)
  activate cn_objs #D74F57
  cn_indexer -> cn_indexer: entryAdded()
  activate cn_indexer #D74F57
  note left
    Indexer implements EntryListener,
    monitors the System Metadata Map
    for inserts, updates, deletes.
  end note
  cn_indexer -> cn_indexer: createTask(PID)
  note left
    Each IndexTask implements Callable
    on the local CN
  end note
  cn_indexer -> cn_index: offer(taskid, IndexTask)
  activate cn_index #D74F57
  cn_objs --> cn_sync: OK
  deactivate cn_objs
  cn_sync -> cn_sysmeta: unlock(PID)
  deactivate cn_sysmeta
  cn_replication -> cn_replication: entryAdded()
  activate cn_replication #D74F57
  note left
  end note
  cn_replication -> cn_replication: createTask(PID)
  note left
    Each ReplTask implements Callable
    on the a CN
  end note
  cn_replication -> cn_repl: offer(taskid, ReplTask)
  activate cn_repl #D74F57
end
deactivate cn_sync
@enduml

@startuml images/06_act.png
(*) --> [processObject()] if "CNRead.getSystemMetadata(PID)" then
  if "Duplicate Object?" then
    -->[Yes] "Duplicate Content"
    --> "Update Replica Info"
    --> "Store System Metadata"
    --> "Notify watchers\nIndex, Replication"
    --> (*)
  else
    -->[No] "Error: Duplicate Identifier"
    --> "Notify MN"
    --> (*)
  endif

else
  ->[FAIL] "New content"
  note right
    The content is unknown to DataONE
    so needs to be examined, its
    location recorded, and optionally
    retrieved and stored on the CN.
  end note
  if "Science Metadata \n or Resource Map?" then
    ->[yes] "Get object from MN"
    --> "Store object on CN"
  else
    -->[no] "Update Replica Info"
  endif
endif
@enduml

Byte stream equivalence of replicated science metadata would require that MNs
record an exact copy of the metadata document received during replication
operations in addition to the content that would be extracted and stored as
part of the normal (existing) operations of a MN. Is this a reasonable
requirement for MNs? Since MNs are required to store a copy of data, it seems
reasonable to assume a copy of the metadata can be stored as well.

It is a requirement (#384) of DataONE that users are able to search the
holdings, and so a mechanism for indexing the content and therefore a
mechanism for specifying how to retrieve attribute values from the different
:term:`science metadata` formats is required. This in turn requires that the
system is able to accurately determine the format of the metadata in order to
utilize the correct parser for extracting the necessary attribute values for
indexing.  Potential resources may be found at:

 formats is required. This in turn requires that the

.. _PRONOM: http://www.nationalarchives.gov.uk/PRONOM/Default.aspx

The following metadata formats are of interest to the DataONE project for the
initial version and are representative of the types of content that will need
to be stored and parsed.

In all cases the descriptive text was retrieved from the URL provided with the
description, and so where there is discrepancy, the referenced location (or
the currently authoritative location) takes precedence.

Types of :term:`science metadata` and their corresponding :attr:`SystemMetdata.ObjectFormat` identifier.

The Dublin Core Metadata Element Set is a vocabulary of fifteen properties for
use in resource description.

The Darwin Core is body of standards. It includes a glossary of terms (in
other contexts these might be called properties, elements, fields, columns,
attributes, or concepts) intended to facilitate the sharing of information
about biological diversity by providing reference definitions, examples, and
commentaries. The Darwin Core is primarily based on taxa, their occurrence in
nature as documented by observations, specimens, and samples, and related
information. Included are documents describing how these terms are managed,
how the set of terms can be extended for new purposes, and how the terms can
be used. The Simple Darwin Core [SIMPLEDWC] is a specification for one
particular way to use the terms - to share data about taxa and their
occurrences in a simply structured way - and is probably what is meant if
someone suggests to "format your data according to the Darwin Core".

The Ecological Metadata Language (EML) is a metadata specification developed
by the ecology discipline and for the ecology discipline. It is based on prior
work done by the Ecological Society of America and associated efforts
(Michener et al., 1997, Ecological Applications). EML is implemented as a
series of XML document types that can by used in a modular and extensible
manner to document ecological data. Each EML module is designed to describe
one logical part of the total metadata that should be included with any
ecological dataset.

The Content Standard for Digital Geospatial Metadata (CSDGM), Vers. 2
(FGDC-STD-001-1998) is the US Federal Metadata standard. The Federal
Geographic Data Committee (FGDC) originally adopted the CSDGM in 1994 and
revised it in 1998. According to Executive Order 12096 all Federal agencies
are ordered to use this standard to document geospatial data created as of
January, 1995. The standard is often referred to as the FGDC Metadata Standard
and has been implemented beyond the federal level with State and local
governments adopting the metadata standard as well.

-bio
(word document available for descriptions, Matt has XSD of FGDCbio)
(Excel spreadsheet listing mapping,
 xslt: EML->FGDC (lossy), FGDC->EML)
(mapping available for EML -> DC (Duane))

The DIF does not compete with other metadata standards. It is simply the
"container" for the metadata elements that are maintained in the IDN database,
where validation for mandatory fields, keywords, personnel, etc. takes place.

The DIF is used to create directory entries which describe a group of data. A
DIF consists of a collection of fields which detail specific information about
the data. Eight fields are required in the DIF; the others expand upon and
clarify the information. Some of the fields are text fields, others require
the use of controlled keywords (sometimes known as "valids").

The DIF allows users of data to understand the contents of a data set and
contains those fields which are necessary for users to decide whether a
particular data set would be useful for their needs.

Mapping to DC available at http://gcmd.nasa.gov/Aboutus/standards/dublin_to_dif.html

ISO 19137:2007 defines a core profile of the spatial schema specified in ISO
19107 that specifies, in accordance with ISO 19106, a minimal set of geometric
elements necessary for the efficient creation of application schemata.

It supports many of the spatial data formats and description languages already
developed and in broad use within several nations or liaison organizations.

The NEXUS file format is a commonly used format for phylogenetic data.
Unfortunately, over time, the format has become overloaded - which has caused
various problems. Meanwhile, new technologies around the XML standard have
emerged. These technologies have the potential to greatly simplify, and
improve robustness, in the processing of phylogenetic data.

The Water Markup Language (WaterML) specification defines an information
exchange schema, which has been used in water data services within the
Hydrologic Information System (HIS) project supported by the U.S. National
Science Foundation, and has been adopted by several federal agencies as a
format for serving hydrologic data. The goal of WaterML was to encode the
semantics of hydrologic observation discovery and retrieval and implement
water data services in a way that is both generic and unambiguous across
different data providers, thus creating the least barriers for adoption by the
hydrologic research community.

ISO 19115 "Geographic Information - Metadata" is a standard of the
International Organization for Standardization (ISO). It is a component of the
series of ISO 191xx standards for Geospatial metadata. ISO 19115 defines how
to describe geographical information and associated services, including
contents, spatial-temporal purchases, data quality, access and rights to use.
The standard defines more than 400 metadata elements and 20 core elements.

The purpose of the ADN (ADEPT/DLESE/NASA) metadata framework is to describe
resources typically used in learning environments (e.g. classroom activities,
lesson plans, modules, visualizations, some datasets) for discovery by the
Earth system education community.

GML profiles are logical restrictions to GML, and may be expressed by a
document, an XML schema or both.

http://www.oceanobs09.net/work/cwp_proposals/docs/100_Hankin_StandardsOceanDataInteroperability_CWPprop.doc

The Data Documentation Initiative is an international effort to establish a
standard for technical documentation describing social science data. A
membership-based Alliance is developing the DDI specification, which is
written in XML.

The MicroArray and Gene Expression (MAGE) provides a standard for the
representation of microarray expression data that would facilitate the
exchange of microarray information between different data systems.

The Earth Science Markup Language (ESML) is a interchange standard that
supports the description of both syntactic (structural) and semantic
information about Earth science data. Semantic tags provide linking of
different domain ontologies to provide a complete machine understandable data
description.

The Cruise Summary Report (CSR), previously known as ROSCOP (Report of
Observations/Samples Collected by Oceanographic Programmes), is an established
international standard designed to gather information about oceanographic
data. ROSCOP was conceived in the late 1960s by the IOC to provide a low level
inventory for tracking oceanographic data collected on Research Vessels.

The ROSCOP form was extensively revised in 1990, and was re-named CSR (Cruise
Summary Report), but the name ROSCOP still persists with many marine
scientists. Most marine disciplines are represented in ROSCOP, including
physical, chemical, and biological oceanography, fisheries, marine
contamination/pollution, and marine meteorology. The ROSCOP database is
maintained by ICES

A metadata specification for representing the contextual and environmental information
associated with marker gene data sets collected in the environment.  The MIENS specification
extends the MIGS/MIMS specification.

The Dryad metadata team has developed a metadata application profile based on
the Dublin Core Metadata Initiative Abstract Model (DCAM) following the Dublin
Core guidelines for application profiles. The Dryad metadata profile is being
developed to conform to the Dublin Core Singapore Framework, a framework
aligning with Semantic Web development and deployment.

Data, in the context of DataONE, is a discrete unit of digital content that is
expected to represent information obtained from some experiment or scientific
study. The :term:`data` is accompanied by :term:`science metadata`, which is a
separate unit of digital content that describes properties of the data. Each
unit of science data or science metadata is accompanied by a :term:`system
metadata` document which contains attributes that describe the digital object
it accompanies (e.g. hash, time stamps, ownership, relationships).

study. The 

, which is a
unit of science data or science metadata is accompanied by a 

 document which contains attributes that describe the digital object

In the initial version of DataONE, science data are treated as opaque sets of
bytes and are stored on :term:`Member Node`\s (MN). A copy of the science
metadata is held by the :term:`Coordinating Node`\s (CN) and is parsed to
extract attributes to assist the discovery process (i.e. users searching for
content).

bytes and are stored on 

s (CN) and is parsed to
content).

\s (CN) and is parsed to
content).

The opaqueness of data in DataONE is likely to change in the future to enable
processing of the data with operations such as translation (e.g. for format
migration), extraction (e.g. for rendering), and merging (e.g. to combine
multiple instances of data that are expressed in different formats). Such
operations rely upon a stable, accessible framework supporting reliable data
access, and so are targeted after the initial requirements of DataONE are met
and the core infrastructure is demonstrably robust.

:doc:`DataPackage` provides a more complete description of data, science metadata,
and system metadata and their relationship to one another.

 provides a more complete description of data, science metadata,

All metadata, including the science metadata and DataONE package metadata MUST
be encoded in the UTF-8 encoding. The DataONE :func:`CN_crud.create` and
:func:`CN_crud.update` methods always expect UTF-8 encoded information, and so
content that contains characters outside of the ASCII character set should be
converted to UTF-8 through an appropriate mechanism before adding to DataONE.

be encoded in the UTF-8 encoding. The DataONE 

 methods always expect UTF-8 encoded information, and so

An original copy of the data is maintained for a long as practicable (ideally,
the original content is never deleted). Derived copies of content, such as
might occur when a new copy of a data object is created to migrate to a
different binary format (e.g. an Excel 1.0 spreadsheet translated to Open
Document Format) always create a new data object that will be noted as an
annotation recorded in the system metadata of the data package.

The data component of a DataONE package is opaque to the DataONE system
(though this may change in the future), and so the endianness of the content
does not affect operations except that it must be preserved. However,
processing modules may utilize content from DataONE and may be sensitive to
the byte ordering of content. As such, the endianness of the data content
should be recorded in the user supplied metadata (the science metadata), and
where not present SHOULD be assumed to be least significant byte first (LSB,
or small-endian).

Describe how endianness is specified in various science metadata
formats.

ISO 2146 (Registry Services for Libraries and Related Organisations) is an
international standard currently under development by ISO TC46 SC4 WG7 to
operate as a framework for building registry services for libraries and
related organizations. It takes the form of an information model that
identifies the objects and data elements needed for the collaborative
construction of registries of all types. It is not bound to any specific
protocol or data schema. The aim is to be as abstract as possible, in order to
facilitate a shared understanding of the common processes involved, across
multiple communities of practice.

Used by the Australian National Data Service (ANDS) for
describing data collections in ANDS, which for many Australian data sets
corresponds to the concept of a 'data set' used here. The term 'collection'
is loosely defined so that different disciplines can apply it appropriately.

See: http://www.nla.gov.au/wgroups/ISO2146/
Schema: http://www.nla.gov.au/wgroups/ISO2146/n198.xsd

A profile of ISO 19115 for Australia.  See:
http://www.osdm.gov.au/ANZLIC_MetadataProfile_v1-1.pdf?ID=303

List and define the minimal set of fields with examples. A starting
point would be the union of the required search properties and the
information required for accurate citation.

Experiment metadata MUST contain a minimal set of fields to be accepted by the
DataONE system.

Define the procedures for content deletion - who is responsible,
procedures for contacting authors, timeliness of response.

This document describes the concept of "data" within the first iteration of
the DataONE system.

Data and science metadata are immutable for the first version of the
DataONE system. As such, resolving the identifiers assigned to the data or the
science metadata will always resolve to the same stream of bytes.

The DataONE :func:`CN_crud.update` method will fail if attempting to modify an
instance of science data.

 method will fail if attempting to modify an

Deletion of content is only available to DataONE administrators (perhaps a
curator role is required?).

- Attribute mapping to the list prepared previously
- Attribute mapping to sysmeta docs
- SOLR examples, specific to Mercury

This document has been DEPRECATED: Please see :doc:`SearchMetadata`

Content here is preserved for notes until the search API is completed.
Synopsis
--------

This document provides an outline for approaches to querying content available
in DataONE through the ``/object/`` collection exposed by the CNs and MNs
(i.e. :func:`MN_replication.listObjects` and :func:`CN_query.search`
methods). The same approach can be applied to the ``/log/`` collection exposed
by the CNs and MNs (i.e. the :func:`CN_query.getLogRecords` and
:func:`MN_crud.getLogRecords` methods).

There are three types of query that can be readily supported by CNs
(name-value pairs, Metacat path query, and Mercury SOLR query), and at least
one by MNs (name-value pairs). There may also be additional query types
specified in the future (e.g. CQL, SPARQL).

Overview
--------

The basic model is that a query applied against a collection acts as a filter,
restricting the results to only those objects whose properties match the
supplied query expression. The default, or unfiltered view of the collection
shows all objects (that the user is authorized to access). The query does not
shape the result, i.e. it does not indicate which fields are returned or the
structure of the response.

There seems to be two basic types of query that need to be supported. One is
querying against fairly distinct and controlled object attributes that are for
the most part, defined by the DataONE system ("system queries"). The other is
for queries that apply to the content of objects that are contributed to
DataONE ("content queries"). In this case, the content, structure, and even
representation is essentially uncontrolled, and so may vary considerably
across the universe of objects that are managed by DataONE.

A longterm goal would be to support a query syntax that is expressive enough
to enable precise discovery of content but also simple enough that at least
common queries can be expressed in a URL.

There are three types of query expression that can be supported easily with
the initial version of the DataONE cyber-infrastructure:

1) Simple name-value pairs combined together with a single logical operator
(e.g. AND).

2) The Path Query syntax / structure that is used by Metacat. This is a
potentially very expressive query that is encoded in an XML structure, and so
can be unwieldy for passing in a URL (POST is typically used) or generation by
hand.

3) The SOLR / Lucene query syntax that is supported by Mercury. Fairly
sophisticated queries can be expressed, but there is no mechanism for querying
against structure (e.g. matching the value of a term that is a child of some
other element). SOLR queries are designed to be transmitted in URLs and are
reasonably simple to create by hand.

The different types of query are described in more detail below.

Since it is feasible that MNs and CNs could support multiple query types, it
is desirable that the client provide a hint about the type of query being
transmitted through a URL parameter such as "``qt``" (query type), with::

  qt=nvp    --> Name, value pairs
  qt=path   --> Metacat path query
  qt=solr   --> SOLR query syntax (used by Mercury)

Simple NV Pairs
---------------

The basic approach here is the use name/value pairs (NVPs) in the URL to
construct a query, with names typically mapping to an attribute + comparison
operator (with comparison operator indicated as a suffix to the attribute),
and values being the value to compare against entries in the database.
Multiple NVPs are combined together with either the logical AND operator or
the logical OR operator. The types of queries that can be expressed are quite
limited, though can be sufficient for restricting results to a portion of a
data set modeled as a flat table.

The primary goal of this query syntax is to enable simple implementation of
range restrictions for collections available on MNs.

An example of how a simple query might express "objects of type data that have
been modified since 6AM on the first of January, 2010 UTC"::

  ../object/?qt=nvp&oclass=data&lastModified_gt=20100101T060000+00

Suggestions for comparison operator suffixes:

======= ===========================
Suffix  Comparison Operator
None    Equals (==) (default)
_eq     Equals (==)
_ne     Not equal (!=)
_lt     Less than (<)
_le     Less than or equals (<=)
_gt     Greater than (>)
_ge     Greater than or equals (>=)

The presence of one or more wildcard characters in the value for an
equivalence operator would invoke the equivalent of a substring search. For
example::

  ../object/?qt=nvp&oclass=d*

could be mapped to the SQL WHERE clause::

  WHERE oclass LIKE 'd%'

The general grammar of the query can be expressed as:

.. productionlist::
   NVPQuery : { `nvpair` }
   nvpair   : `name` + "=" + `value`
   name     : string [+ `operator`]
   operator : "_eq" | "_ne" | "_lt" | "_le" | "_gt" | "_ge"
   value    : string

An alternative approach is to use enumerated triples, so for the same query as
above (with ``a`` referring to "attribute name", ``c`` to "comparison
operator", and ``v`` to "value")::

  ../object/?qt=nvp&a0=oclass&c0=eq&v0=data&
                    a1=lastModified&c1=gt&v1=20100101T060000+00

This approach has an advantage of specifying simple logical operators, e.g.::

  &lop0_1=AND

which would indicate that the logical operator between the first and second
query elements is "AND". This gets messy pretty quickly though when
considering precedence rules.

Metacat Path Query
------------------

.. TODO::
   - Rewrite this section to use the EarthGrid query syntax, which is more
     readable and expresses the same concepts as the pathquery

Metacat is an XML database, and so must support mechanisms for querying not
just the attribute name, but also its location relative to other elements of
the document (similar to XPath). The path query also indicates the elements
that will be returned in the response. An `example path query`_::

  <pathquery version="1.0">
    <meta_file_id>unspecified</meta_file_id>
    <querytitle>unspecified</querytitle>
    <returnfield>dataset/title</returnfield>
    <returnfield>keyword</returnfield>
    <returnfield>originator/individualName/surName</returnfield>
    <returndoctype>eml://ecoinformatics.org/eml-2.0.1</returndoctype>
    <returndoctype>eml://ecoinformatics.org/eml-2.0.0</returndoctype>
    <querygroup operator="UNION">
      <queryterm casesensitive="false" searchmode="contains">
        <value>Plant</value>
        <pathexpr>dataset/title</pathexpr>
      </queryterm>
        <value>plant</value>
        <pathexpr>keyword</pathexpr>
    </querygroup>
  </pathquery>

This query states something like return the field values ``dataset/title``,
``keyword``, and ``originator/individualName/surName`` from documents where
the string "plant" appears in the ``keyword`` attribute or the string "Datos"
appears in the ``dataset/title`` attribute. The comparisons are performed
without consideration of case.

Since path queries are expressed as XML documents, they can get quite large
and so can be unwieldy when sending over a HTTP GET request. However, the
types of queries that can be created can be quite precise and expressive, so
these should be supported by the CN services, which shouldn't involve much
more than passing the query through to the Metacat instance operating as the
document store on the CN.

.. _example path query: https://code.ecoinformatics.org/code/metacat/trunk/docs/user/metacatquery.html

SOLR Query Syntax
-----------------

- http://wiki.apache.org/solr/SolrQuerySyntax

- http://lucene.apache.org/java/2_4_0/queryparsersyntax.html

Query Attributes
----------------

- Best if query attributes were consistent across all the query types

- Distinction between searches against system metadata and science metadata
  (though some overlap of attributes)

- Log searches can probably be pretty simple - just slicing by time

- MNs and CNs should support introspection that lists the supported query
  types and the supported query attributes

Misc Notes

Google visualization api query language: http://code.google.com/apis/visualization/documentation/querylanguage.html

SRU/SRW and CQL: http://www.loc.gov/standards/sru/

OpenSearch: http://www.opensearch.org/Home

XPath: http://www.w3.org/TR/xpath and XQuery: http://www.w3.org/TR/xquery/
(appropriate for querying against a general XML model)

SPARQL (assuming you can express content in an RDF model):
http://www.w3.org/TR/rdf-sparql-query/

TAPIR:
http://www.tdwg.org/dav/subgroups/tapir/1.0/docs/TAPIRSpecification_2008-02-07.html

MetaCat (EarthGRID):
https://code.ecoinformatics.org/code/metacat/trunk/docs/user/metacatquery.html

Possible incomplete section title.
Treating the overline as ordinary text because it's so short.

::
cancelReplication (sess, transID)
listAllObjects ()
listRecords ()
replicationSucceeds (transID)
startReplica (ID, sessionID, transID, srcMN)
sysOp (token)
listChangedRecords(token) -> PIDs

## from v0.3 doc ##
listObjects(token, query) -> list of PIDs
synchronizationComplete(token, PID)
synchronizationFailed(token, PID, problemDescription)

Blank line missing before literal block (after the "::")? Interpreted as a definition list item.

List the records that have changed since the last synchronization event.

token -- A token with authority to retrieve a list of changed records.

This is a callback method used by a CN to indicate to a MN that it has
completed replication of the science metadata identified by PID. When
called, the MN should somehow record that synchronization is complete for
that object so that the next time it reponds to a ``listObjects(token,
query)`` query the object will not be included as requiring
synchronization.

that object so that the next time it reponds to a 

 query the object will not be included as requiring

PID -- The PID of the science metadata object that is now synchronized with the
CN.

The PID of the science metadata object that is now synchronized with the
CN.

A CN could use this method to provide the MN with an updated version of
the system metadata for the object if this method accepted a system
metadata object instead of a simple PID as a parameter.

Exceptions.InvalidIdentifier -- The supplied PID does not identify a science metadata object owned by
the MN.

The supplied PID does not identify a science metadata object owned by
the MN.

Exceptions.InvalidOperation -- The science metadata object identified by the PID is already marked as
synchronized.

The science metadata object identified by the PID is already marked as

Identical to get() except that a list of identifiers is requested.

The issue here is how to serialize the results. There are some conceptual
mechanisms for handling structured markup such as XML, but no consistent
approaches for working with data blobs. Perhaps this method should apply
only to system metadata, for which the serialization can be defined
independently of all the other participating nodes?

token -- (Types.AuthToken) The authentication token returned
by login()

Bytes of the specified objects. How are multiple objects serialized?

Exceptions.NotAuthorized -- The supplied identity does not have READ
permission on the object.

The supplied identity does not have READ

Exceptions.NotFound -- The object specified by PID does not exist.

batchGet specifies that multiple objects may be retrieved in a
single call. How are these objects packaged in the response?

This operation is used by CN applications to obtain a token for system
level operations.

This could be a normal :func:`login` operation using a system user and
secret that is stored in a secure location.

 operation using a system user and

This method is not necessary since the CNs keep track of when an object was
last synchronized. There should not be a requirement for MNs to track if
content has been retrieved by the CNs.

From the capabilities API, this function is listed but has no definition:

This list of requirements for Coordinating Node functionality was developed
during a brainstorming session at the 20090602 ABQ VDC-TWG meeting.

CN can store and search multiple metadata standards
- specifically (list Metadata_formats_of_interest.txt)

xxx concurrent transactions can be supported
- different for different types of transaction e.g. delete vs. insert

CN should ensure that any given data set is available at any time (subject
to policy restrictions) even when xxx% of MNs are offline

CNs must validate that:
- data are available at all replicas
- replicas are valid

CNs ensure that data are available in current formats (data format upgrade?)

CNs should provide services to detect rogue data (viruses, copyright, )

Produced by:\\INTEROP: Creation of an International Virtual Data Center for the Biodiversity,
Ecological and Environmental Sciences (NSF Award 0753138)\\and\\DataNet Full Proposal: DataNetONE (Observation Network for Earth) (NSF Award 0830944)

DataONE Cybersecurity Planning Posture Progression Through Project Lifetime

Requirement 3: Record specific types of relationships between objects

Requirement 6: Record software and hardware specifications for future object rendering

Requirement 7: Record provenance information (e.g., prinicpal, timestamp, event, rights)

Requirement 8: Record information to ensure viability of preserved objects

Requirement 9: Record information to ensure authenticity of preserved objects

391: Enable different classes of users commensurate with their roles.

392: Identity and access control should be interoperable across datanets

393: Access control rule evaluation must be highly scalable and responsive.

761: Users can specify authorization rules for data objects, science metadata objects, and process artifacts separately

762: User identities can be derived from existing institutional directory services

763: Authentication and authorization services are geographically replicated

764: Authentication and access control should be consistently available

766: Users should be able to easily assign proxy privileges to other users and to systems acting on their behalf for limited time durations

768: Need default authz policies that resolve problems associated with inaccessible principals

769: Authorization should support critical roles, such as curators and system administrators

770: Authorization system should be able to express the pseudo-principal concepts like 'public'

771: User identities should have simple string serializations that express both the user identity and namespace from which it is drawn

772: Authentication services should be compatible with existing infrastructure and applications

777: Authorization rules should support common permission levels

External CN APIs (accessible to any caller with appropriate credentials)

queryEngineVersion (v1_1.Types.QueryEngineDescription attribute)

querySchemaVersion (v1_1.Types.QueryEngineDescription attribute)