Referencing Content External to DataONE
=======================================
.. contents::
Here "content external to DataONE" refers to data, metadata, or other
information not accessible directly through the DataONE Member Node or
Coordinating Node APIs.
For example, a researcher may create a data package that contains the usual
data and metadata objects, but would also like to provide a reference to
additional data (the "external data") that can not be retrieved using the
DataONE MNRead.get method.
The external data is not synchronized by DataONE.
There are two mechanisms for showing references to external content in the the
DataONE Search UI, both rely on information stored in Science Metadata:
1. During rendering of metadata by stylesheet transformation.
Some metadata formats have a mechanisms to reference arbitrary links to
information outsie of the data package.
2. By reference to a Service through the ``service*`` index fields
DataONE provides a mechanism where services related to a data package may be
described within a metadata document contained in the data package. A
reference to external content can be considered a simple type of service
(e.g. a HTTP GET).
The mechanisms for post processing allow for more control over the expression
and reliability of such references through the service oriented metadata as
opposed to a direct reference to an arbitrary location with poorly defined
characteristics.
Content creators are thus encouraged to leverage the servie description
mechanism when there is a need to reference content external to DataONE from a
data package.
Index Fields
------------
The solr index in DataONE acts as a common representation of metadata for
synchronized content. Content of the various formats of Science Metadata is
mapped to index fields using various parsing and processing rules.
The index fields relevant to external content are:
=================== ===================================================
Field Description
=================== ===================================================
isService Set to true if document is a member node service
description document. Use to filter search results for to
exclude or include member node services.
serviceTitle A brief, human readable descriptive title for the member
node service.
serviceDescription A human readable description of the member node service to
assist discovery and to evaluate applicability.
serviceType The type of service being provided by the member node.
serviceCoupling One of 'tight', 'mixed', or 'loose'. Tight coupled service
work only on the data described by this metadata document.
Loose coupling means service works on any data. Mixed
coupling means service works on data described by this
metadata document but may work on other data.
serviceEndpoint A URL that indicates how to access the member node service.
serviceInput Aspect of the service that accepts a digital entity. Either
a list of DataONE formatId URLs or PID RESOLVE URLs that the
member node service operates on. A pid RESOLVE URL
indicates a 'tight' coupled service - while a list of
formatIds indicates a loose coupled service.
serviceOutput Aspect of the service that provides a digital entity
resulting from operation of the service. A listing of
DataONE formatId which this member node service produces.
=================== ===================================================
Use of Index Fields in Search UI
--------------------------------
The DataONE `Search UI`_ uses the search index to populate user interface
elements such as the data package view, which is shown when viewing a specific
data package. For example:
https://search.dataone.org/#view/{1BDC13BA-A8C2-4787-8B77-4EB04AE6B416}
Shows a table for "Alternate Data Access" which contains columns:
============= =======================
Column Index Field
============= =======================
Name serviceTitle
Description serviceDescription
Access Type serviceType
URL serviceEndpoint
============= =======================
The Alternate Data Access table is shown if the index field ``isService`` is
true.
.. Search UI: https://search.dataone.org/
Appearance of External Content in the Search UI
-----------------------------------------------
.. TODO::
Mapping ISO-TC211 to Index Fields for Services
----------------------------------------------
The mapping from ISO-TC211 to the index fields is described at the generated
`index documentation`_. An excerpt is repeated here with additional comments.
(See application-context-isotc211-base.xml in the d1_index_task_processor
project)
In ISO 19119, services may be tightly or loosely-coupled to data they operate
on and sit under the srv:SV_ServiceIdentification element. Or they may be
limited to tightly-coupled distribution info and sit under the
gmd:distributionInfo element.
The solr fields may be populated either with one expression checking and/or
concatenating both the srv:srv:SV_ServiceIdentification and
gmd:distributionInfo locations (for example: isotc.isService or
isotc.serviceCoupling)
Or there may be 2 separate expressions for the different scenarios that affect
the same field. (for example: sotc.serviceEndpoint and
isotc.distribServiceEndpoint).
Two expressions are only used for multivalue SolrFields; this way both results
are added - both srv:SV_ServiceIdentification and gmd:distributionInfo
subelements are indexed).
.. index documentation: http://indexer-documentation.readthedocs.io/en/latest/generated/proc_isotc211NoaaSubprocessor.html
isService
~~~~~~~~~
Checks for existence of either srv service description OR distribution
"service" info.
::
boolean(//srv:SV_ServiceIdentification or
//gmd:distributionInfo/gmd:MD_Distribution)
serviceCoupling
~~~~~~~~~~~~~~~
The srv location can explicitly set this and if set will override
distributionInfo.
The serviceCoupling will set:
* 'loose' coupling if srv:SV_CouplingType is loose
* 'tight' coupling if srv:SV_CouplingType is tight
* 'tight' coupling if distribution service info exists and srv:SV_CouplingType
doesn't / is unspecified, empty if neither exists
::
concat(
substring (
'loose',
1 div boolean( //srv:SV_ServiceIdentification
/srv:couplingType
/srv:SV_CouplingType
/@codeListValue = 'loose' )
),
substring (
'tight',
1 div boolean( //srv:SV_ServiceIdentification
/srv:couplingType
/srv:SV_CouplingType
/@codeListValue = 'tight' )
),
substring(
'tight',
1 div boolean( //gmd:distributionInfo
/gmd:MD_Distribution
and not( //srv:SV_ServiceIdentification
/srv:couplingType
/srv:SV_CouplingType
/@codeListValue )
)
),
substring(
'',
1 div boolean(
not( //srv:SV_ServiceIdentification
/srv:couplingType
/srv:SV_CouplingType
/@codeListValue )
and not( //gmd:distributionInfo
/gmd:MD_Distribution )
)
)
)
serviceTitle
~~~~~~~~~~~~
This combines the srv service title with the distribution "service" info's
titles.
::
( //srv:SV_ServiceIdentification
/gmd:citation
/gmd:CI_Citation
/gmd:title
/gco:CharacterString
|
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorTransferOptions
/gmd:MD_DigitalTransferOptions
/gmd:onLine
/gmd:CI_OnlineResource
/gmd:name
/gco:CharacterString
)/text()
serviceDescription
~~~~~~~~~~~~~~~~~~
::
( //srv:SV_ServiceIdentification
/gmd:abstract
/gco:CharacterString
|
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorTransferOptions
/gmd:MD_DigitalTransferOptions
/gmd:onLine
/gmd:CI_OnlineResource
/gmd:description
/gco:CharacterString
)/text()
serviceType
~~~~~~~~~~~
Both are evaluated / indexed, checking the srv and distributionInfo locations
for a service type.
::
//srv:SV_ServiceIdentification
/srv:serviceType
/gco:LocalName
/text()
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorTransferOptions
/gmd:MD_DigitalTransferOptions
/gmd:onLine
/gmd:CI_OnlineResource
/gmd:protocol
/gco:CharacterString
/text()
serviceEndpoint
~~~~~~~~~~~~~~~
Both are evaluated / indexed, checking the srv and distributionInfo locations
for service endpoints.
::
//srv:SV_ServiceIdentification
/srv:containsOperations
/srv:SV_OperationMetadata
/srv:connectPoint
/gmd:CI_OnlineResource
/gmd:linkage
/gmd:URL
/text()
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorTransferOptions
/gmd:MD_DigitalTransferOptions
/gmd:onLine
/gmd:CI_OnlineResource
/gmd:linkage/gmd:URL/text()
|
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:transferOptions
/gmd:MD_DigitalTransferOptions
/gmd:onLine
/gmd:CI_OnlineResource
/gmd:linkage
/gmd:URL
/text()
serviceInput
~~~~~~~~~~~~
Both are evaluated / indexed, checking the srv and distributionInfo locations
for service input.
::
//srv:SV_ServiceIdentification
/srv:operatesOn
/@xlink:href
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorTransferOptions
/@xlink:href
serviceOutput
~~~~~~~~~~~~~
Both are evaluated / indexed, checking the srv and distributionInfo locations
for service output.
::
//srv:SV_ServiceIdentification
/gmd:resourceFormat
/@xlink:href
//gmd:distributionInfo
/gmd:MD_Distribution
/gmd:distributor
/gmd:MD_Distributor
/gmd:distributorFormat
/gmd:MD_Format
/gmd:version
/gco:CharacterString
/text()
Mapping of EML to Index Fields for Services
-------------------------------------------
(See application-context-eml-base.xml)
The EML spec is limited in what info it can hold about services.
EML holds no elements that correspond to these fields::
ServiceType
SerivceInput
ServiceOutput
ServiceCoupling
So info about these can't be indexed.
Supported fields are below:
isService
~~~~~~~~~
Checks for the presence of a distribution url.
::
boolean(//software/implementation/distribution/online/url)
ServiceTitle
~~~~~~~~~~~~
Fetches the software title.
::
//software/title//text()[normalize-space()]
ServiceDescription
~~~~~~~~~~~~~~~~~~
Fetches the software abstract.
::
//software/abstract//text()[normalize-space()]
ServiceEndpoint
~~~~~~~~~~~~~~~
Fetches the distribution url.
::
//software/implementation/distribution/online/url/text()