<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Replication Notes — v2.1.0-beta</title>
<link rel="stylesheet" href="../_static/dataone.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '2.1.0-beta',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/mathjax_pre.js"></script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-MML-AM_CHTML"></script>
<script type="text/javascript" src="../_static/sidebar.js"></script>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Notes for MN API" href="MN_api_notes_comments.html" />
<link rel="prev" title="Cross Domain Indexing and Access for Data and Metadata" href="DataAndMetadata.html" />
<link media="only screen and (max-device-width: 480px)" href="../_static/small_dataone.css" type= "text/css" rel="stylesheet" />
</head>
<body role="document">
<div class="version_notice">
<p>
<span class='bold'>Warning:</span> These documents are under active
development and subject to change (version 2.1.0-beta).<br />
The latest release documents are at:
<a href="https://purl.dataone.org/architecture">https://purl.dataone.org/architecture</a>
</p>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="MN_api_notes_comments.html" title="Notes for MN API"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="DataAndMetadata.html" title="Cross Domain Indexing and Access for Data and Metadata"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html"></a> »</li>
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">General Design and Implementation Notes</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="replication-notes">
<h1>Replication Notes<a class="headerlink" href="#replication-notes" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Author:</th><td class="field-body">Dave Vieglas</td>
</tr>
</tbody>
</table>
<p>This document contains notes about replication that may be of some use in
DataONE.</p>
<div class="section" id="general">
<h2>General<a class="headerlink" href="#general" title="Permalink to this headline">¶</a></h2>
<blockquote>
<div><ol class="arabic simple">
<li>Reduce complex transactions to be written by a single server</li>
<li>Monitor the server log</li>
<li>Be prepared for failure. The more data is distributed and scaled, the
more likely failure will be encountered.</li>
</ol>
</div></blockquote>
</div>
<div class="section" id="coordinating-nodes">
<h2>Coordinating Nodes<a class="headerlink" href="#coordinating-nodes" title="Permalink to this headline">¶</a></h2>
<p>CNs need to keep a copy of all science metadata, and it is expected that
each CN will be replicated with the others, so that each will appear the same.</p>
<p>There will be three CNs to start with.</p>
<p>There’s basically three models for dealing with replication in this case:</p>
<blockquote>
<div><ul class="simple">
<li>single master, two slaves</li>
<li>ring - write one, it writes to the next, and so on</li>
<li>multi-master. Write to any, replication to others</li>
</ul>
</div></blockquote>
<p>Multi-master replication would be the ideal option, as each CN would then
offer identical functionality rather than the alternative of redirecting
(presumably) 2/3 of requests to the master node.</p>
<p>There are basically two categories of information that are replicated:</p>
<blockquote>
<div><ul class="simple">
<li>science metadata, which is information about a data set</li>
<li>system metadata, which is information that is used to keep track of things
such as the science metadata, MN capabilities and state, perhaps groups
and access control lists, identity mappings</li>
</ul>
</div></blockquote>
</div>
<div class="section" id="member-nodes">
<h2>Member Nodes<a class="headerlink" href="#member-nodes" title="Permalink to this headline">¶</a></h2>
<p>Replication of data is achieved by copying content between Member Nodes until
sufficient copies have been made to address concerns of access (x% of MNs may
be offline and the content can still be reached) and efficiency of access
(Internet distance between client and the MN).</p>
</div>
<div class="section" id="resources">
<h2>Resources<a class="headerlink" href="#resources" title="Permalink to this headline">¶</a></h2>
<blockquote>
<div><ul class="simple">
<li>rcs - venerable, functional</li>
<li>inotify - the linux kernel file system monitor</li>
<li>LDAP multi-master (OpenLDAP, 389 Directory Server)</li>
<li>MySQL multi-master</li>
<li>Microsoft’s File System Replication Service and/or Distributed File System</li>
<li>slony1 for Postgresql</li>
<li><a class="reference external" href="http://www.drbd.org/home/what-is-drbd/">DRBD</a> (dual master)</li>
<li><a class="reference external" href="http://git-scm.com/">Git</a></li>
<li><a class="reference external" href="http://mercurial.selenic.com/">Mercurial</a></li>
<li><a class="reference external" href="http://bazaar-vcs.org/en/">Bazaar</a></li>
<li><a class="reference external" href="http://knb.ecoinformatics.org/software/metacat/">Metacat</a></li>
<li><a class="reference external" href="http://hadoop.apache.org/zookeeper">Zookeeper</a></li>
</ul>
</div></blockquote>
</div>
<div class="section" id="coordinating-node-object-store">
<h2>Coordinating Node Object Store<a class="headerlink" href="#coordinating-node-object-store" title="Permalink to this headline">¶</a></h2>
<p>The CN object store (CN-OS) is responsible for holding a copy of all science
and system metadata, enabling retrieval of objects by an identifier and for
ensuring that the local copy is synchronized with the other CNs. The CN-OS is
analogous to a replicated dictionary with the identifiers being keys and the
science or system metadata documents being the values.</p>
<p>The CN-OS should support operations:</p>
<ul class="simple">
<li>Create(pid)</li>
<li>Read(pid)</li>
<li>Update(pid)</li>
<li>Delete(pid)</li>
<li>List(start, stop, restriction)</li>
</ul>
</div>
<div class="section" id="metacat-as-replicating-object-manager">
<h2>Metacat As Replicating Object Manager<a class="headerlink" href="#metacat-as-replicating-object-manager" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="http://knb.ecoinformatics.org/software/metacat/">Metacat</a> is a flexible metadata database that utilizes XML as a common syntax
for representing various types of metadata. In general, Metacat can be
considered functionally equivalent to an XML database.</p>
<p>Metacat supports replication of documents between instances, with replication
being triggered by several events:</p>
<ul class="simple">
<li>Timer, at pre-specified intervals</li>
<li>Inserts, when a document is inserted locally</li>
<li>Updates, when a document is updated locally</li>
<li>Lock, when a lock is requested on a document (replication after lock received)</li>
</ul>
<p>With Metacat filling the role of the Coordinating Node Object Store, each CN
Metacat instance would replicate with the other instances on the CNs. The
configuration for DataONE CNs would be something like:</p>
<table border="1" class="docutils">
<colgroup>
<col width="12%" />
<col width="24%" />
<col width="22%" />
<col width="32%" />
<col width="10%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">host</th>
<th class="head">server</th>
<th class="head">replicate</th>
<th class="head">datareplicate</th>
<th class="head">hub</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>cn1</td>
<td>localhost</td>
<td>0</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-odd"><td>.</td>
<td>cn2</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-even"><td>.</td>
<td>cn3</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-odd"><td>cn2</td>
<td>localhost</td>
<td>0</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-even"><td>.</td>
<td>cn1</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-odd"><td>.</td>
<td>cn3</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-even"><td>cn3</td>
<td>localhost</td>
<td>0</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-odd"><td>.</td>
<td>cn1</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr class="row-even"><td>.</td>
<td>cn2</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="using-a-distributed-lock-manager">
<h2>Using a Distributed Lock Manager<a class="headerlink" href="#using-a-distributed-lock-manager" title="Permalink to this headline">¶</a></h2>
<p>Apache <a class="reference external" href="http://hadoop.apache.org/zookeeper">Zookeeper</a> can be used as a distributed lock manager to assist with
synchronization of records, especially across the CNs. Zookeeper is
interesting because it can potentially scale to many more than three CNs.</p>
<p>In this case we would create a globally synchronous lock on a specific PID
(essentially the same as the Lock operation of Metacat). The basic procedure
is described in the <a class="reference external" href="http://hadoop.apache.org/zookeeper/docs/r3.1.1/recipes.html#sc_recipes_Locks">Zookeeper recipes</a>.</p>
</div>
<div class="section" id="distributed-execution-environment">
<h2>Distributed Execution Environment<a class="headerlink" href="#distributed-execution-environment" title="Permalink to this headline">¶</a></h2>
<p>For example, <a class="reference external" href="http://www.hazelcast.com/">hazelcast</a>. These types of solutions are typically developed with
LAN clusters in mind, and may run into problems with higher latency networks.
Interesting product though.</p>
</div>
<div class="section" id="file-based-object-store-with-distributed-revision-control">
<h2>File Based Object Store with Distributed Revision Control<a class="headerlink" href="#file-based-object-store-with-distributed-revision-control" title="Permalink to this headline">¶</a></h2>
<p>These notes describe a strategy of utilizing the Linux file system and a
<a class="reference external" href="http://en.wikipedia.org/wiki/Distributed_revision_control">distributed version control</a> system (DVC) such as <a class="reference external" href="http://git-scm.com/">Git</a> or <a class="reference external" href="http://mercurial.selenic.com/">Mercurial</a> as the
science and system metadata store for the Coordinating Nodes.</p>
<p>The basic approach is to treat all primary instances of science and system
metadata as files that are stored on the Linux file system. The files are
located in one or more folders that are under revision control using the DVC.
As a new metadata file is added to a CN, it is replicated to the other CNs on
a scheduled basis (e.g. every 10 minutes or when n files have been changed or
added). Change notification hooks (e.g. commit scripts) are used to update
local databases that are used to search science metadata (e.g. the Mercury
index) or the local system metadata database.</p>
<div class="section" id="scenario-1-adding-a-new-document">
<h3>Scenario 1: Adding a new document<a class="headerlink" href="#scenario-1-adding-a-new-document" title="Permalink to this headline">¶</a></h3>
<ol class="arabic simple">
<li>A new document (d) is retrieved from a MN (or is pushed to the CN by the
MN).</li>
<li>The document is added to the file system and added to version control.</li>
<li>In response to some trigger (time or number of commits), the DVC initiates
a sync with other repositories.</li>
</ol>
</div>
<div class="section" id="advantages">
<h3>Advantages<a class="headerlink" href="#advantages" title="Permalink to this headline">¶</a></h3>
<ul class="simple">
<li>Very simple replication mechanism</li>
<li>Scalable, especially through partitioning of the file system</li>
<li>No global write lock necessary</li>
<li>Agnostic with respect to the file types</li>
<li>Many existing tools</li>
<li>Changes replicated through diffs</li>
<li>Enable segmentation of all content through a simple taxonomy</li>
<li>General approach is RCS agnostic - could use Git, Bazaar or any other
distributed RCS system</li>
<li>Change detection is trivial</li>
</ul>
</div>
<div class="section" id="disadvantages">
<h3>Disadvantages<a class="headerlink" href="#disadvantages" title="Permalink to this headline">¶</a></h3>
<ul class="simple">
<li>File replication only, need to implement mechanisms for search etc</li>
<li>No per-object access control</li>
<li>Potential for collisions for object instances</li>
<li>How to (automatically) resolve conflicts?</li>
<li>Limitations on the absolute number of files per folder/filesystem</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="http://dataone.org">
<img class="logo" src="../_static/dataone_logo.png" alt="Logo"/>
</a></p>
<h3><a href="../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Replication Notes</a><ul>
<li><a class="reference internal" href="#general">General</a></li>
<li><a class="reference internal" href="#coordinating-nodes">Coordinating Nodes</a></li>
<li><a class="reference internal" href="#member-nodes">Member Nodes</a></li>
<li><a class="reference internal" href="#resources">Resources</a></li>
<li><a class="reference internal" href="#coordinating-node-object-store">Coordinating Node Object Store</a></li>
<li><a class="reference internal" href="#metacat-as-replicating-object-manager">Metacat As Replicating Object Manager</a></li>
<li><a class="reference internal" href="#using-a-distributed-lock-manager">Using a Distributed Lock Manager</a></li>
<li><a class="reference internal" href="#distributed-execution-environment">Distributed Execution Environment</a></li>
<li><a class="reference internal" href="#file-based-object-store-with-distributed-revision-control">File Based Object Store with Distributed Revision Control</a><ul>
<li><a class="reference internal" href="#scenario-1-adding-a-new-document">Scenario 1: Adding a new document</a></li>
<li><a class="reference internal" href="#advantages">Advantages</a></li>
<li><a class="reference internal" href="#disadvantages">Disadvantages</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h3>Related Topics</h3>
<ul>
<li><a href="../index.html">Documentation Overview</a><ul>
<li><a href="index.html">General Design and Implementation Notes</a><ul>
<li>Previous: <a href="DataAndMetadata.html" title="previous chapter">Cross Domain Indexing and Access for Data and Metadata</a></li>
<li>Next: <a href="MN_api_notes_comments.html" title="next chapter">Notes for MN API</a></li>
</ul></li>
</ul></li>
</ul>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
<div id="copyright">
© Copyright <a href="http://www.dataone.org">2009-2017, DataONE</a>.
[ <a href="../_sources/notes/Replication.txt"
rel="nofollow">Page Source</a> |
<a href='https://redmine.dataone.org/projects/d1/repository/changes/documents/Projects/cicore/architecture/api-documentation/source/notes/Replication.txt'
rel="nofollow">Revision History</a> ]
</div>
<div id="acknowledgement">
<p>This material is based upon work supported by the National Science Foundation
under Grant Numbers <a href="http://www.nsf.gov/awardsearch/showAward?AWD_ID=0830944">083094</a> and <a href="http://www.nsf.gov/awardsearch/showAward?AWD_ID=1430508">1430508</a>.</p>
<p>Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the National Science Foundation.</p>
</div>
</div>
<!--
<hr />
<div id="HCB_comment_box"><a href="http://www.htmlcommentbox.com">HTML Comment Box</a> is loading comments...</div>
<link rel="stylesheet" type="text/css" href="_static/skin.css" />
<script type="text/javascript" language="javascript" id="hcb">
/*<! -*/
(function()
{s=document.createElement("script");
s.setAttribute("type","text/javascript");
s.setAttribute("src", "http://www.htmlcommentbox.com/jread?page="+escape((typeof hcb_user !== "undefined" && hcb_user.PAGE)||(""+window.location)).replace("+","%2B")+"&mod=%241%24wq1rdBcg%24Gg8J5iYSHJWwAJtlYu/yU."+"&opts=21407&num=10");
if (typeof s!="undefined") document.getElementsByTagName("head")[0].appendChild(s);})();
/* ->*/
</script>
-->
</body>
</html>