<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Integration Testing — v2.1.0-beta</title> <link rel="stylesheet" href="../_static/dataone.css" type="text/css" /> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: '../', VERSION: '2.1.0-beta', COLLAPSE_INDEX: false, FILE_SUFFIX: '.html', HAS_SOURCE: true, SOURCELINK_SUFFIX: '.txt' }; </script> <script type="text/javascript" src="../_static/mathjax_pre.js"></script> <script type="text/javascript" src="../_static/jquery.js"></script> <script type="text/javascript" src="../_static/underscore.js"></script> <script type="text/javascript" src="../_static/doctools.js"></script> <script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-MML-AM_CHTML"></script> <script type="text/javascript" src="../_static/sidebar.js"></script> <link rel="author" title="About these documents" href="../about.html" /> <link rel="index" title="Index" href="../genindex.html" /> <link rel="search" title="Search" href="../search.html" /> <link rel="next" title="Infrastructure Versions" href="versions.html" /> <link rel="prev" title="Implementation Priority" href="prioritization.html" /> <link media="only screen and (max-device-width: 480px)" href="../_static/small_dataone.css" type= "text/css" rel="stylesheet" /> </head> <body role="document"> <div class="version_notice"> <p> <span class='bold'>Warning:</span> These documents are under active development and subject to change (version 2.1.0-beta).<br /> The latest release documents are at: <a href="https://purl.dataone.org/architecture">https://purl.dataone.org/architecture</a> </p> </div> <div class="related" role="navigation" aria-label="related navigation"> <h3>Navigation</h3> <ul> <li class="right" style="margin-right: 10px"> <a href="../genindex.html" title="General Index" accesskey="I">index</a></li> <li class="right" > <a href="../py-modindex.html" title="Python Module Index" >modules</a> |</li> <li class="right" > <a href="versions.html" title="Infrastructure Versions" accesskey="N">next</a> |</li> <li class="right" > <a href="prioritization.html" title="Implementation Priority" accesskey="P">previous</a> |</li> <li class="nav-item nav-item-0"><a href="../index.html"></a> »</li> <li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Implementation Notes</a> »</li> </ul> </div> <div class="document"> <div class="documentwrapper"> <div class="bodywrapper"> <div class="body"> <div class="section" id="integration-testing"> <h1>Integration Testing<a class="headerlink" href="#integration-testing" title="Permalink to this headline">¶</a></h1> <p>While each of the individual components of the DataONE system are designed and tested independently, we also need to design effective tests of the full system functionality. To fulfill many of the use cases, many of the components in the DataONE infrastructure need to interact in a sequenced manner with particular inputs and outputs. The integration tests are designed to allow individual components to be exercised in these particular use case scenarios with known inputs in order to verify that the correct output is produced, and that adequate performance, reliability, and scalability metrics can be met.</p> <div class="section" id="test-framework"> <h2>Test framework<a class="headerlink" href="#test-framework" title="Permalink to this headline">¶</a></h2> <p>In order to run these tests effectively, we need a testing framework that can be used to encode and execute the tests in an orderly fashion and to capture the results of the tests over time. This will allow us to trace changes in the test results over time. It would be useful to provide a synopsis view of all of the tests, showing which are passing and which are failing. This testing framework must be able to execute processes across several different languages, allow a multitude of clients to be configured to load test Member Nodes, and allow a multitude of Member Nodes to be configured to load test Coordinating Nodes.</p> </div> <div class="section" id="list-of-integration-tests"> <h2>List of integration tests<a class="headerlink" href="#list-of-integration-tests" title="Permalink to this headline">¶</a></h2> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">This list needs to be segmented to show what tests need to be passed by July 31st, versus what tests can be deferred to later.</p> </div> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">The metrics need to be parts of the tests. Are there things in the metrics that we’re not testing?</p> </div> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">Codes: Integration tests were ranked as High (H) or Low (L) priority, where each person had 10 high and 10 low votes to assign. Unranked tests were considered Medium (M) priority.</p> </div> <table border="1" class="docutils" id="id1"> <caption><span class="caption-text">High-level integration tests with prioritized votes</span><a class="headerlink" href="#id1" title="Permalink to this table">¶</a></caption> <colgroup> <col width="3%" /> <col width="3%" /> <col width="3%" /> <col width="3%" /> <col width="88%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">#</th> <th class="head">High</th> <th class="head">Due July 31?</th> <th class="head">Low</th> <th class="head">Description</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td>1</td> <td>8</td> <td>July 31</td> <td></td> <td>Use case 36 (resolve)</td> </tr> <tr class="row-odd"><td>2</td> <td>8</td> <td>July 31</td> <td></td> <td>Use case 2 (query)</td> </tr> <tr class="row-even"><td>3</td> <td>8</td> <td>July 31</td> <td></td> <td>Completing the loop: publish data set, be sure it is retrievable exactly as submitted</td> </tr> <tr class="row-odd"><td>4</td> <td>6</td> <td>July 31</td> <td></td> <td>Use case 1 (get). Note: need to test for non-existant ID’s, test access control, test for malicious content</td> </tr> <tr class="row-even"><td>5</td> <td>6</td> <td></td> <td></td> <td>Can a downed CN be revived/repopulated?</td> </tr> <tr class="row-odd"><td>6</td> <td>6</td> <td></td> <td></td> <td>Testing for invalid input and known problems such as XSS and SQL Injection</td> </tr> <tr class="row-even"><td>7</td> <td>5</td> <td></td> <td></td> <td>Use case 6 (synchronize). Note: what is needed for reschronize and recovery from outage.</td> </tr> <tr class="row-odd"><td>8</td> <td>5</td> <td>July 31</td> <td>1</td> <td>Do the Java Stacks and Python stacks return the same thing for the same object</td> </tr> <tr class="row-even"><td>9</td> <td>4</td> <td>July 31</td> <td></td> <td>Does CN metadata replication work</td> </tr> <tr class="row-odd"><td>10</td> <td>3</td> <td></td> <td></td> <td>Broadly, test node failures, including CN and MN. Mostly concerned with read access at this point. MN1 submits data, gets replicated to MN2, MN1 goes down, test access to data even in the absence of the owning Member Node.</td> </tr> <tr class="row-even"><td>11</td> <td>2</td> <td></td> <td></td> <td>Use case 16 (log CRUD operations)</td> </tr> <tr class="row-odd"><td>12</td> <td>2</td> <td></td> <td></td> <td>Test that invalid documents are properly logged on harvest and that logs are machine parseable</td> </tr> <tr class="row-even"><td>13</td> <td>2</td> <td>July 31</td> <td></td> <td>Consistency Check on CN’s ( Test that the checksums at CN’s match)</td> </tr> <tr class="row-odd"><td>14</td> <td>2</td> <td></td> <td></td> <td>Use case 9 (MN-MN replication): includes testing replication policies</td> </tr> <tr class="row-even"><td>15</td> <td>2</td> <td></td> <td></td> <td><p class="first">Load testing to determime the break points (how far can we scale).</p> <p>Relates to risk register issue that CN’s don’t scale to handle the load.</p> <p class="last">What is the target that we should be able to handle. Suggestion is that we want to be able to handle 100 Member Node traffic. This may not be a big issue until we get to handling high frequency updates (e.g. sensor data). Thought: look at the milestone 3 metrics and test at 5x that number.</p> </td> </tr> <tr class="row-odd"><td>16</td> <td>1</td> <td></td> <td></td> <td>Does an insertion via a MN replication request work?</td> </tr> <tr class="row-even"><td>17</td> <td>1</td> <td></td> <td></td> <td>Serve up data from a replicated MN when home MN is inavailable</td> </tr> <tr class="row-odd"><td>18</td> <td>1</td> <td></td> <td></td> <td>Data and Metadata re-synchronization when MN and CN come back from outage</td> </tr> <tr class="row-even"><td>19</td> <td>1</td> <td>July 31</td> <td></td> <td>Testing authentication and access control. Includes testing actions that should be rejected.</td> </tr> <tr class="row-odd"><td>20</td> <td>1</td> <td></td> <td></td> <td>Testing that system metadata is properly validated. Resilience against a “loco” node with malformed metadata (both operator error and malicious users)</td> </tr> <tr class="row-even"><td>21</td> <td></td> <td></td> <td></td> <td>Use case 12 (User Authentication)</td> </tr> <tr class="row-odd"><td>22</td> <td></td> <td></td> <td></td> <td>Use case 13 (User authorization)</td> </tr> <tr class="row-even"><td>23</td> <td></td> <td></td> <td></td> <td>Use case 14 (system authenication and authorization)</td> </tr> <tr class="row-odd"><td>24</td> <td></td> <td></td> <td></td> <td>Integration tests across multiple Member Nodes and Coordinating Nodes. For example, inserting documents at the same time but in different places to ensure it works and we don’t have deadlocks showing up.</td> </tr> <tr class="row-even"><td>25</td> <td>1</td> <td></td> <td>1</td> <td>Threshold tests for heartbeats and other status testing.</td> </tr> <tr class="row-odd"><td>26</td> <td>1</td> <td></td> <td>1</td> <td>General penetration testing (WebInspect as a specific example)</td> </tr> <tr class="row-even"><td>27</td> <td></td> <td></td> <td>1</td> <td>Use case 17 (CRUD logs aggregated at CN) Not impmemented yet</td> </tr> <tr class="row-odd"><td>28</td> <td></td> <td></td> <td>1</td> <td>Use case 24 (MN and CN’s support transactions)</td> </tr> <tr class="row-even"><td>29</td> <td></td> <td></td> <td>1</td> <td>Does a MN data/metadata insertion prevent a race condition, even if insertion deviuosly tries to trigger a race condition</td> </tr> <tr class="row-odd"><td>30</td> <td>2</td> <td></td> <td>2</td> <td>Test that the ITK libraries work properly on all supported (advertised) OS platforms, hardware, and software combinations</td> </tr> <tr class="row-even"><td>31</td> <td>1</td> <td></td> <td>2</td> <td>Race condition: What happens if two MN’s submit the same thing?</td> </tr> <tr class="row-odd"><td>32</td> <td>1</td> <td></td> <td>3</td> <td>Testing the logging and auditing functions. Ditto on the reporting functions. There should be unit tests for this within the software stacks, but this needs to verify that, for example, a Member Node action is logged at the MN and at the CN.</td> </tr> <tr class="row-even"><td>33</td> <td></td> <td></td> <td>3</td> <td>Performance testing (particularly as a function of load)</td> </tr> <tr class="row-odd"><td>34</td> <td></td> <td></td> <td>3</td> <td>Accessibility – 508 compliance as a floor, but going beyond this to appropriate level of accessibility.</td> </tr> <tr class="row-even"><td>35</td> <td></td> <td></td> <td>3</td> <td>Test replication policies. (duplicative test of above?) Side issue: unit testing to see if replication policy is inconsistent. More an issue for rule-based replication policies. Precedence of operations may fix this, but what happens when the replication policy resolves to zero replication nodes allowed (or replication nodes less than required number of replication nodes)?</td> </tr> <tr class="row-odd"><td>36</td> <td></td> <td></td> <td>3</td> <td>Usability testing (generally) – push off to a different category from Integration testing.</td> </tr> <tr class="row-even"><td>37</td> <td></td> <td></td> <td>4</td> <td>Response time test (both API and GUI). Relates back to performance testing.</td> </tr> <tr class="row-odd"><td>38</td> <td></td> <td></td> <td>4</td> <td>Testing validity of science metadata (compliance with standards). See issue in parking lot.</td> </tr> <tr class="row-even"><td>39</td> <td></td> <td></td> <td>4</td> <td>Use case 3 (register – manual operation)</td> </tr> <tr class="row-odd"><td>40</td> <td></td> <td></td> <td>4</td> <td>Deployment testing – can we test that dependancies have been resolved, perhaps a particular issue for the ITK, but also for the various Member Node stacks.</td> </tr> <tr class="row-even"><td>41</td> <td></td> <td></td> <td>5</td> <td>Test that the libraries can check for dependancies, so that when the library intializes it checks that the necessary dependancies have been installed. Some of this is handled with egg installations. ??what happens if something changes after the installation? Library needs to manage these issues, and emit appropriate error codes.</td> </tr> <tr class="row-odd"><td>42</td> <td></td> <td></td> <td>5</td> <td>Use case 10 (MN status reports)</td> </tr> </tbody> </table> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">How many failure modes do we want to test? Need a Failure Mode Effect Analysis (FMEA) to prioritize the test cases to write against. Comment on FMEA: Assign, for a given failure mode a score of 1-10 for each of a) how likely is this to occur, b) what is the cost if this occurs, c) how early can the failure mode be detected, d) how much does it cost to mitigate the failure mode. Idea is that you multiply across the scores and prioritize accordingly. A failure mode that is very likely, is catastrophic, cannot be detected until it occurs, and is easy to mitigate prioritizes to the top of the list.</p> </div> </div> </div> </div> </div> </div> <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> <div class="sphinxsidebarwrapper"> <p class="logo"><a href="http://dataone.org"> <img class="logo" src="../_static/dataone_logo.png" alt="Logo"/> </a></p> <h3><a href="../index.html">Table Of Contents</a></h3> <ul> <li><a class="reference internal" href="#">Integration Testing</a><ul> <li><a class="reference internal" href="#test-framework">Test framework</a></li> <li><a class="reference internal" href="#list-of-integration-tests">List of integration tests</a></li> </ul> </li> </ul> <h3>Related Topics</h3> <ul> <li><a href="../index.html">Documentation Overview</a><ul> <li><a href="index.html">Implementation Notes</a><ul> <li>Previous: <a href="prioritization.html" title="previous chapter">Implementation Priority</a></li> <li>Next: <a href="versions.html" title="next chapter">Infrastructure Versions</a></li> </ul></li> </ul></li> </ul> <div id="searchbox" style="display: none" role="search"> <h3>Quick search</h3> <form class="search" action="../search.html" method="get"> <div><input type="text" name="q" /></div> <div><input type="submit" value="Go" /></div> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> <script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> <div class="clearer"></div> </div> <div class="footer"> <div id="copyright"> © Copyright <a href="http://www.dataone.org">2009-2017, DataONE</a>. [ <a href="../_sources/implementation/integration-testing.txt" rel="nofollow">Page Source</a> | <a href='https://redmine.dataone.org/projects/d1/repository/changes/documents/Projects/cicore/architecture/api-documentation/source/implementation/integration-testing.txt' rel="nofollow">Revision History</a> ] </div> <div id="acknowledgement"> <p>This material is based upon work supported by the National Science Foundation under Grant Numbers <a href="http://www.nsf.gov/awardsearch/showAward?AWD_ID=0830944">083094</a> and <a href="http://www.nsf.gov/awardsearch/showAward?AWD_ID=1430508">1430508</a>.</p> <p>Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.</p> </div> </div> <!-- <hr /> <div id="HCB_comment_box"><a href="http://www.htmlcommentbox.com">HTML Comment Box</a> is loading comments...</div> <link rel="stylesheet" type="text/css" href="_static/skin.css" /> <script type="text/javascript" language="javascript" id="hcb"> /*<! -*/ (function() {s=document.createElement("script"); s.setAttribute("type","text/javascript"); s.setAttribute("src", "http://www.htmlcommentbox.com/jread?page="+escape((typeof hcb_user !== "undefined" && hcb_user.PAGE)||(""+window.location)).replace("+","%2B")+"&mod=%241%24wq1rdBcg%24Gg8J5iYSHJWwAJtlYu/yU."+"&opts=21407&num=10"); if (typeof s!="undefined") document.getElementsByTagName("head")[0].appendChild(s);})(); /* ->*/ </script> --> </body> </html>