<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
   "http://www.w3.org/TR/html4/strict.dtd">

<html>
<head>

  <meta http-equiv="Content-Type" content="text/html;charset=US-ASCII">

<!-- This should replace the style block once it is published somewhere accessible
  <link href="../../base_www/style.hinc" rel="stylesheet" type="text/css" />
-->

  <style type="text/css">

    body { color: #000000; background-color: #FFFFFF; }
    del { text-decoration: line-through; color: #8B0040; }
    ins { text-decoration: underline; color: #005100; }

    p.example { margin-left: 2em; }
    pre.example { margin-left: 2em; }
    div.example { margin-left: 2em; }

    code.extract { background-color: #F5F6A2; }
    pre.extract { margin-left: 2em; background-color: #F5F6A2;
      border: 1px solid #E1E28E; }

    p.function { }
    .attribute { margin-left: 2em; }
    .attribute dt { float: left; font-style: italic;
      padding-right: 1ex; }
    .attribute dd { margin-left: 0em; }

    blockquote.std { color: #000000; background-color: #F1F1F1;
      border: 1px solid #D1D1D1;
      padding-left: 0.5em; padding-right: 0.5em; }
    blockquote.stddel { text-decoration: line-through;
      color: #000000; background-color: #FFEBFF;
      border: 1px solid #ECD7EC;
      padding-left: 0.5empadding-right: 0.5em; ; }

    blockquote.stdins { text-decoration: underline;
      color: #000000; background-color: #C8FFC8;
      border: 1px solid #B3EBB3; padding: 0.5em; }

    table { border: 1px solid black; border-spacing: 0px;
      margin-left: auto; margin-right: auto; }
    th { text-align: left; vertical-align: top;
      padding-left: 0.8em; border: none; }
    td { text-align: left; vertical-align: top;
      padding-left: 0.8em; border: none; }
      
    table.docinfo { border: none; border-spacing: 0px;
      margin-left: auto; margin-right: none; float: right; }

  </style>

  <title>Improving the Verification of C++ Programs</title>

</head>

<body>
  <table border="1" class="docinfo">
    <tr> <th>Doc. No.:</th> <td>N3880</td> </tr>
    <tr> <th>Date:</th>     <td>2014-01-16</td> </tr>
    <tr> <th>Reply To:</th> <td>Michael Price<br/>
                                &lt;<a href="mailto:michael.b.price.dev@gmail.com">michael.b.price.dev@gmail.com</a>&gt;</td> </tr>
  </table>
  
  <br/>
  <h1>Improving the Verification of C++ Programs</h1>

  <h2><a name="problem_statement">I. Problem Statement</a></h2>

  <p>
    The verification of software has always been one of the fundamental
    responsibilities of our profession. As computing resources grew,
    so did the complexity of our software systems. The
    complexity (i.e. sophistication) of the languages used to construct
    software systems grew as well. Unfortunately, it took a very long while for
    our development processes to catch up, particularly those processes related
    to system verification. Over the past few decades, we've seen dramatic
    changes in our approach to building quality software. Agile methodologies
    have brought forth the requirement for rapid iteration and comprehensive
    automated verification of behavior. Programming languages that have been
    created (or evolved) in this environment have incorporated these
    new requirements with differing levels of success.
  </p>

  <p>
    At the beginning of a software development project, the question of
    <em>which</em> language to use is often one of the very first that is
    answered. Strengths and weaknesses of the options should be considered
    carefully. It is my experience that quite often, concerns about
    <em>verifiability</em> weigh heavily. In my opinion, the lack of a
    standardized method for program verificationis a significant barrier to
    adoption of C++ in many projects, only behind a relatively paltry standard
    library and the indecipherable nature of template error messages (both of
    which the committee has made significant headway on).
  </p>

  <h2><a name="verification_overview">II. An Overview of (Agile) Verification</a></h2>

  <p>
    In order for agile processes to work well, the agile team must be able to
    iterate quickly without creating undesired regressions in the behavior of
    the product. This means that it is important to catch errors early and fix
    them quickly. In order to achieve that goal, automated verification of the
    product is vital.
  </p>

  <p>
    In the book <em>Succeeding with Agile</em>, Mike Cohn introduced the concept
    of the test pyramid. The test pyramid is meant to convey the "ideal"
    distribution of tests across the different layers of a system, with a vast
    majority of them appearing at the "unit" level and very few appearing at the
    "user interface" level. It is a useful representation that I will reference
    in Section III, so I've included it here.
  </p>

  <svg width="300" height="250">
    <polygon points="150,25 275,225 25,225" style="fill:white; stroke:black; stroke-width:2" />
    <line x1="108" y1="92" x2="191" y2="92" style="stroke:black; stroke-width:1" />
    <line x1="67" y1="158" x2="234" y2="158" style="stroke:black; stroke-width:1" />
    <text x="142" y="75" fill="black" font-weight="bold">UI</text>
    <text x="127" y="130" fill="black" font-weight="bold">Service</text>
    <text x="136" y="194" fill="black" font-weight="bold">Unit</text>
  </svg>

  <p>
    The following are some of the characteristics that a verification framework
    could include (not meant to be an exhaustive list).
  </p>

  <h3><a name="verification_overview/registration">Test Registration</a></h3>

  <p>
    There must be a way to signify that a particular portion of code represents
    a test that verifies the functionality of some component of the program.
    Attributes (or annotations) are often used for this purpose in languages
    that support those concepts. In some languages, there are reserved keywords
    for this purpose, while in many others, there is a registry of tests that
    each test must be added to in order to run.
  </p>

  <h3><a name="verification_overview/assertions">Assertions</a></h3>

  <p>
    The fundamental action in a test is to assert that the state of the code
    under test is correct.  Typically, there will be a sophisticated set of
    assertion capabilities, including those that wrap the normal comparison
    operations. There is often the ability to provide a specific message to be
    recorded in case an assertion fails.
  </p>

  <h3><a name="verification_overview/reuse">Code Reuse</a></h3>

  <p>
    Tests that require the program to be in a certain state before (or after) an
    assertion is made are usually grouped with tests with similar requirements.
    The ability to reuse that code in multiple tests is sometimes provided in
    the form of <em>fixtures</em> or setup and teardown methods.
  </p>

  <h3><a name="verification_overview/mocking">Mocking</a></h3>

  <p>
    Providing tests to verify the behavior of some code can sometimes lead to
    dependencies on external systems (e.g. database or networks). This is
    undesirable since it increases the cycle time for a team and could also
    require the configuration of additional resources (sometimes manually). In
    order to avoid these costs, some languages or libraries will provide the
    ability to replace a real resource with a mock resource. This allows tests
    to be shifted from being a Service test to a Unit test, according to the
    test pyramid.
  </p>

  <h3><a name="verification_overview/termination">Abnormal Termination ("Death Tests")</a></h3>

  <p>
    Many times, systems are designed to terminate upon some condition so as to
    prevent corruption of data or because of constraints imposed by the
    language. The ability to verify that these design choices are implemented
    correctly is quite useful.
  </p>

  <h3><a name="verification_overview/results">Observing Results</a></h3>

  <p>
    Tests are only useful if their results are accessible and structured in a
    way such that test failures are easy to locate and understand. XML is a
    popular output format (sometimes with a well-defined schema), while others
    present the results as a part of an IDE or as compiler output.
  </p>

  <h3><a name="verification_overview/coverage">Code Coverage</a></h3>

  <p>
    The value of verification is highly dependent on the amount of code that is
    actually executed during the verification process. The ability to know what
    portions of code are currently covered by automated verification is an
    extremely useful tool in guiding development teams to apply their available
    test-writing time in the areas that need it the most.
  </p>

  <h2><a name="unique_view">III. A Unique View</a></h2>

  <p>
    I believe that there is a hidden layer to the test pyramid that was
    introduced in Section II, the compiler. Conceptually, the process of
    verification consists of taking a representation of the problem
    domain and comparing it against a set of well-known domain constraints.
    The compiler is the very first test suite that any code ever sees (outside
    of the humain brain).
  </p>

  <svg width="300" height="250">
    <polygon points="150,25 275,225 25,225" style="fill:white; stroke:black; stroke-width:2" />
    <line x1="118" y1="75" x2="182" y2="75" style="stroke:black; stroke-width:1" />
    <line x1="88" y1="125" x2="213" y2="125" style="stroke:black; stroke-width:1" />
    <line x1="57" y1="175" x2="243" y2="175" style="stroke:black; stroke-width:1" />
    <text x="142" y="65" fill="black" font-weight="bold">UI</text>
    <text x="127" y="107" fill="black" font-weight="bold">Service</text>
    <text x="136" y="154" fill="black" font-weight="bold">Unit</text>
    <text x="103" y="204" fill="black" font-weight="bold">Design/Domain</text>
  </svg>

  <p>
    If you've done something that violates a constraint
    specified by the language or library (a very broad domain indeed), then it
    fails. Indeed, a language that utilizes static typing has a clear advantage
    in this layer, even more so for languages that allow for the definition of
    further domain constraints (e.g. <code>static_assert</code>). This is a way
    of looking at the problem that suits the strengths of C++ and is ripe with
    opportunity.
  </p>

 <h2><a name="areas_of_improvement">IV. Potential Areas of Improvement for C++</a></h2>

  <p>
    Following is a list of ideas to serve as a starting point for further
    discussion. They are ordered roughly in their perceived difficulty/impact
    on the language and tools.
  </p>

  <ul>
    <li>Capabilities for better <code>static_assert</code> messages</li>
    <li>Better support for verifying API design decisions (i.e. testing compile-time failures)</li>
    <li>Discrete classification of causes of <em>ill-formed</em>ness</li>
    <li>Better source code contextual information capabilities (as called for by Reflection study group)</li>
    <li>A <code>std::test_exception</code> (bikeshed) type to serve as a basis for...</li>
    <li>An full-featured assertion library</li>
    <li>Registration of code to be run during verification</li>
    <li>Automated generation of verification programs</li>
  </ul>

  <h2><a name="going_forward">V. Going Forward</a></h2>

  <p>
    My request and recommendation going forward would be that
    these topics be grouped under a new study group. I presume that some of the
    simpler ideas could have proposals written for them fairly quickly, but
    many of the other ideas are complex enough to require a dedicated group to
    thoroughly investigate them.
  </p>

  <h2><a name="acknowledgements">VI. Acknowledgements</a></h2>

  <p>
    I'd like to thank my employer, Perceptive Software, for their continued
    support of my work to improve the C++ language. I also greatly appreciate
    the reviews provided by my associates: Chris Sammis, Matthew Dissinger, and
    Andrew Regier.
  </p>

</body>
</html>