<html>

<head>
<meta http-equiv="Content-Language" content="en-us">
<meta name="GENERATOR" content="Microsoft FrontPage 5.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<title>Electronicl Review Process</title>
</head>

<body bgcolor="#FFFFFF">

<p>Doc. no.:&nbsp; N1631=04-0071<br>
Date:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 11 April 2004<br>
Project:&nbsp;&nbsp;&nbsp;&nbsp; Programming Language C++<br>
Reply to:&nbsp;&nbsp; David Abrahams &lt;dave@boost-consulting.com&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Beman Dawes &lt;bdawes@acm.org&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Jeremy Siek &lt;jsiek@osl.iu.edu&gt;</p>

<h1>Electronic Review Process</h1>

<p><a href="#Introduction">Introduction</a><br>
<a href="#Overview">Process Overview</a><br>
<a href="#Prerequisites">Prerequisites</a><br>
<a href="#Comments">What to include in Review Comments</a><br>
<a href="#Results">Results</a><br>
<a href="#Authors">Notes for Proposal Authors</a><br>
<a href="#Review_Manager">Notes for Review Managers</a><br>
<a href="#FAQ">FAQ</a></p>
<h2><a name="Introduction">Introduction</a></h2>
<p>This document describes an electronic review process designed to augment the 
physical review process used by the C++ committee's working groups (WG's) at 
meetings to reach a consensus.</p>
<p>Under ISO, national body, and committee rules, official action on an issue or 
proposal is taken only by formal vote of the full committee.</p>
<p>To ensure that proposals which come before the full committee are truly ready 
for a formal vote, working group review occurs among domain experts meeting 
physically. Proposals are only brought forward to the full committee when the WG chair determines that sufficient 
working group consensus exists to warrant a full committee vote.</p>
<p>The committee wishes to extend this process to allow reviews to occur 
electronically between  meetings. The intent is to improve the process 
by ensuring:</p>
<ul>
  <li>More eyes on each proposal.<br>
&nbsp;</li>
  <li>Faster review/response cycle.<br>
&nbsp;</li>
  <li>More work  done between meetings and fewer stalls waiting for review 
  responses.<br>
&nbsp;</li>
  <li>Work is not completed at the last minute before a mailing deadline.<br>
&nbsp;</li>
  <li>Decisions are not made in haste; all arguments, especially those of domain 
  experts, are given time to &quot;gestate&quot; between meetings.</li>
</ul>
<p>A separate &quot;review&quot; mailing list reflector will be set up to encourage wide 
participation in electronic reviews.</p>
<p>Regardless of whether review occurs electronically between meetings or  in an 
actual working 
group meeting, final acceptance of any proposal always requires a formal vote by the full committee.</p>
<h2>Process <a name="Overview">Overview</a></h2>
<ol>
  <li>Proposal authors request on the review reflector that a review be 
  scheduled. WG chairs may also nominate issues (with their proposed resolutions) for 
  electronic review.</li>
  <li>The appropriate WG chair appoints a <a href="#Review_Manager">
  review manager</a>.</li>
  <li>The review manager coordinates the review schedule with authors and WG 
  chairs 
  via private email. Reviews are usually scheduled to last at least 14 days, to 
  allow full participation, but may be shorter or longer as experience and the 
  needs of specific proposals dictate.</li>
  <li>The review manager begins the review period by posting a notice of the 
  review, including a summary of the proposal, on both the review reflector and 
  appropriate WG reflector.</li>
  <li><a href="#Comments">Review comments</a> are posted by review reflector participants.</li>
  <li>After the conclusion of the review period, the review manager posts
  <a href="#Results">results</a> on the review and WG reflectors.</li>
  <li>At the next meeting, the WG chair in consultation with the working group 
  makes the final determination to submit the proposal to the full committee for 
  a formal vote.</li>
</ol>
<h2><a name="Prerequisites">Prerequisites</a></h2>
<ul>
  <li>Enough preliminary discussion has occurred to make it clear that the 
  proposal has significant interest and support.<br>
&nbsp;</li>
  <li>The document to be reviewed normally must contain full wording for any proposed 
  change, written in standardese, and identifying the exact document to be 
  changed (WP, TR, etc.) and location for the change. WG chairs may grant 
  exceptions for reviews of incomplete proposals if such a review would expedite 
  WG progress.<br>
&nbsp;</li>
  <li>Only numbered committee documents are reviewed.<br>
&nbsp;</li>
  <li>The document to be reviewed must be transmitted to the committee 
  vice-chair for inclusion in the next mailing.<br>
&nbsp;</li>
  <li>The document to be reviewed must be available on a web site. FTP is 
  discouraged because it is often blocked by corporate firewalls. <i>[Comment: 
  It would be nice to have a common web site to post proposals, but it would 
  have to be one that could be updated by multiple committee members so that no 
  one person could become a bottleneck.]</i></li>
</ul>
<h2>What to include in Review <a name="Comments">Comments</a></h2>
<p>A reviewer's comments may be brief or lengthy, but basically the review manager needs 
your evaluation of the proposal.&nbsp;You can even vote with no additional 
commentary, though comments are valuable and strongly encouraged. If you identify problems along the way, please 
note if they are minor, serious, or showstoppers.</p>
<p>Here are some points you might want to touch on in your review:</p>
<ul>
  <li>What is your evaluation of the motivation and potential usefulness of the 
  proposal?<br>
&nbsp;</li>
  <li>What is your evaluation of the concepts and overall design underlying the 
  proposal?<br>
&nbsp;</li>
  <li>What is your evaluation of the proposed wording? Is the standardese 
  relatively clear?<br>
&nbsp;</li>
  <li>What problems did you detect?<br>
&nbsp;</li>
  <li>Are there any other suggested improvements?<br>
&nbsp;</li>
  <li>Are there any questions for the authors?<br>
&nbsp;</li>
  <li>How much effort did you put into your evaluation? A glance? A quick 
  reading? In-depth study?<br>
&nbsp;</li>
  <li>Are you knowledgeable about the problem domain? </li>
</ul>
<p>And finally, every review should answer this question: &quot;Should the proposal 
be accepted by the committee?&quot;&nbsp; Be sure to say this explicitly so that your 
other comments don't obscure your overall opinion. It may also be helpful if 
your answer is complete enough to give guidance for further work. Possible 
responses:</p>
<ul>
  <li>I support the proposal as written.</li>
  <li>I support the proposal, but feel it needs to be revised to address issues 
  raised in review.</li>
  <li>I do not support the proposal. I might possibly support a future revision 
  that made major changes to address issues raised in review.</li>
  <li>I do not support the proposal and feel that it has enough problems that I 
  am unlikely to support even a much revised version. </li>
</ul>
<h2><a name="Results">Results</a></h2>
<p>At the conclusion of the review period, the review manager will post a 
message with the final tallies of the reviewer's votes and a summary of the main 
issues that were raised during the review. This vote plays the same role as the 
straw polls that occur during working group meetings; it provides a mechanism 
for measuring consensus. As always, the final determination of whether or not 
sufficient consensus exists to warrant forwarding a proposal to the full 
committee is made by working group chairs in consultation with their WG's.</p>
<p>If there are suggestions or conditions that must be met in a revised version 
of a proposal, they should be stated in the review summary.</p>
<h2>Notes for Proposal <a name="Authors">Authors</a></h2>
<p>A proposal should remain stable during the review period; it will just 
confuse and irritate reviewers if a revised document appears during a review 
period.</p>
<p>Authors should respond to reviewer's detailed comments. For example, if a 
reviewer suggests some change in the proposal, the authors might:</p>
<ul>
  <li>Disagree, and 
give rationale.</li>
  <li>Be undecided, and ask for other views.</li>
  <li>Agree, and indicate plans to make the 
change in a revised paper.</li>
</ul>
<h2>Notes for <a name="Review_Manager">Review Manager</a>s</h2>
<p>Before a proposal can be scheduled for electronic review, someone not 
connected with it must volunteer to be the &quot;Review Manager&quot; 
for the proposal.</p>
<p>The review manager for an electronic  review performs similar 
functions to a working group chair during in-person working group reviews; they 
coordinate the process and tally the votes for whether or not enough 
consensus exists to forward the proposal to the full committee for a formal 
vote. They also mediate if discussion becomes overheated.</p>
<p>It is expected that working group chairs may act as review managers 
for proposals relevant to their working group. WG chairs may ask someone else to 
act as review manager when the chair is the proposal's author, or to spread the 
workload.</p>
<p>The review manager:</p>
<ul>
  <li>Checks the proposal to make sure the <a href="#Prerequisites">
  prerequisites</a> are met.</li>
  <li>Finalizes the schedule with the working group chair and the submitter .
  </li>
  <li>On the first day of the review, posts a notice of the review schedule on 
  both the review reflector and the working group reflector.<ul>
    <li>The notice should include a brief description of the proposal, to let 
    readers know if it is a proposal they are interested in reviewing. </li>
  </ul>
  </li>
  <li>Urges people to do reviews if they aren't forthcoming. </li>
  <li>Follows review discussions regarding the proposal, moderating or answering 
  questions as needed. </li>
  <li>After the review period ends, posts a message on both the review reflector and the working group 
  reflector informing members of the <a href="#Results">review results</a>. </li>
</ul>
<p>In other words, it is the review manager's responsibility to make sure the 
review process works smoothly.</p>
<h2><a name="FAQ">FAQ</a></h2>
<p><b>Who can participate in an Electronic Review? </b>Anyone with access to the 
review reflector. As with any committee reflector traffic, committee officers 
monitor discussion to ensure acceptable use. And as with other reviews, WG 
chairs factor in any divergence between straw voters and voting members when 
determining if enough consensus exists to warrant a full committee vote.</p>

<p><b>Why are the review periods so short/long?</b> The Boost experience with 
electronic reviews is that there is a tension between the desire to give 
reviewers plenty of time to prepare reviews, and practical factors such as 
reviewers delaying until near the end of the review period, workload for 
submitters and review managers, and lack of discussion focus when review periods 
are lengthy. The length of review periods can be adjusted as committee 
experience with electronic reviews grows.</p>

<p><b>Why aren't the reviews scheduled right after between meeting mailings?</b> 
The initial strategy is to spread the committee's workload out over time, rather 
than concentrate it in a few specific time periods, and to decouple reviews from 
other events like mailings to simplify the process. Schedule timeslots can be 
adjusted as committee experience with electronic reviews grows.</p>

<p><b>Why aren't documents to be reviewed posted on the committee web site?</b> 
Logistics. We don't want the webmaster to become a bottleneck or to be 
overloaded. Perhaps committee web site procedures can be worked out, or a 
permanent committee Wiki can be established, but we don't want to delay progress 
until these administrative details are resolved.</p>

<p><b>The <a href="#Prerequisites">prerequisites</a> are tilted toward mature 
proposals. Won't this inhibit discussion of &quot;half-ready&quot; proposals?</b> Yes, and 
that is intended. The individual working group reflectors are the appropriate 
place for discussions of &quot;good ideas&quot;, &quot;trial balloons&quot;, &quot;half-ready&quot; proposals, 
and the like. Electronic reviews involve enough expenditure of committee effort 
that they are best reserved for mature proposals, or for less-mature proposals 
important enough to justify electronic review.</p>

<p><b>How can an electronic review replace all meeting review time?</b> It 
can't. The goal of electronic reviews isn't to eliminate all meeting review 
time, but rather to allow&nbsp; meetings to be more productive. Electronic 
review allows authors to refine proposals, meeting participants to vote in 
confidence, and with minimal time be spent in meetings on review aspects which 
can be handled well electronically.</p>

<hr>
<p>Revised
<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%d %B, %Y" startspan -->11 April, 2004<!--webbot bot="Timestamp" endspan i-checksum="29903" --></p>

</body>

</html>