<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta charset="UTF-8"/>
  <meta name="viewport" content="width=device-width, initial-scale=1"/>
  <meta name="title" content="Modules, Macros, and Build Systems"/>

  <title>Modules, Macros, and Build Systems</title>

  <style type="text/css">

html
{
  font-family: "Helvetica Neue", Helvetica, "Segoe UI", Arial, freesans, sans-serif;
  font-weight: normal;
  font-size: 18px;
  line-height: 1.4em;
  letter-spacing: 0.01em;

  color: #292929;
}

body {margin: 0;} /* There is non-0 default margin for body. */

/* See notes on what's going on here. */
body {min-width: 17em;}
@media only screen and (min-width: 360px)
{
  body {min-width: 19em;}
}

/*
 * Header (optional).
 */

#header-bar
{
  width: 100%;

  background: rgba(0, 0, 0, 0.04);
  border-bottom: 1px solid rgba(0, 0, 0, 0.2);

  padding: .4em 0 .42em 0;
  margin: 0 0 1.4em 0;
}

#header
{
  /* Same as in #content. */
  max-width: 41em;
  margin: 0 auto 0 auto;
  padding: 0 .4em 0 .4em;

  -webkit-box-sizing: border-box;
  -moz-box-sizing: border-box;
  box-sizing: border-box;

  width: 100%;
  display: table;
  border: none;
  border-collapse: collapse;
}

#header-logo, #header-menu
{
  display: table-cell;
  border: none;
  padding: 0;
  vertical-align: middle;
}

#header-logo {text-align: left;}
#header-menu {text-align: right;}

/* These overlap with #header's margin because of border collapsing. */
#header-logo {padding-left: .4em;}
#header-menu {padding-right: .4em;}

#header-logo a
{
  color: #000;
  text-decoration: none;
  outline: none;
}
#header-logo a:visited {color: #000;}
#header-logo a:hover, #header-logo a:active {color: #000;}

#header-menu a
{
  font-size: 0.889em;
  line-height: 1.4em;
  text-align: right;
  margin-left: 1em;
  white-space: nowrap;
  letter-spacing: 0;
}

#header-menu a
{
  color: #000;
  outline: none;
}
#header-menu a:visited {color: #000;}
#header-menu a:hover, #header-menu a:active
{
  color: #3870c0;
  text-decoration: none;
}

/* Flexbox-based improvements though the above works reasonably well. */
#header-menu-body
{
  width: 100%;

  display: -webkit-inline-flex;
  display: inline-flex;

  -webkit-flex-flow: row wrap;
  flex-flow: row wrap;

  -webkit-justify-content: flex-end;
  justify-content: flex-end;
}

/* Whether we want it (and at which point) depends on the size of the menu. */
/*
@media only screen and (max-width: 567px)
{
  #header-menu-body
  {
    -webkit-flex-direction: column;
    flex-direction: column;
  }
}
*/

/*
 * Content.
 */

#content
{
  max-width: 41em;
  margin: 0 auto 0 auto;
  padding: 0 .4em 0 .4em; /* Space between text and browser frame. */

  -webkit-box-sizing: border-box;
  -moz-box-sizing: border-box;
  box-sizing: border-box;
}

/*
 * Footer (optional).
 */

#footer
{
  color: #767676;
  font-size: 0.7223em;
  line-height: 1.3em;
  margin: 2.2em 0 1em 0;
  text-align: center;
}

#footer a
{
  color: #767676;
  text-decoration: underline;
}
#footer a:visited {color: #767676;}
#footer a:hover, #footer a:active {color: #3870c0;}

/* Screen size indicator in the footer. The before/after content is in case
   we don't have any content in the footer. Margin is to actually see the
   border separate from the browser frame. */

/*
#footer:before {content: "\A0";}
#footer:after {content: "\A0";}

#footer
{
  border-left: 1px solid;
  border-right: 1px solid;
  margin-left: 1px;
  margin-right: 1px;
}

@media only screen and (max-width: 359px)
{
  #footer {border-color: red;}
}

@media only screen and (min-width: 360px) and (max-width: 567px)
{
  #footer {border-color: orange;}
}

@media only screen and (min-width: 568px) and (max-width: 1023px)
{
  #footer {border-color: blue;}
}

@media only screen and (min-width: 1024px)
{
  #footer {border-color: green;}
}
*/

/*
 * Common elements.
 */

p, li, dd {text-align: justify;}
.code {text-align: left;} /* Manually aligned. */
pre {text-align: left;}   /* If it is inside li/dd. */

/* Notes. */

.note
{
  color: #606060;
}

div.note
{
  margin: 1em 0 1em 0;

  padding-left: 0.5em;
  border: 0.25em;
  border-left-style: solid;
  border-color: #808080;

  page-break-inside: avoid;
}

div.note :first-child {margin-top:    0;}
div.note :last-child  {margin-bottom: 0;}

span.note::before {content: "[Note: "}
span.note::after  {content: "]"}

/* Links. */
a
{
  color: #3870c0;
  /*color: #4078c0;*/
  text-decoration: none;
}

a:hover, a:active
{
/*color: #006fbf;*/
/*color: #0087e7;*/
  text-decoration: underline;
}

a:visited
{
/*color: #003388;*/
  color: #00409c;
}

/* Standard lists. */
ul, ol, dl {margin: 1em 0 1em 0;}
ul li, ol li {margin: 0 0 .4em 0;}
ul li {list-style-type: circle;}
dl dt {margin: 0 0 0 0;}
dl dd {margin: 0 0 .6em 1.8em;}

code, pre
{
  font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
  font-size: 0.92em;
  letter-spacing: 0;
}

pre {white-space: pre-wrap;}

@media only screen and (max-width: 567px)
{
  pre {word-break: break-all;}
}

/* Use page rather than system font settings. */
input
{
  font-family: inherit;
  font-weight: inherit;
  font-size:   inherit;
  line-height: inherit;
}


pre
{
  background-color: rgba(0, 0, 0, 0.05);
  border-radius: 0.2em;
  padding: .8em .4em .8em .4em;
  margin: 2em -.4em 2em -.4em; /* Use margins of #content. */
}


code
{
  background-color: rgba(0, 0, 0, 0.05);
  border-radius: 0.2em;
  padding: .2em .32em .18em .32em;
}

/*
code::before
{
  letter-spacing: -0.2em;
  content: "\00a0";
}

code::after
{
  letter-spacing: -0.2em;
  content: "\00a0";
}
*/


/* Bases:
 *
 * common.css
 * pre-box.css
 * code-box.css
 *
 */

#content
{
  max-width: 43.6em;
  padding-left: 3em; /* Reserve for headings. */
}

h1
{
  font-weight: normal;
  font-size: 2em;
  line-height: 1.4em;
  margin: 1.6em 0 .6em -1.4em;
}

h1.preface
{
  margin-left: -.56em;
}

h2
{
  font-weight: normal;
  font-size: 1.556em;
  line-height: 1.4em;
  margin: 1.6em 0 .6em -.8em;
}

h3
{
  font-weight: normal;
  font-size: 1.3em;
  line-height: 1.4em;
  margin: 1.6em 0 .6em -.2em;
}

/* Title page */

#titlepage {
  margin: 0 0 4em 0;
  border-bottom: 1px solid black;
}

#titlepage .title {
  font-weight: normal;
  font-size: 2.333em;
  line-height: 1.4em;
  letter-spacing: 0;
  text-align: center;
  margin: 2em 0 2em 0;
}

#titlepage p {
  font-size: 0.889em;
  line-height: 1.4em;
  margin: 2em 0 .6em 0;
}


table.toc
{
  border-style      : none;
  border-collapse   : separate;
  border-spacing    : 0;

  margin            : 0.2em 0 0.2em 0;
  padding           : 0 0 0 0;
}

table.toc tr
{
  padding           : 0 0 0 0;
  margin            : 0 0 0 0;
}

table.toc * td, table.toc * th {
  border-style      : none;
  margin            : 0 0 0 0;
  vertical-align    : top;
}

table.toc * th
{
  font-weight       : normal;
  padding           : 0 0.8em 0 0;
  text-align        : left;
  white-space       : nowrap;
}

table.toc * table.toc th
{
  padding-left      : 1em;
}

table.toc * td
{
  padding           : 0 0 0 0;
  text-align        : left;
}

table.toc * td.preface
{
  padding-left      : 1.35em;
}


/*
 * Property list table.
 */
.proplist
{
  width: calc(100%); /* Fill the page. */

  table-layout: fixed;

  border: none;
  border-spacing: 0 0;
}

.proplist th, .proplist td {padding: .1em 0 .1em 0;}

.proplist th
{
  font-weight: normal;
  text-align: left;

  width: 7em;
}
.proplist th:after {content: ":";}
  </style>

</head>
<body>
<div id="content">

  <div id="titlepage">
    <div class="title">Modules, Macros, and Build Systems</div>

    <table class="proplist">
      <tbody>
	<tr><th>Document</th><td><a href="https://wg21.link/P1052R0">P1052R0</a></td></tr>
	<tr><th>Audience</th><td>EWG, SG15</td></tr>
	<tr><th>Authors</th><td>Boris Kolpackov</td></tr>
	<tr><th>Reply-To</th><td>boris@codesynthesis.com</td></tr>
	<tr><th>Date</th><td>2018-05-02</td></tr>
      </tbody>
    </table>
  </div>
  <h2 id="toc">Contents</h2>

  <table class="toc">
    <tr><th>1</th><td><a href="#abstract">Abstract</a></td></tr>
    <tr><th>2</th><td><a href="#background">Background</a></td></tr>
    <tr><th>3</th><td><a href="#macros">Modules with Macros</a></td></tr>
    <tr><th>4</th><td><a href="#atom">The Atom Proposal</a></td></tr>
    <tr><th>5</th><td><a href="#merged">The Merged Proposal</a></td></tr>
    <tr><th>6</th><td><a href="#ack">Acknowledgements</a></td></tr>
  </table>

  <h2 id="abstract">1 Abstract</h2>

  <p><i>One of the main challenges of building modularized projects is
  discovering the set of modules imported by each translation unit. This is a
  relatively straightforward process if modules are a purely language-level
  mechanism (as is currently the case in Modules TS). If, however, modules
  start affecting the preprocessor (for example, by supporting exportation of
  macros), then we believe this discovery will result in complexity that most
  build system (and other tools) vendors will have no capacity to
  handle.</i></p>

  <p><i>While it appears that the authors of the Atom proposal have recognized
  this issue and tried to resolve it, we believe their current approach is
  unworkable. While there appears to be no immediate plans to support
  exportation of macros either in IS or Modules TS, there are plans to merge
  legacy header modules from the Atom proposal into Modules TS which would
  make the combined result still suffering from these problems.</i></p>

  <h2 id="background">2 Background</h2>

  <p>In order to build a project that uses modules, a build system needs to
  obtain <i>module dependency information</i> &#8211; a set of modules
  imported by each translation unit (TU). This information is necessary both
  to establish the order in which TUs can be compiled as well as which TUs
  must be recompiled. Specifically, a module interface unit must be compiled
  (into a binary module interface or BMI) before any TU that imports it and if
  a module interface unit has changed, then all the TUs that import it must be
  recompiled.</p>

  <div class="note">
  <p>Just like with header dependency information, we believe having the user
  specify module dependency information manually is not a scalable
  approach.</p>
  </div>

  <div class="note">
  <p>Compiling all the module interface units as an <i>ad hoc</i> pre-build
  step is not a workable approach since module interface units may import each
  other.</p>
  </div>

  <p>Extracting module dependency information in Modules TS (<a
  href="https://wg21.link/n4720">N4720</a>) is a relatively straightforward
  process: the TU is preprocessed, tokenized, and <i>shallow-parsed</i> to
  collect the names of imported modules. The parsing can be shallow because
  all the module-related declarations are <i>top-level</i> and the parser can
  safely ignore all the tokens inside <code>{}</code>. <span class="note">With
  the exception for <code>export&#160;{&#160;import&#160;M;&#160;}</code>
  which is still easy to recognize and handle.</span></p>

  <div class="note">
  <p>Extracting module dependency information can naturally be combined with
  the header dependency extraction which requires essentially a full
  preprocessor run.</p>
  </div>

  <div class="note">
  <p>While we would expect the compiler vendors to provide this functionality
  (perhaps combined with the header dependency extraction that they already
  provide), our experience with <a
  href="https://build2.org"><code>build2</code></a> shows that this can also
  be implemented by the build system (or other tools) with good results
  (simple and reliable code with good performance).</p>
  </div>

  <h2 id="macros">3 Modules with Macros</h2>

  <p>Making modules capable of exporting macros will make <code>import</code>
  a preprocessor directive rather than (or, more precisely, in addition to) a
  language declaration since via exported macros it will now be able to affect
  the preprocessor state. This "hoisting" of <code>import</code> into the
  preprocessor will significantly complicate module dependency extraction.</p>

  <p>Specifically, the approach described in the previous section will no
  longer work since a previously-imported module may now effect (via a
  preprocessor macro) the importation of subsequent modules. For example:</p>

  <pre>import foo; // May export macro FOO.

#ifdef FOO
import bar;
#endif</pre>

  <p>In this model the build system will no longer be able to determine the
  module dependency information at the outset, before starting the
  compilation. And the compiler may not have access to all the (up-to-date)
  BMIs to perform the compilation. As a result, the compiler will have to
  <i>call back</i> into the build system on encountering every
  <code>import</code> directive in order to obtain an (up-to-date) BMI that it
  can use (and which the build system might still have to compile, potentially
  triggering a recursive chain of callbacks).</p>

  <p>Besides the sheer complexity of this approach, implementing it in a
  parallel build system immediately presents a number of practical challenges
  (contemplating challenges for a distributed build system is left as an
  exercise for the reader):</p>

  <ol>
  <li>In this model discovery of imported modules is inherently a serial
  operation. Which means, for a single TU, building of its imported modules is
  non-parallelizable. While this may not be an issue for most from-scratch
  builds (since we will, presumably, be compiling multiple TU that import
  different sets of modules and/or in different order), this can become a
  major problem for incremental builds.</li>

  <li>A callback into a parallel build system may determine that the requested
  module interface unit is already being compiled (as a result of being
  imported by another TU) and therefore would have to wait. In this case, to
  achieve full resource utilization, the build system would have to reuse the
  "job" to compile another TU (which can again get blocked and be reused,
  recursive). As a result, in this model, we may end up with a large number of
  waiting compiler processes that still holdup other (than CPU) system
  resources (most critically, RAM).</li>
  </ol>

  <h2 id="atom">4 The Atom Proposal</h2>

  <p>We understand that the authors of the Atom Proposal (<a
  href="https://wg21.link/p0947r1">P0947R1</a>) have recognized this issue and
  tried to resolve it by imposing a number of restrictions on the location of
  the import directives/declarations as well as the kind of macro expansions
  that can be performed around them. Specifically, <a
  href="https://wg21.link/p0947r1#import-and-cpp">Section 4.3, "Preprocessor
  Impact"</a> states:</p>

  <p><i>"We also wish to permit the set of imports of a translation unit to be
  determined without knowledge of the contents of the imported translation
  units. In particular, the full set of dependencies should be discoverable
  (for instance, by a build tool or a non-compiler parser of source code)
  without the need to consult external files, safe in the knowledge that no
  macro will (for instance) <code>#define&#160;import</code>. However,
  preprocessor action should still be permitted in the import declaration
  region, to allow constructs such as:</i></p>

  <pre>#ifdef BUILDING_ON_UNIX
import support.unix;
#else
import support.windows;
#endif</pre>

  <p><i>To this end, macro expansion before the end of the initial sequence of
  import-declarations is disallowed from expanding an imported macro."</i></p>

  <p>Firstly, we believe it will be hard for external tools to extract the
  imported module set without support from the compiler because  the required
  semantics will have to be along these lines:</p>

  <p><i>Preprocess the import region and stop (we have to stop since
  continuing preprocessing requires BMIs). Then parse the import declarations
  and return the set of imported module names.</i></p>

  <p>The stop part is something that external tools will have a hard time
  doing without some sort of support from the compiler.</p>

  <p>The second, bigger, issue is how to stop. We don't see how this can be
  achieved without loading the BMIs because without doing so there is no way
  of knowing whether a macro is module-exported or not. Consider these two
  side-by-side examples:</p>

  <pre>import foo; // May export FOO.  | import foo; // Doesn't export FOO.
                                |
#ifndef FOO                     | #ifndef FOO
#  error need FOO               |   import bar;
#endif                          | #endif</pre>

  <p>Without loading <code>foo</code>'s BMI the two cases are
  indistinguishable. However, the compiler has to somehow stop before
  <code>#ifdef</code> on the left hand side but continue on the right.</p>

  <h2 id="merged">5 The Merged Proposal</h2>

  <p>From <a href="https://wg21.link/p0983r0">P0983R0</a> we understand that
  while there are currently no plans to add support for exportation of macros
  into IS or Modules TS, there are plans to merge legacy header modules from
  the Atom proposal into Modules TS. Such modules will be able to export
  macros and therefore the combined result will be affected by the problems
  described in the previous sections.</p>

  <h2 id="ack">6 Acknowledgements</h2>

  <p>Thanks to Nathan Sidwell and Jens Maurer for comments on early drafts of
  this paper.</p>

</div>

</body>
</html>
