<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    <title>issue 454: problems and solutions</title>
  </head>
  <body>
    <table>
      <tr>
        <td align="left">Doc. no.</td>
        <td align="left">N2683=08-0193</td>
      </tr>
      <tr>
        <td align="left">Date:</td>
        <td align="left">2008-06-27</td>
      </tr>
      <tr>
        <td align="left">Project:</td>
        <td align="left">Programming Language C++</td>
      </tr>
      <tr>
        <td align="left">Reply to:</td>
        <td align="left">
          <a href="mailto:sebor@roguewave.com">Martin Sebor
        </td>
      </tr>
    </table>
    <hr><!-------------------------------------------------------->
    <h1>

issue 454: problems and solutions
    </h1>
    <!------------------------------------------------------------>

  <discussion>
      <h3>Problems with proposed resolution (as
of <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2612.html">N2612</a>)</h3>
      <ol>
        <li>

<b>Limited Applicability</b>: Only two operating systems are known to
provide APIs for the manipulation of wide character file names:
Symbian OS
(see <a href="http://www.symbian.com/Developer/techlib/v70sdocs/doc_source/reference/cpp/libc/stdio_h/wfopenFunction.html"><code>wfopen()</code></a>
in the Symbian Developer Library) and Microsoft Windows
(see <a href="http://msdn.microsoft.com/en-us/library/yeby3zcb.aspx"><code>_wfopen()</code></a>
in the  Microsoft Visual C++ Run-Time Library Reference). Most other
popular operating systems, including UNIX flavors such as AIX,
FreeBSD, HP-UX, IRIX, Linux, Mac OS/X, Solaris, and Tru64 UNIX,
provide only narrow (8-bit clean) character interfaces. Some operating
systems (such as HP OpenVMS) preclude the use of internationalized
file names due to the restrictions they place on the characters used
in such names.

        </li>
        <li>

<b>Inadequate Interface</b>: It is insufficient to provide an
interface to open or create files with wide character names without
also providing interfaces to rename or remove files with such
names. This is especially problematic when the translation from the
internal <code>wchar_t</code> representation of the file name to the
external representation is unspecified because it leaves programs with
no means to portably rename or remove files created using the first
interface. Another common operation involving file names that would
suffer from the lack of interoperability due to the unspecified nature
of the conversion is the traversal over the list of file names in a
given directory (the POSIX functions <code>opendir()</code>
and <code>readdir()</code>).

        </li>
        <li>

<b>Insufficient Control</b>: The historical practice common on UNIX
platforms is to allow the use of arbitrary encodings for file names,
leaving it up to each user to decide on the most suitable encoding.
While with the advent of Unicode this practice is slowly becoming
displaced by the use of UTF-8 by all, it is still in widespread use.

        </li>
        <li>

<b>Underspecification</b>: The behavior of C++ file streams is
specified in terms of underlying <code>FILE</code>s, that is, <i>"as
if"</i> <code>basic_filebuf::open()</code> called the C
function <code>fopen()</code>. Since there is no
corresponding <code>wchar_t</code> overload of <code>fopen()</code>
(or <code>wfopen()</code>), introducing a <code>wchar_t</code>
overload of <code>basic_filebuf::open()</code> opens a hole in the
specification leaving the behavior of <code>basic_filebuf</code>
objects opened using this overload substantially unspecified.

        </li>

      </ol>
    </discussion>
    <resolution>
      <h3>Solutions</h3>
      <p>

The simplest solution for dealing with problem (1) above in C++ is to
do nothing. Users on the two platforms that provide
proprietary <code>wchar_t</code> file system APIs (such
as <code>wfopen()</code>) can use those APIs. On all other platforms,
users can use the existing standard interfaces (<code>fopen()</code>
or <code>filebuf::open()</code>).

      </p>
      <p>

To adequately address problem (2) above, the C++ standard would need
to add overloads for all functions that take narrow file name
now. These include the following
functions: <code>fopen()</code>, <code>freopen()</code>, <code>remove()</code>, <code>rename()</code>, <code>tmpnam()</code>,
and possibly also <code>system()</code>. However, since these
functions are specified by the C standard, and since previous
enhancements in the area where the C and C++ standards overlap have
proved to be exceedingly problematic in practice, the question of
adding these extensions would be best left to the C committee. Another
solution to problem (2) is for C++ to provide its own comprehensive
interface to the file system, independent of C, such as that proposed
in <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n1975.html">N1975</a>,
Filesystem Library Proposal for TR2 (Revision 3).

      </p>
      <p>

To address problem (3), it is necessary to provide a means for
programs to select the desired external encoding of file names. The
solution is to add a new bit to the <code>ios_base::openmode</code>
bitmask, let's call it
<code>ios_base::cvtname</code>, that, when set in
the <code>mode</code> argument in a call
to <code>basic_filebuf::open()</code>, has the effect of causing the
function to convert the file name argument using
the <code>codecvt</code> facet installed in the locale imbued in
the <code>basic_filebuf</code> object. Thus,
a <code>basic_filebuf</code> object opened by calling
the <code>wchar_t</code> overload to <code>open()</code> with
the <code>ios_base::cvtname</code> bit set in the <code>mode</code>
argument will have predictable and portable behavior, namely that of
opening a file whose name is the result of converting the <code>const
wchar_t*</code> argument to a NTBS using the <code>codecvt</code>
facet obtained as if by
calling <code>use_facet&lt;codecvt&lt;char_type&gt;
&gt;(this-&gt;getloc())</code>. With
the <code>ios_base::cvtname</code> bit clear, the behavior or the
function will be implementation-defined to allow implementers to
provide reasonable default behavior appropriate for each platform.

      </p>
      <p>

The solution to problem (3) also largely resolves problem (4). While
the default behavior of programs that do not set
the <code>ios_base::cvtname</code> bit isn't specified by the standard
(it is implementation-defined) or guaranteed to be portable from one
implementation of C++ to another, the behavior of programs that do set
the bit is precisely specified by C++ and guaranteed to have
predictable results. That is not to say that the behavior of such
programs is guaranteed to be the same across different operating
systems, only that it will be the same on the same operating system
regardless of the implementation of C++.

      </p>
      <h3>Proposal</h3>
      <p>

We urge the library working group to defer this issue until such time
that problem (2) has been addressed in the C standard, or until C++
has provided a comprehensive interface to the file system such as that
proposed in the Filesystem Library Proposal.

      </p>
      <p>

Should the library working group choose to adopt
the <code>wchar_t</code> overloads for file stream classes despite the
problems noted above, we propose the following resolution.

      </p>
      <p>

Add a declaration of a new <code>ios_base::openmode</code>
bit, <code>ios_base::cvtname</code>, to the definition of
class <code>ios_base</code> in <sref ref="ios.base"/>:

      </p>
      <blockquote>
        <pre>
typedef T3 openmode;
static const openmode app;
static const openmode ate;
static const openmode binary;
static const openmode in;
static const openmode out;
static const openmode trunc;
<ins>static const openmode cvtname;</ins>
        </pre>
      </blockquote>
      <p>

Add a row (the last row below) to Table 117: <code>openmode</code>
effects in <sref ref="ios::openmode"/>:

      </p>
      <blockquote>

      Table 117: <code>openmode</code> effects
      <table border="1">
        <thead>
          <tr>
            <th>Element</th>
            <th>Effect(s) if set</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td><tt>app</tt></td>
            <td>seek to end before each write</td>
          </tr>
          <tr>
            <td><tt>ate</tt></td>
            <td>open and seek to end immediately after opening</td>
          </tr>
          <tr>
            <td><tt>binary</tt></td>
            <td>
              perform input and output in binary mode
              (as opposed to text mode)</td>
          </tr>
          <tr>
            <td><tt>in</tt></td>
            <td>open for input</td>
          </tr>
          <tr>
            <td><tt>out</tt></td>
            <td>open for output</td>
          </tr>
          <tr>
            <td><tt>trunc</tt></td>
            <td>truncate an existing stream when opening</td>
          </tr>
          <tr>
            <td><ins><tt>cvtname</tt></ins></td>
            <td>
              <ins>convert a file name according to the rules
                of <code>codecvt</code> before opening</ins>
            </td>
          </tr>
        </tbody>
      </table>
      </blockquote>
      <p>

Add the following member function declarations to the definition of
class template <code>basic_filebuf</code> in <sref ref="filebuf"/>:

      </p>
      <blockquote>
        <pre>
// <i><sref ref="filebuf.members"/> Members</i>
bool is_open() const;
basic_filebuf&lt;charT,traits&gt;* open (const char *s,
                                   ios_base::openmode mode);
<ins>basic_filebuf&lt;charT,traits&gt;* open (const char *ws,</ins>
                                   <ins>ios_base::openmode mode);</ins>
basic_filebuf&lt;charT,traits&gt;* open (const string &amp;s,
                                   ios_base::openmode mode);
<ins>basic_filebuf&lt;charT,traits&gt;* open (const wstring &amp;ws,</ins>
                                   <ins>ios_base::openmode mode);</ins>
        </pre>
      </blockquote>
      <p>

Change <sref ref="filebuf.members"/>, p2 as follows:

      </p>
      <blockquote>
        <pre>

basic_filebuf&lt;charT,traits&gt;*
open (const char* s, ios_base::openmode mode);
        </pre>
        <p>

<i>Effects</i>: If <code>is_open() != false</code>, returns a null
pointer.  Otherwise, <del>initializes the filebuf as required. It then
</del>if <code>(mode &amp; ios_base::cvtname)
!= <ins>0</ins><del>false</del></code>, the
function <del>then</del>opens a file, if possible, whose name is the
NTBS <code>s</code> ("as if" by
calling <code>std::fopen(s, modstr))</code>. The
NTBS <code>modstr</code> is determined from <code>mode &amp; ~<ins>(</ins>ios_base::ate<ins> | ios_base::cvtname)</ins></code> as
indicated in Table 120. <ins>If <code>(mode &amp; ios_base::cvtname) ==
0</code>, the behavior of the function is
implementation-defined.</ins>

        </p>
      </blockquote>
      <p>

[Note: the phrase "initializes the filebuf as required" above has been
stricken since the object is already initialized and cannot be
re-initialized in an ordinary member function.]

      </p>
      <p>

Add a new paragraph immediately after <sref ref="filebuf.members"/>, p2, above Table 123, with the following text:

      </p>
      <blockquote>
        <pre>
      
basic_filebuf&lt;charT,traits&gt;*
open (const wchar_t* ws, ios_base::openmode mode);
        </pre>
        <p>

<i>Effects</i>: If <code>is_open() != false</code>, returns a null
pointer.  Otherwise, if <code>(mode &amp; ios_base::cvtname) !=
0</code>, converts the WCBS <code>ws</code> to an MBCS <code>s</code>
"as if" by a call to <code>a_codecvt.out()</code>. If the conversion
succeeds, the function returns <code>open(s, mode &amp;
~ios_base::cvtname)</code>. Otherwise, if the conversion fails, the
function fails. If <code>(mode &amp; ios_base::cvtname) == 0</code>,
the behavior of the function is implementation-defined.

        </p>
      </blockquote>
      <p>

[Note: the symbol <code>a_codecvt</code> referenced above is an
exposition-only member of class <code>basic_filebuf</code> defined
in <sref ref="filebuf"/>, p5.]

      </p>
      <p>

Add another new paragraph below the one above:

      </p>
      <blockquote>
        <p>

<i>Note</i>: In both overloads of <code>open()</code>,
when <code>(mode &amp; ios_base::cvtname) == 0</code>, implementations
are expected to convert the file name argument to the "natural"
representation for the system. For a system that "naturally"
represents a filename as a WCBS , the NTBS <code>s</code> in the first
signature is expected to be mapped to a WCBS; if so, it follows the
same mapping rules as the first argument to <code>open()</code>.

        </p>
      </blockquote>
      <p>

Also add the corresponding overloads to class
templates <code>basic_ifstream</code> in <sref ref="ifstream"/>,
<sref ref="ifstream.cons"/>, <sref ref="ofstream"/>,
and <code>basic_ofstream</code> in <sref ref="ofstream.cons"/>, as
follows.

      </p>
      [an error occurred while processing this directive]
      <p>

Add the following member function declarations to the definition of
class template <code>basic_ifstream</code> in <sref ref="ifstream"/>:

      </p>
      <blockquote>
        <pre>
// <i><sref ref="ifstream.cons"/>Constructors</i>
basic_ifstream ();
explicit basic_ifstream (const char* s,
                         ios_base::openmode mode = ios_base::in);
<ins>explicit basic_ifstream (const wchar_t* s,</ins>
                         <ins>ios_base::openmode mode = ios_base::in);</ins>
explicit basic_ifstream (const string &amp;s,
                         ios_base::openmode mode = ios_base::in);
<ins>explicit basic_ifstream (const wstring &amp;s,</ins>
                         <ins>ios_base::openmode mode = ios_base::in);</ins>

// <i><sref ref="ifstream.members"/>Members</i>
basic_ifstream&lt;charT, traits&gt;* rdbuf() const;

bool is_open() const;
void open (const char *s, ios_base::openmode mode = ios_base::in);
<ins>void open (const wchar_t *s, ios_base::openmode mode = ios_base::in);</ins>
void open (const string&amp; ios_base::openmode mode = ios_base::in);
<ins>void open (const wstring &amp; ios_base::openmode mode = ios_base::in);</ins>
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="ifstream.cons"/>, p2:

      </p>
      <blockquote>
        <pre>
explicit basic_ifstream (const wchar_t* s,
                         ios_base::openmode mode = ios_base::in);
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="ifstream.cons"/>, p3:

      </p>
      <blockquote>
        <pre>
explicit basic_ifstream (const wstring &amp;s,
                         ios_base::openmode mode = ios_base::in);
        </pre>
      </blockquote>
      <p>

Add the following signature below the first declaration (but above
the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="ifstream.members"/>, p3:

      </p>
      <blockquote>
        <pre>
void open (const wchar_t *s,
           ios_base::openmode mode = ios_base::in);
        </pre>
      </blockquote>
      <p>

Add the following signature immediately below the second declaration
(but above the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="ifstream.members"/>, p4:

      </p>
      <blockquote>
        <pre>
void open (const wstring &amp;s,
           ios_base::openmode mode = ios_base::in);
        </pre>
      </blockquote>
      <p>

[Note: the effects as well as the names of the formal function
parameters of the added <code>wchar_t</code> overloads are exactly the
same as those of the existing <code>char</code> functions and so don't
need to be spelled out separately from the existing <i>Effects</i>
clauses.]

      </p>
      [an error occurred while processing this directive]
      <p>

Add the following member function declarations to the definition of
class template <code>basic_ofstream</code> in <sref ref="ofstream"/>:

      </p>
      <blockquote>
        <pre>
// <i><sref ref="ofstream.cons"/>Constructors</i>
basic_ofstream ();
explicit basic_ofstream (const char* s,
                         ios_base::openmode mode = ios_base::out);
<ins>explicit basic_ofstream (const wchar_t* s,</ins>
                         <ins>ios_base::openmode mode = ios_base::out);</ins>
explicit basic_ofstream (const string &amp;s,
                         ios_base::openmode mode = ios_base::out);
<ins>explicit basic_ofstream (const wstring &amp;s,</ins>
                         <ins>ios_base::openmode mode = ios_base::out);</ins>

// <i><sref ref="ofstream.members"/>Members</i>
basic_isftream&lt;charT, traits&gt;* rdbuf() const;

bool is_open() const;
void open (const char *s, ios_base::openmode mode = ios_base::out);
<ins>void open (const wchar_t *s, ios_base::openmode mode = ios_base::out);</ins>
void open (const string &amp; ios_base::openmode mode = ios_base::out);
<ins>void open (const wstring &amp; ios_base::openmode mode = ios_base::out);</ins>
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="ofstream.cons"/>, p2:

      </p>
      <blockquote>
        <pre>
explicit basic_ofstream (const wchar_t* s,
                         ios_base::openmode mode = ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="ofstream.cons"/>, p3:

      </p>
      <blockquote>
        <pre>
explicit basic_ofstream (const wstring &amp;s,
                         ios_base::openmode mode = ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature below the first declaration (but above
the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="ofstream.members"/>, p3:

      </p>
      <blockquote>
        <pre>
void open (const wchar_t *s,
           ios_base::openmode mode = ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature immediately below the second declaration
(but above the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="ofstream.members"/>, p4:

      </p>
      <blockquote>
        <pre>
void open (const wstring &amp;s,
           ios_base::openmode mode = ios_base::out);
        </pre>
      </blockquote>
      <p>

[Note: the effects as well as the names of the formal function
parameters of the added <code>wchar_t</code> overloads are exactly the
same as those of the existing <code>char</code> functions and so don't
need to be spelled out separately from the existing <i>Effects</i>
clauses.]

      </p>
      [an error occurred while processing this directive]
      <p>

Add the following member function declarations to the definition of
class template <code>basic_fstream</code> in <sref ref="fstream"/>:

      </p>
      <blockquote>
        <pre>
// <i><sref ref="fstream.cons"/>Constructors</i>
basic_fstream ();
explicit basic_fstream (const char* s,
                        ios_base::openmode mode = ios_base::in | ios_base::out);
<ins>explicit basic_fstream (const wchar_t* s,</ins>
                       <ins>ios_base::openmode mode = ios_base::in | ios_base::out);</ins>
explicit basic_fstream (const string &amp;s,
                        ios_base::openmode mode = ios_base::in | ios_base::out);
<ins>explicit basic_fstream (const wstring &amp;s,</ins>
                       <ins>ios_base::openmode mode = ios_base::in | ios_base::out);</ins>

// <i><sref ref="fstream.members"/>Members</i>
basic_fstream&lt;charT, traits&gt;* rdbuf() const;

bool is_open() const;
void open (const char *s, ios_base::openmode mode = ios_base::in | ios_base::out);
<ins>void open (const wchar_t *s, ios_base::openmode mode = ios_base::in | ios_base::out);</ins>
void open (const string &amp; ios_base::openmode mode = ios_base::in | ios_base::out);
<ins>void open (const wstring &amp; ios_base::openmode mode = ios_base::in | ios_base::out);</ins>
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="fstream.cons"/>, p2:

      </p>
      <blockquote>
        <pre>
explicit basic_fstream (const wchar_t* s,
                        ios_base::openmode mode = ios_base::in | ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature below the declaration (but above
the <i>Effects</i> clause) of the constructor in
<sref ref="fstream.cons"/>, p3:

      </p>
      <blockquote>
        <pre>
explicit basic_fstream (const wstring &amp;s,
                        ios_base::openmode mode = ios_base::in | ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature below the first declaration (but above
the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="fstream.members"/>, p3:

      </p>
      <blockquote>
        <pre>
void open (const wchar_t *s,
           ios_base::openmode mode = ios_base::in | ios_base::out);
        </pre>
      </blockquote>
      <p>

Add the following signature immediately below the second declaration
(but above the <i>Effects</i> clause) of <code>open()</code> in
<sref ref="fstream.members"/>, p4:

      </p>
      <blockquote>
        <pre>
void open (const wstring &amp;s,
           ios_base::openmode mode = ios_base::in | ios_base::out);
        </pre>
      </blockquote>
      <p>

[Note: the effects as well as the names of the formal function
parameters of the added <code>wchar_t</code> overloads are exactly the
same as those of the existing <code>char</code> functions and so don't
need to be spelled out separately from the existing <i>Effects</i>
clauses.]

      </p>
    </resolution>
  </body>
</html>
