<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <title>d1795r1.html</title>
  <meta name="generator" content="Haroopad 0.13.1" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  <style>div.oembedall-githubrepos{border:1px solid #DDD;border-radius:4px;list-style-type:none;margin:0 0 10px;padding:8px 10px 0;font:13.34px/1.4 helvetica,arial,freesans,clean,sans-serif;width:452px;background-color:#fff}div.oembedall-githubrepos .oembedall-body{background:-moz-linear-gradient(center top,#FAFAFA,#EFEFEF);background:-webkit-gradient(linear,left top,left bottom,from(#FAFAFA),to(#EFEFEF));border-bottom-left-radius:4px;border-bottom-right-radius:4px;border-top:1px solid #EEE;margin-left:-10px;margin-top:8px;padding:5px 10px;width:100%}div.oembedall-githubrepos h3{font-size:14px;margin:0;padding-left:18px;white-space:nowrap}div.oembedall-githubrepos p.oembedall-description{color:#444;font-size:12px;margin:0 0 3px}div.oembedall-githubrepos p.oembedall-updated-at{color:#888;font-size:11px;margin:0}div.oembedall-githubrepos ul.oembedall-repo-stats{border:none;float:right;font-size:11px;font-weight:700;padding-left:15px;position:relative;z-index:5;margin:0}div.oembedall-githubrepos ul.oembedall-repo-stats li{border:none;color:#666;display:inline-block;list-style-type:none;margin:0!important}div.oembedall-githubrepos ul.oembedall-repo-stats li a{background-color:transparent;border:none;color:#666!important;background-position:5px -2px;background-repeat:no-repeat;border-left:1px solid #DDD;display:inline-block;height:21px;line-height:21px;padding:0 5px 0 23px}div.oembedall-githubrepos ul.oembedall-repo-stats li:first-child a{border-left:medium none;margin-right:-3px}div.oembedall-githubrepos ul.oembedall-repo-stats li a:hover{background:5px -27px no-repeat #4183C4;color:#FFF!important;text-decoration:none}div.oembedall-githubrepos ul.oembedall-repo-stats li:first-child a:hover{border-bottom-left-radius:3px;border-top-left-radius:3px}ul.oembedall-repo-stats li:last-child a:hover{border-bottom-right-radius:3px;border-top-right-radius:3px}span.oembedall-closehide{background-color:#aaa;border-radius:2px;cursor:pointer;margin-right:3px}div.oembedall-container{margin-top:5px;text-align:left}.oembedall-ljuser{font-weight:700}.oembedall-ljuser img{vertical-align:bottom;border:0;padding-right:1px}.oembedall-stoqembed{border-bottom:1px dotted #999;float:left;overflow:hidden;width:730px;line-height:1;background:#FFF;color:#000;font-family:Arial,Liberation Sans,DejaVu Sans,sans-serif;font-size:80%;text-align:left;margin:0;padding:0}.oembedall-stoqembed a{color:#07C;text-decoration:none;margin:0;padding:0}.oembedall-stoqembed a:hover{text-decoration:underline}.oembedall-stoqembed a:visited{color:#4A6B82}.oembedall-stoqembed h3{font-family:Trebuchet MS,Liberation Sans,DejaVu Sans,sans-serif;font-size:130%;font-weight:700;margin:0;padding:0}.oembedall-stoqembed .oembedall-reputation-score{color:#444;font-size:120%;font-weight:700;margin-right:2px}.oembedall-stoqembed .oembedall-user-info{height:35px;width:185px}.oembedall-stoqembed .oembedall-user-info .oembedall-user-gravatar32{float:left;height:32px;width:32px}.oembedall-stoqembed .oembedall-user-info .oembedall-user-details{float:left;margin-left:5px;overflow:hidden;white-space:nowrap;width:145px}.oembedall-stoqembed .oembedall-question-hyperlink{font-weight:700}.oembedall-stoqembed .oembedall-stats{background:#EEE;margin:0 0 0 7px;padding:4px 7px 6px;width:58px}.oembedall-stoqembed .oembedall-statscontainer{float:left;margin-right:8px;width:86px}.oembedall-stoqembed .oembedall-votes{color:#555;padding:0 0 7px;text-align:center}.oembedall-stoqembed .oembedall-vote-count-post{font-size:240%;color:#808185;display:block;font-weight:700}.oembedall-stoqembed .oembedall-views{color:#999;padding-top:4px;text-align:center}.oembedall-stoqembed .oembedall-status{margin-top:-3px;padding:4px 0;text-align:center;background:#75845C;color:#FFF}.oembedall-stoqembed .oembedall-status strong{color:#FFF;display:block;font-size:140%}.oembedall-stoqembed .oembedall-summary{float:left;width:635px}.oembedall-stoqembed .oembedall-excerpt{line-height:1.2;margin:0;padding:0 0 5px}.oembedall-stoqembed .oembedall-tags{float:left;line-height:18px}.oembedall-stoqembed .oembedall-tags a:hover{text-decoration:none}.oembedall-stoqembed .oembedall-post-tag{background-color:#E0EAF1;border-bottom:1px solid #3E6D8E;border-right:1px solid #7F9FB6;color:#3E6D8E;font-size:90%;line-height:2.4;margin:2px 2px 2px 0;padding:3px 4px;text-decoration:none;white-space:nowrap}.oembedall-stoqembed .oembedall-post-tag:hover{background-color:#3E6D8E;border-bottom:1px solid #37607D;border-right:1px solid #37607D;color:#E0EAF1}.oembedall-stoqembed .oembedall-fr{float:right}.oembedall-stoqembed .oembedall-statsarrow{background-image:url(http://cdn.sstatic.net/stackoverflow/img/sprites.png?v=3);background-repeat:no-repeat;overflow:hidden;background-position:0 -435px;float:right;height:13px;margin-top:12px;width:7px}.oembedall-facebook1{border:1px solid #1A3C6C;padding:0;font:13.34px/1.4 verdana;width:500px}.oembedall-facebook2{background-color:#627add}.oembedall-facebook2 a{color:#e8e8e8;text-decoration:none}.oembedall-facebookBody{background-color:#fff;vertical-align:top;padding:5px}.oembedall-facebookBody .contents{display:inline-block;width:100%}.oembedall-facebookBody div img{float:left;margin-right:5px}div.oembedall-lanyard{-webkit-box-shadow:none;-webkit-transition-delay:0s;-webkit-transition-duration:.4000000059604645s;-webkit-transition-property:width;-webkit-transition-timing-function:cubic-bezier(0.42,0,.58,1);background-attachment:scroll;background-clip:border-box;background-color:transparent;background-image:none;background-origin:padding-box;border-width:0;box-shadow:none;color:#112644;display:block;float:left;font-family:'Trebuchet MS',Trebuchet,sans-serif;font-size:16px;height:253px;line-height:19px;margin:0;max-width:none;min-height:0;outline:#112644 0;overflow-x:visible;overflow-y:visible;padding:0;position:relative;text-align:left;vertical-align:baseline;width:804px}div.oembedall-lanyard .tagline{font-size:1.5em}div.oembedall-lanyard .wrapper{overflow:hidden;clear:both}div.oembedall-lanyard .split{float:left;display:inline}div.oembedall-lanyard .prominent-place .flag:active,div.oembedall-lanyard .prominent-place .flag:focus,div.oembedall-lanyard .prominent-place .flag:hover,div.oembedall-lanyard .prominent-place .flag:link,div.oembedall-lanyard .prominent-place .flag:visited{float:left;display:block;width:48px;height:48px;position:relative;top:-5px;margin-right:10px}div.oembedall-lanyard .place-context{font-size:.889em}div.oembedall-lanyard .prominent-place .sub-place{display:block}div.oembedall-lanyard .prominent-place{font-size:1.125em;line-height:1.1em;font-weight:400}div.oembedall-lanyard .main-date{color:#8CB4E0;font-weight:700;line-height:1.1}div.oembedall-lanyard .first{width:48.57%;margin:0 0 0 2.857%}.mermaid .label{color:#333}.node circle,.node polygon,.node rect{fill:#cde498;stroke:#13540c;stroke-width:1px}.edgePath .path{stroke:green;stroke-width:1.5px}.cluster rect{fill:#cdffb2;rx:40;stroke:#6eaa49;stroke-width:1px}.cluster text{fill:#333}.actor{stroke:#13540c;fill:#cde498}text.actor{fill:#000;stroke:none}.actor-line{stroke:grey}.messageLine0{stroke-width:1.5;stroke-dasharray:"2 2";marker-end:"url(#arrowhead)";stroke:#333}.messageLine1{stroke-width:1.5;stroke-dasharray:"2 2";stroke:#333}#arrowhead{fill:#333}#crosshead path{fill:#333!important;stroke:#333!important}.messageText{fill:#333;stroke:none}.labelBox{stroke:#326932;fill:#cde498}.labelText,.loopText{fill:#000;stroke:none}.loopLine{stroke-width:2;stroke-dasharray:"2 2";marker-end:"url(#arrowhead)";stroke:#326932}.note{stroke:#6eaa49;fill:#fff5ad}.noteText{fill:#000;stroke:none;font-family:'trebuchet ms',verdana,arial;font-size:14px}.section{stroke:none;opacity:.2}.section0,.section2{fill:#6eaa49}.section1,.section3{fill:#fff;opacity:.2}.sectionTitle0,.sectionTitle1,.sectionTitle2,.sectionTitle3{fill:#333}.sectionTitle{text-anchor:start;font-size:11px;text-height:14px}.grid .tick{stroke:lightgrey;opacity:.3;shape-rendering:crispEdges}.grid path{stroke-width:0}.today{fill:none;stroke:red;stroke-width:2px}.task{stroke-width:2}.taskText{text-anchor:middle;font-size:11px}.taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px}.taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}.taskText0,.taskText1,.taskText2,.taskText3{fill:#fff}.task0,.task1,.task2,.task3{fill:#487e3a;stroke:#13540c}.taskTextOutside0,.taskTextOutside1,.taskTextOutside2,.taskTextOutside3{fill:#000}.active0,.active1,.active2,.active3{fill:#cde498;stroke:#13540c}.activeText0,.activeText1,.activeText2,.activeText3{fill:#000!important}.done0,.done1,.done2,.done3{stroke:grey;fill:lightgrey;stroke-width:2}.doneText0,.doneText1,.doneText2,.doneText3{fill:#000!important}.crit0,.crit1,.crit2,.crit3{stroke:#f88;fill:red;stroke-width:2}.activeCrit0,.activeCrit1,.activeCrit2,.activeCrit3{stroke:#f88;fill:#cde498;stroke-width:2}.doneCrit0,.doneCrit1,.doneCrit2,.doneCrit3{stroke:#f88;fill:lightgrey;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}.activeCritText0,.activeCritText1,.activeCritText2,.activeCritText3,.doneCritText0,.doneCritText1,.doneCritText2,.doneCritText3{fill:#000!important}.titleText{text-anchor:middle;font-size:18px;fill:#000}text{font-family:'trebuchet ms',verdana,arial;font-size:14px}html{height:100%}body{margin:0!important;padding:5px 20px 26px!important;background-color:#fff;font-family:"Lucida Grande","Segoe UI","Apple SD Gothic Neo","Malgun Gothic","Lucida Sans Unicode",Helvetica,Arial,sans-serif;font-size:.9em;overflow-x:hidden;overflow-y:auto}br,h1,h2,h3,h4,h5,h6{clear:both}hr.page{background:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAYAAAAECAYAAACtBE5DAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyJpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMC1jMDYwIDYxLjEzNDc3NywgMjAxMC8wMi8xMi0xNzozMjowMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENTNSBNYWNpbnRvc2giIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6OENDRjNBN0E2NTZBMTFFMEI3QjRBODM4NzJDMjlGNDgiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6OENDRjNBN0I2NTZBMTFFMEI3QjRBODM4NzJDMjlGNDgiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo4Q0NGM0E3ODY1NkExMUUwQjdCNEE4Mzg3MkMyOUY0OCIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo4Q0NGM0E3OTY1NkExMUUwQjdCNEE4Mzg3MkMyOUY0OCIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PqqezsUAAAAfSURBVHjaYmRABcYwBiM2QSA4y4hNEKYDQxAEAAIMAHNGAzhkPOlYAAAAAElFTkSuQmCC) repeat-x;border:0;height:3px;padding:0}hr.underscore{border-top-style:dashed!important}body >:first-child{margin-top:0!important}img.plugin{box-shadow:0 1px 3px rgba(0,0,0,.1);border-radius:3px}iframe{border:0}figure{-webkit-margin-before:0;-webkit-margin-after:0;-webkit-margin-start:0;-webkit-margin-end:0}kbd{border:1px solid #aaa;-moz-border-radius:2px;-webkit-border-radius:2px;border-radius:2px;-moz-box-shadow:1px 2px 2px #ddd;-webkit-box-shadow:1px 2px 2px #ddd;box-shadow:1px 2px 2px #ddd;background-color:#f9f9f9;background-image:-moz-linear-gradient(top,#eee,#f9f9f9,#eee);background-image:-o-linear-gradient(top,#eee,#f9f9f9,#eee);background-image:-webkit-linear-gradient(top,#eee,#f9f9f9,#eee);background-image:linear-gradient(top,#eee,#f9f9f9,#eee);padding:1px 3px;font-family:inherit;font-size:.85em}.oembeded .oembed_photo{display:inline-block}img[data-echo]{margin:25px 0;width:100px;height:100px;background:url(../img/ajax.gif) center center no-repeat #fff}.spinner{display:inline-block;width:10px;height:10px;margin-bottom:-.1em;border:2px solid rgba(0,0,0,.5);border-top-color:transparent;border-radius:100%;-webkit-animation:spin 1s infinite linear;animation:spin 1s infinite linear}.spinner:after{content:'';display:block;width:0;height:0;position:absolute;top:-6px;left:0;border:4px solid transparent;border-bottom-color:rgba(0,0,0,.5);-webkit-transform:rotate(45deg);transform:rotate(45deg)}@-webkit-keyframes spin{to{-webkit-transform:rotate(360deg)}}@keyframes spin{to{transform:rotate(360deg)}}p.toc{margin:0!important}p.toc ul{padding-left:10px}p.toc>ul{padding:10px;margin:0 10px;display:inline-block;border:1px solid #ededed;border-radius:5px}p.toc li,p.toc ul{list-style-type:none}p.toc li{width:100%;padding:0;overflow:hidden}p.toc li a::after{content:"."}p.toc li a:before{content:"• "}p.toc h5{text-transform:uppercase}p.toc .title{float:left;padding-right:3px}p.toc .number{margin:0;float:right;padding-left:3px;background:#fff;display:none}input.task-list-item{margin-left:-1.62em}.markdown{font-family:"Hiragino Sans GB","Microsoft YaHei",STHeiti,SimSun,"Lucida Grande","Lucida Sans Unicode","Lucida Sans",'Segoe UI',AppleSDGothicNeo-Medium,'Malgun Gothic',Verdana,Tahoma,sans-serif;padding:20px}.markdown a{text-decoration:none;vertical-align:baseline}.markdown a:hover{text-decoration:underline}.markdown h1{font-size:2.2em;font-weight:700;margin:1.5em 0 1em}.markdown h2{font-size:1.8em;font-weight:700;margin:1.275em 0 .85em}.markdown h3{font-size:1.6em;font-weight:700;margin:1.125em 0 .75em}.markdown h4{font-size:1.4em;font-weight:700;margin:.99em 0 .66em}.markdown h5{font-size:1.2em;font-weight:700;margin:.855em 0 .57em}.markdown h6{font-size:1em;font-weight:700;margin:.75em 0 .5em}.markdown h1+p,.markdown h1:first-child,.markdown h2+p,.markdown h2:first-child,.markdown h3+p,.markdown h3:first-child,.markdown h4+p,.markdown h4:first-child,.markdown h5+p,.markdown h5:first-child,.markdown h6+p,.markdown h6:first-child{margin-top:0}.markdown hr{border:1px solid #ccc}.markdown p{margin:1em 0;word-wrap:break-word}.markdown ol{list-style-type:decimal}.markdown li{display:list-item;line-height:1.4em}.markdown blockquote{margin:1em 20px}.markdown blockquote>:first-child{margin-top:0}.markdown blockquote>:last-child{margin-bottom:0}.markdown blockquote cite:before{content:'\2014 \00A0'}.markdown .code{border-radius:3px;word-wrap:break-word}.markdown pre{border-radius:3px;word-wrap:break-word;border:1px solid #ccc;overflow:auto;padding:.5em}.markdown pre code{border:0;display:block}.markdown pre>code{font-family:Consolas,Inconsolata,Courier,monospace;font-weight:700;white-space:pre;margin:0}.markdown code{border-radius:3px;word-wrap:break-word;border:1px solid #ccc;padding:0 5px;margin:0 2px}.markdown img{max-width:100%}.markdown mark{color:#000;background-color:#fcf8e3}.markdown table{padding:0;border-collapse:collapse;border-spacing:0;margin-bottom:16px}.markdown table tr td,.markdown table tr th{border:1px solid #ccc;margin:0;padding:6px 13px}.markdown table tr th{font-weight:700}.markdown table tr th>:first-child{margin-top:0}.markdown table tr th>:last-child{margin-bottom:0}.markdown table tr td>:first-child{margin-top:0}.markdown table tr td>:last-child{margin-bottom:0}@import url(http://fonts.googleapis.com/css?family=Roboto+Condensed:300italic,400italic,700italic,400,300,700);.haroopad{padding:20px;color:#222;font-size:15px;font-family:"Roboto Condensed",Tauri,"Hiragino Sans GB","Microsoft YaHei",STHeiti,SimSun,"Lucida Grande","Lucida Sans Unicode","Lucida Sans",'Segoe UI',AppleSDGothicNeo-Medium,'Malgun Gothic',Verdana,Tahoma,sans-serif;background:#fff;line-height:1.6;-webkit-font-smoothing:antialiased}.haroopad a{color:#3269a0}.haroopad a:hover{color:#4183c4}.haroopad h2{border-bottom:1px solid #e6e6e6}.haroopad h6{color:#777}.haroopad hr{border:1px solid #e6e6e6}.haroopad blockquote>code,.haroopad h1>code,.haroopad h2>code,.haroopad h3>code,.haroopad h4>code,.haroopad h5>code,.haroopad h6>code,.haroopad li>code,.haroopad p>code,.haroopad td>code{font-family:Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:85%;background-color:rgba(0,0,0,.02);padding:.2em .5em;border:1px solid #efefef}.haroopad pre>code{font-size:1em;letter-spacing:-1px;font-weight:700}.haroopad blockquote{border-left:4px solid #e6e6e6;padding:0 15px;color:#777}.haroopad table{background-color:#fafafa}.haroopad table tr td,.haroopad table tr th{border:1px solid #e6e6e6}.haroopad table tr:nth-child(2n){background-color:#f2f2f2}.hljs{display:block;overflow-x:auto;padding:.5em;background:#fdf6e3;color:#657b83;-webkit-text-size-adjust:none}.diff .hljs-header,.hljs-comment,.hljs-doctype,.hljs-javadoc,.hljs-pi,.lisp .hljs-string{color:#93a1a1}.css .hljs-tag,.hljs-addition,.hljs-keyword,.hljs-request,.hljs-status,.hljs-winutils,.method,.nginx .hljs-title{color:#859900}.hljs-command,.hljs-dartdoc,.hljs-hexcolor,.hljs-link_url,.hljs-number,.hljs-phpdoc,.hljs-regexp,.hljs-rules .hljs-value,.hljs-string,.hljs-tag .hljs-value,.tex .hljs-formula{color:#2aa198}.css .hljs-function,.hljs-built_in,.hljs-chunk,.hljs-decorator,.hljs-id,.hljs-identifier,.hljs-localvars,.hljs-title,.vhdl .hljs-literal{color:#268bd2}.hljs-attribute,.hljs-class .hljs-title,.hljs-constant,.hljs-link_reference,.hljs-parent,.hljs-type,.hljs-variable,.lisp .hljs-body,.smalltalk .hljs-number{color:#b58900}.css .hljs-pseudo,.diff .hljs-change,.hljs-attr_selector,.hljs-cdata,.hljs-header,.hljs-pragma,.hljs-preprocessor,.hljs-preprocessor .hljs-keyword,.hljs-shebang,.hljs-special,.hljs-subst,.hljs-symbol,.hljs-symbol .hljs-string{color:#cb4b16}.hljs-deletion,.hljs-important{color:#dc322f}.hljs-link_label{color:#6c71c4}.tex .hljs-formula{background:#eee8d5}.MathJax_Hover_Frame{border-radius:.25em;-webkit-border-radius:.25em;-moz-border-radius:.25em;-khtml-border-radius:.25em;box-shadow:0 0 15px #83A;-webkit-box-shadow:0 0 15px #83A;-moz-box-shadow:0 0 15px #83A;-khtml-box-shadow:0 0 15px #83A;border:1px solid #A6D!important;display:inline-block;position:absolute}.MathJax_Hover_Arrow{position:absolute;width:15px;height:11px;cursor:pointer}#MathJax_About{position:fixed;left:50%;width:auto;text-align:center;border:3px outset;padding:1em 2em;background-color:#DDD;color:#000;cursor:default;font-family:message-box;font-size:120%;font-style:normal;text-indent:0;text-transform:none;line-height:normal;letter-spacing:normal;word-spacing:normal;word-wrap:normal;white-space:nowrap;float:none;z-index:201;border-radius:15px;-webkit-border-radius:15px;-moz-border-radius:15px;-khtml-border-radius:15px;box-shadow:0 10px 20px gray;-webkit-box-shadow:0 10px 20px gray;-moz-box-shadow:0 10px 20px gray;-khtml-box-shadow:0 10px 20px gray;filter:progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}.MathJax_Menu{position:absolute;background-color:#fff;color:#000;width:auto;padding:2px;border:1px solid #CCC;margin:0;cursor:default;font:menu;text-align:left;text-indent:0;text-transform:none;line-height:normal;letter-spacing:normal;word-spacing:normal;word-wrap:normal;white-space:nowrap;float:none;z-index:201;box-shadow:0 10px 20px gray;-webkit-box-shadow:0 10px 20px gray;-moz-box-shadow:0 10px 20px gray;-khtml-box-shadow:0 10px 20px gray;filter:progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}.MathJax_MenuItem{padding:2px 2em;background:0 0}.MathJax_MenuArrow{position:absolute;right:.5em;color:#666}.MathJax_MenuActive .MathJax_MenuArrow{color:#fff}.MathJax_MenuArrow.RTL{left:.5em;right:auto}.MathJax_MenuCheck{position:absolute;left:.7em}.MathJax_MenuCheck.RTL{right:.7em;left:auto}.MathJax_MenuRadioCheck{position:absolute;left:1em}.MathJax_MenuRadioCheck.RTL{right:1em;left:auto}.MathJax_MenuLabel{padding:2px 2em 4px 1.33em;font-style:italic}.MathJax_MenuRule{border-top:1px solid #CCC;margin:4px 1px 0}.MathJax_MenuDisabled{color:GrayText}.MathJax_MenuActive{background-color:Highlight;color:HighlightText}.MathJax_Menu_Close{position:absolute;width:31px;height:31px;top:-15px;left:-15px}#MathJax_Zoom{position:absolute;background-color:#F0F0F0;overflow:auto;display:block;z-index:301;padding:.5em;border:1px solid #000;margin:0;font-weight:400;font-style:normal;text-align:left;text-indent:0;text-transform:none;line-height:normal;letter-spacing:normal;word-spacing:normal;word-wrap:normal;white-space:nowrap;float:none;box-shadow:5px 5px 15px #AAA;-webkit-box-shadow:5px 5px 15px #AAA;-moz-box-shadow:5px 5px 15px #AAA;-khtml-box-shadow:5px 5px 15px #AAA;filter:progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}#MathJax_ZoomOverlay{position:absolute;left:0;top:0;z-index:300;display:inline-block;width:100%;height:100%;border:0;padding:0;margin:0;background-color:#fff;opacity:0;filter:alpha(opacity=0)}#MathJax_ZoomFrame{position:relative;display:inline-block;height:0;width:0}#MathJax_ZoomEventTrap{position:absolute;left:0;top:0;z-index:302;display:inline-block;border:0;padding:0;margin:0;background-color:#fff;opacity:0;filter:alpha(opacity=0)}.MathJax_Preview{color:#888}#MathJax_Message{position:fixed;left:1px;bottom:2px;background-color:#E6E6E6;border:1px solid #959595;margin:0;padding:2px 8px;z-index:102;color:#000;font-size:80%;width:auto;white-space:nowrap}#MathJax_MSIE_Frame{position:absolute;top:0;left:0;width:0;z-index:101;border:0;margin:0;padding:0}.MathJax_Error{color:#C00;font-style:italic}footer{position:fixed;font-size:.8em;text-align:right;bottom:0;margin-left:-25px;height:20px;width:100%}</style>
</head>
<body class="markdown haroopad">
<h1 id="p1795r1:-system-topology-discovery-for-heterogeneous-&amp;-distributed-computing"><a name="p1795r1:-system-topology-discovery-for-heterogeneous-&amp;-distributed-computing" href="#p1795r1:-system-topology-discovery-for-heterogeneous-&amp;-distributed-computing"></a>P1795r1: System topology discovery for heterogeneous &amp; distributed computing</h1><p><strong>Date: 2019-10-07</strong></p><p><strong>Audience: SG1, SG14</strong></p><p><strong>Authors: Gordon Brown, Ruyman Reyes, Michael Wong, Mark Hoemmen, Jeff Hammond, Tom Scogland, Domagoj Šarić</strong></p><p><strong>Emails: gordon@codeplay.com, ruyman@codeplay.com, michael@codeplay.com, mhoemme@sandia.gov, jeff.science@gmail.com, tscogland@llnl.gov, domagoj.saric@microblink.com</strong></p><p><strong>Reply to: gordon@codeplay.com</strong></p><h1 id="acknowledgements"><a name="acknowledgements" href="#acknowledgements"></a>Acknowledgements</h1><p>This paper is the result of discussions from many contributors within the heterogeneous C++ group, including H. Carter Edwards, Thomas Rodgers, Patrice Roy, Carl Cook, Jeff Hammond, Hartmut Kaiser, Christian Trott, Paul Blinzer, Alex Voicu, Nat Goodspeed and Tony Tye.</p><h1 id="changelog"><a name="changelog" href="#changelog"></a>Changelog</h1><h3 id="p1437r1-(bel-2019)"><a name="p1437r1-(bel-2019)" href="#p1437r1-(bel-2019)"></a>P1437r1 (BEL 2019)</h3><ul>
<li>Introduce terms of art for <em>system topology</em>, <em>system resource</em> and <em>topology traversal policy</em>.</li><li>Introduce minimal design for <code>system_topology</code> class.</li><li>Introduce minimal design for <code>system_resource</code> class.</li><li>Introduce free function <code>this_system::discover_topology</code> for performing  runtime system topology discovery.</li><li>Introduce free function <code>traverse_topology</code> for traversing a <code>system_topology</code> using a <em>topology traversal policy</em> to return a collection of <code>execution_resource</code>s,</li></ul><h3 id="p1437r0-(col-2019)"><a name="p1437r0-(col-2019)" href="#p1437r0-(col-2019)"></a>P1437r0 (COL 2019)</h3><ul>
<li>Split off from <a href="http://wg21.link/p0796">[17]</a>, focussing on a mechanism for discovering the topology and affinity properties of a given system.</li><li>Temporarily remove the proposed wording.</li><li>Update the front matter to re-focus the motivation and goals of the paper.</li></ul><h3 id="changelog-from-p0796"><a name="changelog-from-p0796" href="#changelog-from-p0796"></a>Changelog from P0796</h3><p>For the earlier changelogs from prior to the split from P0796 see Appendix A.</p><h1 id="preface"><a name="preface" href="#preface"></a>Preface</h1><p>This paper is the result of a request from SG1 at the 2018 San Diego meeting to split <a href="http://wg21.link/p0796">[17]</a> into two separate papers, one for the high-level interface and one for the low-level interface. This paper focusses on the low-level interface; a mechanism for discovering the topology and affinity properties of a given system. <a href="http://wg21.link/p1436">[18]</a> focusses on the high-level interface, a series of properties for querying affinity relationships and requesting affinity on work being executed.</p><h1 id="1.-background"><a name="1.-background" href="#1.-background"></a>1. Background</h1><p>Computer systems are no longer homogeneous platforms. From desktop workstations to high-performance supercomputers, and from mobile devices to purpose-built embedded SoCs, every system has some form of co-processor along side the traditional multi-core CPU, and often more than one. Furthermore, the architectures of these co-processors range from many-core CPUs, GPUs, FPGAs and DSPs to specifically designed vision and machine learning processors. In larger supercomputer systems there are thousands of these processors in some configuration of nodes, connected physically or via network adapters.</p><p>The way these processors access memory is also far from homogeneous. For example, the system may present a single shared virtual address space <a href="https://www.kernel.org/doc/html/v4.18/vm/hmm.html">[21]</a> <a href="https://www.khronos.org/registry/OpenCL/sdk/2.1/docs/man/xhtml/sharedVirtualMemory.html">[22]</a>, or it may have different address spaces mutually inaccessible other than through special functions <a href="https://www.khronos.org/registry/OpenCL/specs/opencl-2.2.pdf">[4]</a>. Different memory regions may have different levels of consistency, cache coherency, and support for atomic operations. Different parts of the system may have different access latencies or bandwidths to different memory regions (so-called “NUMA affinity regions”) <a href="https://www.open-mpi.org/projects/hwloc/">[2]</a>. Some parts of memory may be persistent. Different systems may configure the same types of memory in different ways around the processors.</p><p>In order to program these new systems and the architectures that inhabit them, it’s vital that applications are capable of understating both what architectures are available and the properties of those architectures, namely their observable behaviors, capabilities and limitations. However, the current C++ standard provides no way to achieve this, so developers have to rely entirely on third party and operating system libraries.</p><h1 id="2.-goals:-what-this-paper-is,-and-what-it-is-not"><a name="2.-goals:-what-this-paper-is,-and-what-it-is-not" href="#2.-goals:-what-this-paper-is,-and-what-it-is-not"></a>2. Goals: what this paper is, and what it is not</h1><p>This paper seeks to define, within C++, a facility for discovering execution resources available to a system that are capable of executing work, and for querying their properties.</p><p>However, it is not the goal of this proposal to introduce support in the C++ language or the standard library for all of the various heterogeneous architectures available today. The authors of this paper recognize that this is unrealistic as it would require significant changes to the C++ machine model and would be extremely volatile to future developments in architecture and system design.</p><p>Instead, it seeks to define a single, unified, and stable layer in the C++ Standard Library. Applications, libraries, and programming models (such as SYCL <a href="https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf">[3]</a>, Kokkos <a href="https://github.com/kokkos/kokkos">[19]</a>, HPX <a href="https://github.com/STEllAR-GROUP/hpx">[13]</a> or TBB <a href="https://www.threadingbuildingblocks.org/">[12]</a>) can build on this layer; hardware vendors can support it via standards such as OpenCL <a href="https://www.khronos.org/registry/OpenCL/specs/opencl-2.2.pdf">[4]</a>, CUDA <a href="https://developer.nvidia.com/cuda-zone">[20]</a>, OpenMP <a href="http://www.openmp.org/wp-content/uploads/openmp-TR5-final.pdf">[6]</a>, MPI <a href="http://mpi-forum.org/docs/">[16]</a>, Hwloc <a href="https://www.open-mpi.org/projects/hwloc/">[2]</a>, HSA <a href="http://www.hsafoundation.com/standards/">[5]</a> and HMM <a href="https://www.kernel.org/doc/html/v4.18/vm/hmm.html">[21]</a>; and it can be extended when necessary.</p><p>This layer will not be characterized in terms of specific categories of hardware such as CPUs, GPUs and FPGAs as these are broad concepts that are subject to change over time and have no foundation in the C++ machine model. It will instead define a number of abstract properties of system architectures that are not tied to any specific hardware.</p><p>The initial set of properties that this paper would propose be defined in the C++ standard library would reflect a generalization of the observable behaviors, capabilities and limitations of common architectures available in heterogeneous and distributed systems today. However the intention is that the interface be extensible so that that vendors can provide their own extensions to provide visibility into the more niche characteristics of certain architectures.</p><p>It is intended that this layer be defined as a natural extension of the Executors proposal, a unified interface for execution. The current executors proposal [[14]][p0443r11] already provides a route to supporting heterogeneous and distributed systems, however it is missing a way to identify what architectures a system has.</p><h1 id="3.-motivation"><a name="3.-motivation" href="#3.-motivation"></a>3. Motivation</h1><p>There are many reasons why such a feature within C++ would benefit developers and the C++ ecosystem as a whole, and those can differ from one domain to another. We’ve attempted to outline some of these benefits here.</p><h2 id="improve-performance"><a name="improve-performance" href="#improve-performance"></a>Improve performance</h2><p>The clearest benefit is performance. Exposing, even at an abstract level, the properties of the underlying architecture that a program is running on, allows application and libraries to be fine tuned. This may result in significant performance improvements that would only otherwise be possible via third party or operating system libraries <a href="https://link.springer.com/chapter/10.1007/978-3-642-30961-8_2">[1]</a> <a href="https://github.com/dcdillon/cpuaff">[7]</a> <a href="https://github.com/memkind/memkind">[8]</a> <a href="https://docs.oracle.com/cd/E26502_01/html/E29031/pbind-1m.html">[9]</a> <a href="https://linux.die.net/man/2/sched_setaffinity">[10]</a> <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms686247(v=vs.85).aspx">[11]</a> <a href="https://docs.google.com/viewer?a=v&amp;pid=sites&amp;srcid=bGJsLmdvdnxwYWRhbC13b3Jrc2hvcHxneDozOWE0MjZjOTMxOTk3NGU3">[15]</a>.</p><p>This includes but is not limited to how to structure data to ensure access patterns along with execution on the architecture to achieve coalesced memory access and optimal cache utilization and where to initialize data to make efficient use of hardware locality and process affinity.</p><p>There is a general trend to move towards a unified address space in heterogeneous and distributed systems via standards like HMM. However, there are still many architectures that still require distinct address spaces, are not yet in a position to move to a single address space, and may never be. Even if you were to consider a single unified address the ultimate goal for heterogeneous and distributed systems, this actually makes the case for affinity in C++ stronger. As long as different address spaces exist, the distinction between different hardware memory regions and their capabilities is clear, but with a single unified address space, potentially with cache coherency, distinguishing different memory regions becomes much more subtle. Therefore, it becomes much more important to understand the various memory regions and their affinity relationships in order to achieve good performance on various architectures.</p><h2 id="provide-a-unified-interface"><a name="provide-a-unified-interface" href="#provide-a-unified-interface"></a>Provide a unified interface</h2><p>C++ is a major language when it comes to heterogeneous and distributed computing, and while it is a rapidly growing domain, it is still very challenging to develop in. There are a large number of C++ based third party and OS libraries. However, developing for heterogeneous and distributed systems often involves a combination of these libraries, which introduces a number of challenges</p><p>Firstly it’s common that different architectures are discovered via different libraries, You may want to use CUDA for NVidia GPUs, OpenMP for Intel CPUs, SYCL for Intel GPUs, Hwloc for the higher-level nodes and so on. This means that you have to collect together resources discovered from a different libraries, which very often do not provide a consistent representation or any form of interoperability, and find some way for them to represent them in a coherent view.</p><p>Secondly, many of these libraries report the same underlying hardware. For example OpenMP, SYCL and Hwloc will all report the same Intel CPU. This means you have to collate the resources together such that resources from different libraries representing the same hardware are joined together, to avoid resource contention.</p><h2 id="categorize-limitations"><a name="categorize-limitations" href="#categorize-limitations"></a>Categorize limitations</h2><p>There are many architectures available within heterogeneous and distributed systems which cannot support the full range of C++ features. This includes, but is not limited to dynamic allocation, recursion, dynamic polymorphism, RTTI, double precision floating point and some forms of atomic operations.</p><p>It’s crucial to allow developers to identify these limitations and which apply to the architecture they are running on, because in many cases if a C++ feature that is not supported on the architecture is used, the application would fail to execute or potentially crash.</p><h2 id="facilitate-generic-code"><a name="facilitate-generic-code" href="#facilitate-generic-code"></a>Facilitate generic code</h2><p>Developing algorithms for heterogeneous and distributed systems requires at least an abstract understanding of the underlying architecture(s) being targeted in order to achieve optimal performance. In some cases, it may call for a much more in-depth understanding. This means that each algorithm may require a different implementation on each architecture.</p><p>Another factor here is that in many heterogeneous and distributed programming models, the architectures available on a particular system are not known until runtime, where the topology of the system is discovered.</p><p>Having a unified interface for performing this topology discovery and querying the properties of the architectures available on a system would dramatically improve developers’ ability to write generic algorithms.</p><h2 id="increase-accessibility"><a name="increase-accessibility" href="#increase-accessibility"></a>Increase accessibility</h2><p>Providing support for heterogenous and distributed computing as a first-class citizen of C++ will improve its accessibility and increase its utilization in libraries and applications, ultimately making the ecosystem stronger. This will become increasingly more important as heterogeneous and distributed computing becomes crucial to gaining the necessary performance in applications in more domains of C++. </p><h2 id="provide-a-broader-standardization"><a name="provide-a-broader-standardization" href="#provide-a-broader-standardization"></a>Provide a broader standardization</h2><p>The C++ standard is in a crucial position for heterogeneous and distributed computing domains. It is the common point between a number of different programming languages, models and libraries targeting a wide range of different architectures. This means that C++ has a unique opportunity to provide a single standard that not only covers the requirements of a single domain, but all of them, allowing for a convergence within the ecosystem and much more interoperability across different architectures.</p><p>For example, a unified C++ interface for topology discovery could provide access to GPUs from Nvidia, AMD, Intel, and ARM via their respective open standards or proprietary frameworks. At the same time, it could give access to NUMA-aware systems via Hwloc.</p><p>Another example of this is that while Hwloc is highly used in many domains, it now does not always accurately represent existing systems. This is because Hwloc presents their topology as strictly hierarchical, which no longer accurately describes many systems. A unified C++ interface does not need to be bound to the limitations of a single library, and can provide a much broader representation of a system’s execution resource topology.</p><h1 id="5.-proposed-direction"><a name="5.-proposed-direction" href="#5.-proposed-direction"></a>5. Proposed direction</h1><p>This paper aims to build on the unified executors proposal, detailed in P0443 [[14]][p0443r11], so this proposal and any others that stem from it will target P0443 as a baseline, and aim to integrate with its direction as closely as possible.</p><p>Below we outline a proposed direction:</p><ul>
<li><p>Propose an abstract definition of an execution resource, as a hardware or software abstraction capable of creating execution agents.</p>
</li><li><p>Propose an abstract definition of an execution resource topology, as a non-hierarchical, non-acyclic graph of execution resources with various different kinds of connections.</p>
</li><li><p>Propose an interface for inspecting a system’s execution resource topology from various different perspectives, including, but not limited to:</p>
<ul>
<li>A depth-based view of the system as a containment hierarchy.</li><li>A memory-centric view of which execution resources may access which memory regions, and the properties of that access.</li><li>A network-centric view of how the execution resources and memory regions are connected.</li></ul>
</li><li><p>Propose an interface for querying properties of an execution resource, reflecting the observable behaviors, capabilities and limitations of the architecture the execution resource represents.</p>
</li><li><p>Propose an interface for querying the relative affinity properties between different execution resources and memory regions within a system’s execution resource topology.</p>
</li><li><p>Propose an interface for binding execution agents and initialization of data to specific execution resources.</p>
</li></ul><p>As a result of the above this paper may also:</p><ul>
<li>Propose a lifetime model for execution agents.</li><li>Propose some additions to the C++ machine model to facilitate describing these additional properties.</li></ul><h1 id="6.-proposal"><a name="6.-proposal" href="#6.-proposal"></a>6. Proposal</h1><h2 id="header-``-synopsis"><a name="header-``-synopsis" href="#header-``-synopsis"></a>Header <code>&lt;system&gt;</code> synopsis</h2><pre class="cpp hljs"><code class="cpp" data-origin="<pre><code class=&quot;cpp&quot;>namespace experimental {

/* system_topology */

class system_topology {

  system_topology() = delete;

};

/* system_resource */

class system_resource {

  /* to be defined */

};

/* traverse_topology */

template &amp;lt;class T&amp;gt;
to-be-decided&amp;lt;system_resource&amp;gt; traverse_topology(const system_topology &amp;amp;, const T &amp;amp;) noexcept;

/* this_system::discover_topology */

namespace this_system {

system_topology discover_topology();

}  // namespace this_system

}  // experimental
</code></pre>"><span class="hljs-keyword">namespace</span> experimental {

<span class="hljs-comment">/* system_topology */</span>

<span class="hljs-keyword">class</span> system_topology {

  system_topology() = <span class="hljs-keyword">delete</span>;

};

<span class="hljs-comment">/* system_resource */</span>

<span class="hljs-keyword">class</span> system_resource {

  <span class="hljs-comment">/* to be defined */</span>

};

<span class="hljs-comment">/* traverse_topology */</span>

<span class="hljs-keyword">template</span> &lt;<span class="hljs-keyword">class</span> T&gt;
to-be-decided&lt;system_resource&gt; traverse_topology(<span class="hljs-keyword">const</span> system_topology &amp;, <span class="hljs-keyword">const</span> T &amp;) <span class="hljs-keyword">noexcept</span>;

<span class="hljs-comment">/* this_system::discover_topology */</span>

<span class="hljs-keyword">namespace</span> this_system {

<span class="hljs-function">system_topology <span class="hljs-title">discover_topology</span><span class="hljs-params">()</span></span>;

}  <span class="hljs-comment">// namespace this_system</span>

}  <span class="hljs-comment">// experimental</span>
</code></pre><h2 id="terms-of-art"><a name="terms-of-art" href="#terms-of-art"></a>Terms of art</h2><p>The term <em>system resource</em> refers to a hardware or software abstraction of an execution, memory, network or I/O resource within a system.</p><p>The term <em>system topology</em> refers to a possibly cyclic graph of <em>execution resources</em> connected to the abstract machine, and their various properties.</p><blockquote>
<p>[<em>Note:</em> The current definition of <em>system topology</em> is currently incomplete and will be developed over the course of this proposal as the various C++ domains are represented. <em>—end note</em>]</p>
</blockquote><p>The term <em>topology traversal policy</em> refers to a policy that describes the way in which a <em>system topology</em> is traversed in order to to produce a collection of <em>system resources</em>.</p><h2 id="class-`system_topology`"><a name="class-`system_topology`" href="#class-`system_topology`"></a>Class <code>system_topology</code></h2><p>The <code>system_topology</code> class provides an abstraction of a read-only snapshot of the <em>system topology</em> at a particular point in time. A <code>system_topology</code> object may not maintain or otherwise be associated with the lifetime of operating system or third party library resources.</p><h3 id="`system_topology`-constructors"><a name="`system_topology`-constructors" href="#`system_topology`-constructors"></a><code>system_topology</code> constructors</h3><pre class="cpp hljs"><code class="cpp" data-origin="<pre><code class=&quot;cpp&quot;>system_topology() = delete;
</code></pre>">system_topology() = <span class="hljs-keyword">delete</span>;
</code></pre><p><em>Effects:</em> Explicitly deleted.</p><h2 id="class-`system_resource`"><a name="class-`system_resource`" href="#class-`system_resource`"></a>Class <code>system_resource</code></h2><p>The <code>system_resource</code> class provides an abstraction of a read-only snapshot of a <em>system resource</em> from the <em>system topology</em> at a particular point in time. A <code>system_resource</code> object may not maintain or otherwise be associated with the lifetime of operating system or third party library resources.</p><blockquote>
<p>[<em>Note:</em> The <code>system_resource</code> class is intended to reflect the properties of a <em>system resource</em> and it’s relationships with other <em>system resources</em>, however the precise definition is still to be decided. <em>—end note</em>]</p>
</blockquote><h2 id="free-functions"><a name="free-functions" href="#free-functions"></a>Free functions</h2><h3 id="`this_system::discover_topology`"><a name="`this_system::discover_topology`" href="#`this_system::discover_topology`"></a><code>this_system::discover_topology</code></h3><p>The free function <code>this_system::discover_topology</code> performs runtime discovery of the <em>system topology</em> and returns a <code>system_topology</code> object.</p><pre class="cpp hljs"><code class="cpp" data-origin="<pre><code class=&quot;cpp&quot;>namespace this_system {
  system_topology discover_topology();
}  // namespace this_system
</code></pre>"><span class="hljs-keyword">namespace</span> this_system {
  <span class="hljs-function">system_topology <span class="hljs-title">discover_topology</span><span class="hljs-params">()</span></span>;
}  <span class="hljs-comment">// namespace this_system</span>
</code></pre><p><em>Returns:</em> A <code>system_topology</code> object representing a snapshot of the <em>system topology</em> at the current point in time.</p><p><em>Requires:</em> Calls to <code>this_system::discover_topology()</code> may not introduce a data race with any other call to <code>this_system::discover_topology()</code>.</p><p><em>Effects:</em> Performs runtime discovery of the system topology and constructs a <code>system_topology</code> object. May invoke the operating system or third party libraries in discovering topology information, but must release any resources acquired for this purpose before returning.</p><p><em>Throws:</em> Any exception thrown as a result of performing runtime discovery of the system topology.</p><h3 id="`traverse_topology`"><a name="`traverse_topology`" href="#`traverse_topology`"></a><code>traverse_topology</code></h3><p>The free function <code>traverse_topology</code> performs a traversal of a <code>system_topology</code> object using a <em>topology traversal policy</em> specified by the tag type <code>T</code> and returns a sequence of <code>system_resource</code> objects.</p><pre class="cpp hljs"><code class="cpp" data-origin="<pre><code class=&quot;cpp&quot;>template &amp;lt;class T&amp;gt;
to-be-decided&amp;lt;system_resource&amp;gt; traverse_topology(const system_topology &amp;amp;, const T &amp;amp;) noexcept;
</code></pre>"><span class="hljs-keyword">template</span> &lt;<span class="hljs-keyword">class</span> T&gt;
to-be-decided&lt;system_resource&gt; traverse_topology(<span class="hljs-keyword">const</span> system_topology &amp;, <span class="hljs-keyword">const</span> T &amp;) <span class="hljs-keyword">noexcept</span>;
</code></pre><p><em>Returns:</em> A sequence of <code>system_resource</code> objects representing the <em>system resources</em> matching the criteria of the <em>topology traversal policy</em>.</p><p><em>Effects:</em> Traverses the <code>system_topology</code> object provided and identifies any <em>system resources</em> which match the criteria of the <em>topology traversal policy</em>, adding a single <code>system_resource</code> to the sequence returned for each match found.</p><p><em>Throws:</em> May not throw.</p><blockquote>
<p>[<em>Note:</em> The exact representation of <em>system resources</em> returned by <code>traverse_topology</code> is still to be decided as this will have implications on lifetimes. One option is to return a container of <em>system_resource</em> objects by-value such as a <code>vector</code>, however this would require some form of reference counting. Another option is to return a reference to a reference to the <em>system_resource</em> objects via a   <code>span</code> or a <code>ranges::view</code>, however this would require the <code>system_topology</code> object to remain alive. <em>—end note</em>]</p>
</blockquote><h1 id="7.-open-questions"><a name="7.-open-questions" href="#7.-open-questions"></a>7. Open questions</h1><ul>
<li>How granular should topology discovery be, should they whole topology be discovered in a single operation or should it be done in multiple nested operations, only discovering what is needed at each layer?</li><li>What kind of <em>topology traversal policies</em> would people list to see standardized?</li><li>How should we support notification of a topology update, polling or callback?</li><li>Should we also provide an interface for compile-time topology discovery?</li></ul><h1 id="references"><a name="references" href="#references"></a>References</h1><p><a href="https://link.springer.com/chapter/10.1007/978-3-642-30961-8_2">[1]</a> The Design of OpenMP Thread Affinity</p><p><a href="https://www.open-mpi.org/projects/hwloc/">[2]</a> Portable Hardware Locality</p><p><a href="https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf">[3]</a> SYCL 1.2.1</p><p><a href="https://www.khronos.org/registry/OpenCL/specs/opencl-2.2.pdf">[4]</a> OpenCL 2.2</p><p><a href="http://www.hsafoundation.com/standards/">[5]</a> HSA</p><p><a href="http://www.openmp.org/wp-content/uploads/openmp-TR5-final.pdf">[6]</a> OpenMP 5.0</p><p><a href="https://github.com/dcdillon/cpuaff">[7]</a> cpuaff</p><p><a href="https://github.com/memkind/memkind">[8]</a> MEMKIND</p><p><a href="https://docs.oracle.com/cd/E26502_01/html/E29031/pbind-1m.html">[9]</a> Solaris pbind()</p><p><a href="https://linux.die.net/man/2/sched_setaffinity">[10]</a> Linux sched_setaffinity() </p><p><a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms686247(v=vs.85).aspx">[11]</a> Windows SetThreadAffinityMask()</p><p><a href="https://www.threadingbuildingblocks.org/">[12]</a> TBB</p><p><a href="https://github.com/STEllAR-GROUP/hpx">[13]</a> HPX</p><p>[p0443r11]:<br><a href="http://wg21.link/p0443r11">http://wg21.link/p0443r11</a><br>[[14]][p0443r11] A Unified Executors Proposal for C++</p><p><a href="https://docs.google.com/viewer?a=v&amp;pid=sites&amp;srcid=bGJsLmdvdnxwYWRhbC13b3Jrc2hvcHxneDozOWE0MjZjOTMxOTk3NGU3">[15]</a> Exposing the Locality of new Memory Hierarchies to HPC Applications</p><p><a href="http://mpi-forum.org/docs/">[16]</a> MPI</p><p><a href="http://wg21.link/p0796">[17]</a> Supporting Heterogeneous &amp; Distributed Computing Through Affinity</p><p><a href="http://wg21.link/p1436">[18]</a> Executor properties for affinity-based execution</p><p><a href="https://github.com/kokkos/kokkos">[19]</a> Kokkos project</p><p><a href="https://developer.nvidia.com/cuda-zone">[20]</a> CUDA</p><p><a href="https://www.kernel.org/doc/html/v4.18/vm/hmm.html">[21]</a> Heterogeneous Memory Management</p><p><a href="https://www.khronos.org/registry/OpenCL/sdk/2.1/docs/man/xhtml/sharedVirtualMemory.html">[22]</a> OpenCL 2.x Shared Virtual Memory</p><h1 id="appendix-a:-changelog-from-p0796"><a name="appendix-a:-changelog-from-p0796" href="#appendix-a:-changelog-from-p0796"></a>Appendix A: Changelog from P0796</h1><h2 id="p0796r3-(san-2018)"><a name="p0796r3-(san-2018)" href="#p0796r3-(san-2018)"></a>P0796r3 (SAN 2018)</h2><ul>
<li>Remove reference counting requirement from <code>execution_resource</code>.</li><li>Change lifetime model of <code>execution_resource</code>: it now either consistently identifies some underlying resource, or is invalid; context creation rejects an invalid resource.ster</li><li>Remove <code>this_thread::bind</code> &amp; <code>this_thread::unbind</code> interfaces.</li><li>Make <code>execution_resource</code>s iterable by replacing <code>execution_resource::resources</code> with <code>execution_resource::begin</code> and <code>execution_resource::end</code>.</li><li>Add <code>size</code> and <code>operator[]</code> for <code>execution_resource</code>.</li><li>Rename <code>this_system::get_resources</code> to <code>this_system::discover_topology</code>.</li><li>Introduce <code>memory_resource</code> to represent the memory component of a system topology.</li><li>Remove <code>can_place_memory</code> and <code>can_place_agents</code> from the <code>execution_resource</code> as these are no longer required.</li><li>Remove <code>memory_resource</code> and <code>allocator</code> from the <code>execution_context</code> as these no longer make sense.</li><li>Update the wording to describe how execution resources and memory resources are structured.</li><li>Refactor <code>affinity_query</code> to be between an <code>execution_resource</code> and a <code>memory_resource</code>.</li></ul><h2 id="p0796r2-(rap-2018)"><a name="p0796r2-(rap-2018)" href="#p0796r2-(rap-2018)"></a>P0796r2 (RAP 2018)</h2><ul>
<li>Introduce a free function for retrieving the execution resource underlying the current thread of execution.</li><li>Introduce <code>this_thread::bind</code> &amp; <code>this_thread::unbind</code> for binding and unbinding a thread of execution to an execution resource.</li><li>Introduce <code>bulk_execution_affinity</code> executor properties for specifying affinity binding patterns on bulk execution functions.</li></ul><h2 id="p0796r1-(jax-2018)"><a name="p0796r1-(jax-2018)" href="#p0796r1-(jax-2018)"></a>P0796r1 (JAX 2018)</h2><ul>
<li>Introduce proposed wording.</li><li>Based on feedback from SG1, introduce a pair-wise interface for querying the relative affinity between execution resources.</li><li>Introduce an interface for retrieving an allocator or polymorphic memory resource.</li><li>Based on feedback from SG1, remove requirement for a hierarchical system topology structure, which doesn’t require a root resource.</li></ul><h2 id="p0796r0-(abq-2017)"><a name="p0796r0-(abq-2017)" href="#p0796r0-(abq-2017)"></a>P0796r0 (ABQ 2017)</h2><ul>
<li>Initial proposal.</li><li>Enumerate design space, hierarchical affinity, issues to the committee.</li></ul>

<footer style="position:fixed; font-size:.8em; text-align:right; bottom:0px; margin-left:-25px; height:20px; width:100%;">generated by <a href="http://pad.haroopress.com" target="_blank">haroopad</a></footer>
</body>
</html>
