Refereeing in an outliner [0089]

Most scientific conferences solicit and organize reviews for papers using a web platform such as HotCRP and EasyChair; although these are not the same, the idea is similar. Once you have been assigned to referee a paper, you will receive a web form with several sections and large text areas in which to put the components of your review; not all conferences ask for the same components, but usually one is expected to include the following in addition to your (numerical) assessment of the paper’s merit and your expertise:

  1. A summmary of the paper
  2. Your assessment of the paper
  3. Detailed comments for the authors
  4. Questions to be addressed by author response
  5. Comments for the PC (program committee) and other reviewers

Usually you will be asked to enter your comments under each section in a plain text format like Markdown. The first thing a new referee learns is not to type answers directly into the web interface, because this is an extremely reliable way to lose hours of your time when a browser or server glitch deletes all your work. Most of us instead write out our answers in a separate text editor, and paste them into the web interface when we are satisfied with them. In the past, I have done this with text files on my computer, but today I want to discuss how to draft referee reports as outlines in Bike; then I will show you how to convert them to the correct plain text format that can be pasted into your conference’s preferred web-based refereeing platform.

To start off, have a look at the figure below, which shows my refereeing template outline in Bike.

Figure 1. A Bike outline for conference refereeing [0086]

This Bike outline contains the skeleton of a referee report for a major computer science conference. Bike’s row types are used extensively: for example, sections are formatted as headings, and prompts are formatted as notes

As you can see, there is a healthy combination of hierarchy and formatting in a Bike outline.

2. Rich text editing in Bike [008B]

Bike is a rich text editor, but one that (much like GNU TeXmacs) avoids the classic pitfalls of nearly all rich text editors, such as the ubiquitous and dreaded “Is the space italic?!” user-experience failure; I will not outline Bike innovative approach to rich text editing here, but I suggest you check it out for yourself.

3. Row types in Bike [008A]

One of the most useful features of Bike’s approach to formatting is the concept of a row type, which is a semantic property of a row that has consequences for its visual presentation. Bike currently supports the following row types:

  • Plain rows
  • Heading rows, formatted in boldface
  • Note rows, formatted in gray italics
  • Quote rows, formatted with a vertical bar to their left
  • Ordered rows, formatted with an (automatically chosen) numeral to their left
  • Unordered rows, formatted with a bullet to their left
  • Task rows, formatted with a checkbox to their left

My refereeing template uses several of these row types (headings, notes, tasks) as well as some of the rich text formatting (highlighting). When I fill out the refereeing outline, I will use other row types as well — including quotes, ordered, and unordered rows. I will create a subheading under Detailed comments for the author to contain my questions and comments, which I enter in as ordered rows; then I make a separate subheading at the same level for Typographical errors, which I populate with unordered rows. Unordered rows are best for typos, because they are always accompanied already by line numbers. If I need to quote an extended portion of the paper, I will use a quote row.

When working on the report outline, I will constantly be focusing on individual sections to avoid not only distractions but also the intense mental weight of unfinished sections. Focusing means that the entire outline is narrowed to a subtree that can be edited away from its context; this is achieved in Bike by pressing the gray south-easterly arrows to the right of each heading, as seen in the figure.

References

    Reference. Bike Outliner: Structured focused writing [0085]

    Use Bike to think, write, organize.

    Make lists, take notes, create documents.

    Bike’s an unusually fast outliner designed for your Mac.

    Reference. The jolly writer: your guide to GNU TeXmacs [van-der-hoeven-2020]

    Do you wish to write beautiful scientific documents with GNU TeXmacs? Then be guided by The Jolly Writer on your way to mastering the secrets of elegant layout and mathematical typography.

    The book covers the main features of the latest version of TeXmacs: its wysiwyg user interface, mathematics, tables, professional typography, technical pictures, laptop presentations, structured editing, interfaces to systems for symbolic and numeric computations, customizable style sheets, the versioning tool, and much more.

    The primary aim of The Jolly Writer is to guide you through this world of possibilities. While hacking your way through the jungle of style tags, keyboard shortcuts, mathematical homoglyphs, hanging punctuation, structured insertions, morphing animations, among others, you will slowly but surely unravel their mysteries and deeper why. At the end of the road, maybe you will become a jolly writer yourself.

    @book{van-der-hoeven-2020,
  title={The Jolly Writer},
  author={Van der Hoeven, Joris},
  year={2020},
  publisher={Scypress}
}

    Context

      Scientific refereeing using Bike Outliner [0084]

      I have long been an enthusiast for outliners, a genre of computer software that deserves more than almost any other to be called an “elegant weapon for a more civilized age”. Recently I have been enjoying experimenting with Jesse Grosjean’s highly innovative outliner for macOS called Bike, which builds on a lot of his previous (highly impressive) work in the area with a level of fit and finish that is rare even in the world of macOS software. Bike costs $29.99 and is well-worth it; watch the introductory video or try the demo to see for yourself.

      The purpose of outliners is to provide room to actively think; Grothendieck is said to have been unable to think at all without a pen in his hand, and I think of outliners as one way to elevate the tactile aspect of active thinking using the unique capabilities of software. Tools for thinking must combat stress and mental weight, and the most immediate way that outliners achieve this is through the ability to focus on a subtree — narrowing into a portion of the hierarchy and treating it as if it were the entire document, putting its context aside. This feature, which some of my readers may recognize from Emacs org-mode, is well-represented in Bike — without, of course, suffering the noticeable quirks that come from the Emacs environment, nor the ill-advised absolute/top-down model of hierarchy sadly adopted by org-mode.

      As a scientist in academia, one of the most frequent things I am doing when I am not writing my own papers or working with students is refereeing other scientists’ papers. For those who are unfamiliar, this means carefully studying a paper and then producing a detailed and well-structured report that includes a summary of the paper, my personal assessment of its scientific validity and value, and a long list of corrections, questions, comments, and suggestions. Referee reports of this kind are then used by journal editors and conference program committees to decide which papers deserve to be published.

      In this post, I will give an overview of my refereeing workflow with Bike and how I overcame the challenges transferring finished referee reports from Bike into the text-based formats used by conference refereeing platforms like HotCRP and EasyChair using a combination of XSLT 2.0 and Pandoc. This tutorial applies to Bike 1.14; I hope the format will not change too much, but I cannot make promises about what I do not control.

      Refereeing in an outliner [0089]

      Most scientific conferences solicit and organize reviews for papers using a web platform such as HotCRP and EasyChair; although these are not the same, the idea is similar. Once you have been assigned to referee a paper, you will receive a web form with several sections and large text areas in which to put the components of your review; not all conferences ask for the same components, but usually one is expected to include the following in addition to your (numerical) assessment of the paper’s merit and your expertise:

      1. A summmary of the paper
      2. Your assessment of the paper
      3. Detailed comments for the authors
      4. Questions to be addressed by author response
      5. Comments for the PC (program committee) and other reviewers

      Usually you will be asked to enter your comments under each section in a plain text format like Markdown. The first thing a new referee learns is not to type answers directly into the web interface, because this is an extremely reliable way to lose hours of your time when a browser or server glitch deletes all your work. Most of us instead write out our answers in a separate text editor, and paste them into the web interface when we are satisfied with them. In the past, I have done this with text files on my computer, but today I want to discuss how to draft referee reports as outlines in Bike; then I will show you how to convert them to the correct plain text format that can be pasted into your conference’s preferred web-based refereeing platform.

      To start off, have a look at the figure below, which shows my refereeing template outline in Bike.

      Figure. A Bike outline for conference refereeing [0086]

      This Bike outline contains the skeleton of a referee report for a major computer science conference. Bike’s row types are used extensively: for example, sections are formatted as headings, and prompts are formatted as notes

      As you can see, there is a healthy combination of hierarchy and formatting in a Bike outline.

      Rich text editing in Bike [008B]

      Bike is a rich text editor, but one that (much like GNU TeXmacs) avoids the classic pitfalls of nearly all rich text editors, such as the ubiquitous and dreaded “Is the space italic?!” user-experience failure; I will not outline Bike innovative approach to rich text editing here, but I suggest you check it out for yourself.

      Row types in Bike [008A]

      One of the most useful features of Bike’s approach to formatting is the concept of a row type, which is a semantic property of a row that has consequences for its visual presentation. Bike currently supports the following row types:

      • Plain rows
      • Heading rows, formatted in boldface
      • Note rows, formatted in gray italics
      • Quote rows, formatted with a vertical bar to their left
      • Ordered rows, formatted with an (automatically chosen) numeral to their left
      • Unordered rows, formatted with a bullet to their left
      • Task rows, formatted with a checkbox to their left

      My refereeing template uses several of these row types (headings, notes, tasks) as well as some of the rich text formatting (highlighting). When I fill out the refereeing outline, I will use other row types as well — including quotes, ordered, and unordered rows. I will create a subheading under Detailed comments for the author to contain my questions and comments, which I enter in as ordered rows; then I make a separate subheading at the same level for Typographical errors, which I populate with unordered rows. Unordered rows are best for typos, because they are always accompanied already by line numbers. If I need to quote an extended portion of the paper, I will use a quote row.

      When working on the report outline, I will constantly be focusing on individual sections to avoid not only distractions but also the intense mental weight of unfinished sections. Focusing means that the entire outline is narrowed to a subtree that can be edited away from its context; this is achieved in Bike by pressing the gray south-easterly arrows to the right of each heading, as seen in the figure.

      From an outline to a plain text referee report [008C]

      Although we have seen how pleasant it is to use an outliner like Bike to draft a referee report, but we obviously cannot submit a .bike file to a conference reviewing website or a journal editor. Most conference reviewing systems accept plain text or Markdown responses, and so our goal will be to convert a Bike outline into reasonably formatted Markdown.

      It happens that Bike’s underlying format is HTML, so one idea would be to use Pandoc to process this HTML into Markdown. This would work, except that Bike’s model is sufficiently structured that it must make deeply idiosyncratic use of HTML tags, as can be seen from the listing below.

      Listing. The source code to a typical Bike outline [008G] 

      <?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta charset="utf-8"/>
  </head>
  <body>
    <ul id="2sbcmmms">
      <li id="3C" data-type="heading">
        <p>Tasks</p>
        <ul>
          <li id="cs" data-type="task">
            <p>read through paper on iPad and highlight</p>
          </li>
        </ul>
      </li>
      <li id="XZ" data-type="heading">
        <p>Syntax and semantics of foo bar baz</p>
        <ul>
          <li id="Kp">
            <p><mark>Overall merit:</mark><span> </span></p>
          </li>
          <li id="d9">
            <p><mark>Reviewer Expertise:</mark></p>
          </li>
          <li id="uM" data-type="heading">
            <p>Summary of the paper</p>
            <ul>
              <li id="oB" data-type="note">
                <p>Please give a brief summary of the paper</p>
              </li>
              <li id="TWZ">
                <p>This paper describes the syntax and semantics of foo, bar, and baz.</p>
              </li>
            </ul>
          </li>
          <li id="V8" data-type="heading">
            <p>Assessment of the paper</p>
            <ul>
              <li id="vD" data-type="note">
                <p>Please give a balanced assessment of the paper's strengths and weaknesses and a clear justification for your review score.</p>
              </li>
            </ul>
          </li>
          <li id="zo" data-type="heading">
            <p>Detailed comments for authors</p>
            <ul>
              <li id="o0" data-type="note">
                <p>Please give here any additional detailed comments or questions that you would like the authors to address in revising the paper.</p>
              </li>
              <li id="bgy" data-type="heading">
                <p>Minor comments</p>
                <ul>
                  <li id="tMq" data-type="unordered">
                    <p>line 23: "teh" =&gt; "the"</p>
                  </li>
                  <li id="EmX" data-type="unordered">
                    <p>line 99: "fou" =&gt; "foo"</p>
                  </li>
                </ul>
              </li>
            </ul>
          </li>
          <li id="aN" data-type="heading">
            <p>Questions to be addressed by author response</p>
            <ul>
              <li id="7s" data-type="note">
                <p>Please list here any specific questions you would like the authors to address in their author response. Since authors have limited time in which to prepare their response, please only ask questions here that are likely to affect your accept/reject decision.</p>
              </li>
            </ul>
          </li>
          <li id="4S" data-type="heading">
            <p>Comments for PC and other reviewers</p>
            <ul>
              <li id="bN" data-type="note">
                <p>Please list here any additional comments you have which you want the PC and other reviewers to see, but not the authors.</p>
              </li>
              <li id="i2b">
                <p>In case any one is wondering, I am an expert in foo, but not in bar nor baz.</p>
              </li>
            </ul>
          </li>
        </ul>
      </li>
    </ul>
  </body>
</html>

      The Markdown that would result from postprocessing a Bike outline directly with Pandoc would be deeply unsuitable for submission. We will, however, use a version of this idea: first we will preprocess the Bike format into more conventional (unstructured) HTML using XSLT 2.0, and then we will use Pandoc to convert this into Markdown.

      System requirements to convert Bike outlines to Markdown [008E]

      XSLT 2.0 is unfortunately only implemented by proprietary tools like Saxon, developed by Saxonica. Nonetheless, it is possible to freely install Saxon on macOS using Homebrew:

      brew install saxon

      You must also install Pandoc, which is also conveniently available as a binary on Homebrew:

      brew install pandoc

      With the system requirements out of the way, we can proceed to prepare an XSLT stylesheet that will convert Bike’s idiosyncratic use of HTML tags to more conventional HTML that can be processed into Markdown by Pandoc. The stylesheet bike-to-html.xsl is described and explained in the listing below.

      Listing. An XSLT 2.0 transformer to convert Bike outlines to HTML [0087]

      We can write convert Bike outlines to reasonable HTML using an XSLT 2.0 stylesheet, bike-to-html.xsl detailed below.

      <?xml version="1.0"?>

<xsl:stylesheet version="2.0"
  xmlns="http://www.w3.org/1999/xhtml"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:html="http://www.w3.org/1999/xhtml"
  exclude-result-prefixes="xhtml">

  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
  <xsl:strip-space elements="*" />

      We will allow several tags to be copied verbatim into the output, as Bike uses these in the same way that idiomatic HTML does.

        <xsl:template match="html:html | html:body | html:code | html:strong | html:em | html:mark">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*" />
    </xsl:copy>
  </xsl:template>

      Bike leaves behind a lot of empty span elements; we drop these.

        <xsl:template match="html:span">
    <xsl:apply-templates />
  </xsl:template>

      Bike uses ul for all lists; the list type is determined not at this level, but rather by each individual item’s @data-type attribute. To get this data into the HTML list model, we must group items that have the same @data-type and wrap them in an appropriate list-forming element.

      To do this, we use XSLT 2.0’s xsl:for-each-group instruction to group adjacent li elements by their @data-type attribute. (It is extremely difficult and error-prone to write equivalent code in the more widely available XSLT 1.0.) We must convert @data-type to a string: otherwise, the transformer will crash when it encounters an item without a @data-type attribute.

        <xsl:template match="html:ul">
    <xsl:for-each-group select="html:li" group-adjacent="string(@data-type)">
      <xsl:choose>
        <xsl:when test="@data-type='ordered' or @data-type='task'">
          <ol>
            <xsl:apply-templates select="current-group()" />
          </ol>
        </xsl:when>
        <xsl:when test="@data-type='unordered'">
          <ul>
            <xsl:apply-templates select="current-group()" />
          </ul>
        </xsl:when>
        <xsl:otherwise>
          <xsl:apply-templates select="current-group()" />
        </xsl:otherwise>
      </xsl:choose>
    </xsl:for-each-group>
  </xsl:template>

      Next, we match each individual li element; the content of a list item is stored in a p element directly under li, so we let the transformer fall thorugh the parent and then format the content underneath according to the @data-type of the item.

        <xsl:template match="html:li">
    <xsl:apply-templates />
  </xsl:template>

  <xsl:template
    match="html:li[@data-type='ordered' or @data-type='unordered' or @data-type='task']/html:p">
    <li>
      <xsl:apply-templates />
    </li>
  </xsl:template>

      Bike has correctly adopted the optimal explicit and relative model of hierarchy, in contrast to HTML; this means that the depth of a heading is not reflected in the element that introduces it, but is instead inferred from its actual position in the outline hierarchy. To convert Bike outlines to idiomatic HTML, we must flatten the hierarchy and introduce explicit heading levels; luckily, this is easy to accomplish in XSLT by counting the ancestors of heading type.

        <xsl:template match="html:li[@data-type='heading']/html:p">
    <xsl:element
      name="h{count(ancestor::html:li[@data-type='heading'])}">
      <xsl:apply-templates />
    </xsl:element>
  </xsl:template>

      The remainder of the row types are not difficult to render; you may prefer alternative formatting depending on your goals.

        <xsl:template match="html:li[@data-type='quote']/html:p">
    <blockquote>
      <xsl:apply-templates />
    </blockquote>
  </xsl:template>

  <xsl:template match="html:li[@data-type='note']/html:p">
    <p>
      <em>
        <xsl:apply-templates />
      </em>
    </p>
  </xsl:template>

  <xsl:template match="html:li[not(@data-type)]/html:p">
    <p>
      <xsl:apply-templates />
    </p>
  </xsl:template>
</xsl:stylesheet>

      Next, we can use Saxon to convert a Bike outline to idiomatic HTML using the stylesheet above.

      cat review.bike | saxon -xsl:bike-to-html.xsl - > review.html

      Go ahead and open the resulting HTML file in a text editor and a browser to see the results.

      Listing. A Bike outline transformed to idiomatic HTML [008F]

      The following is the result of transforming an example Bike outline to idiomatic HTML using an XSLT stylesheet.

      <?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
   <body>
      <h1>Tasks</h1>
      <ol>
         <li>read through paper on iPad and highlight</li>
      </ol>
      <h1>Syntax and semantics of foo bar baz</h1>
      <p>
         <mark>Overall merit:</mark>
      </p>
      <p>
         <mark>Reviewer Expertise:</mark>
      </p>
      <h2>Summary of the paper</h2>
      <p>
         <em>Please give a brief summary of the paper</em>
      </p>
      <p>This paper describes the syntax and semantics of foo, bar, and baz.</p>
      <h2>Assessment of the paper</h2>
      <p>
         <em>Please give a balanced assessment of the paper's strengths and weaknesses and a clear justification for your review score.</em>
      </p>
      <h2>Detailed comments for authors</h2>
      <p>
         <em>Please give here any additional detailed comments or questions that you would like the authors to address in revising the paper.</em>
      </p>
      <h3>Minor comments</h3>
      <ul>
         <li>line 23: "teh" =&gt; "the"</li>
         <li>line 99: "fou" =&gt; "foo"</li>
      </ul>
      <h2>Questions to be addressed by author response</h2>
      <p>
         <em>Please list here any specific questions you would like the authors to address in their author response. Since authors have limited time in which to prepare their response, please only ask questions here that are likely to affect your accept/reject decision.</em>
      </p>
      <h2>Comments for PC and other reviewers</h2>
      <p>
         <em>Please list here any additional comments you have which you want the PC and other reviewers to see, but not the authors.</em>
      </p>
      <p>In case any one is wondering, I am an expert in foo, but not in bar nor baz.</p>
   </body>
</html>

      Next, we will process this HTML file using Pandoc; unfortunately, Pandoc leaves behind a lot of garbage character escapes that are not suitable for submission anywhere, so we must filter those out using sed.

      cat review.html | pandoc -f html -t markdown-raw_html-native_divs-native_spans-fenced_divs-bracketed_spans-smart | sed 's/\\//g'

      Listing. A Bike outline transformed to Markdown [008H]

      The following is the result of converting the idiomatic HTML representation of a Bike outline to Markdown using Pandoc, with some light postprocessing by sed.

      # Tasks

1.  read through paper on iPad and highlight

# Syntax and semantics of foo bar baz

Overall merit:

Reviewer Expertise:

## Summary of the paper

*Please give a brief summary of the paper*

This paper describes the syntax and semantics of foo, bar, and baz.

## Assessment of the paper

*Please give a balanced assessment of the paper's strengths and
weaknesses and a clear justification for your review score.*

## Detailed comments for authors

*Please give here any additional detailed comments or questions that you
would like the authors to address in revising the paper.*

### Minor comments

-   line 23: "teh" => "the"
-   line 99: "fou" => "foo"

## Questions to be addressed by author response

*Please list here any specific questions you would like the authors to
address in their author response. Since authors have limited time in
which to prepare their response, please only ask questions here that are
likely to affect your accept/reject decision.*

## Comments for PC and other reviewers

*Please list here any additional comments you have which you want the PC
and other reviewers to see, but not the authors.*

In case any one is wondering, I am an expert in foo, but not in bar nor
baz.

      We can compose all these tasks into a one-liner as follows:

      cat review.bike | saxon -xsl:bike-to-html.xsl - | pandoc -f html -t markdown-raw_html-native_divs-native_spans-fenced_divs-bracketed_spans-smart | sed 's/\\//g'

      A convenient Bike-to-Markdown script [008I]

      I have gathered the scripts to convert Bike outlines into Markdown via idiomatic HTML in a Git repository where they can be easily downloaded. If you have any improvements to these scripts, please submit them as a patch to my public inbox! I am also interested in whether it is possible to write the XSLT 2.0 stylesheet as equivalent XSLT 1.0, to avoid requiring the proprietary Saxon tool. Feel free also to send comments on this post to my public inbox, or discuss with me on Mastodon.

      ☕. Crowdfunding and sponsorship [0088]

      Apart from my day-job at the University of Cambridge, I am independently researching tools for scientific thought and developing software like Forester that you can use to unlock your brain. If you have benefited from this work or the writings on my blog, please considering supporting me with a sponsorship on Ko-fi.

      Related

        Rich text editing in Bike [008B]

        Bike is a rich text editor, but one that (much like GNU TeXmacs) avoids the classic pitfalls of nearly all rich text editors, such as the ubiquitous and dreaded “Is the space italic?!” user-experience failure; I will not outline Bike innovative approach to rich text editing here, but I suggest you check it out for yourself.

        Row types in Bike [008A]

        One of the most useful features of Bike’s approach to formatting is the concept of a row type, which is a semantic property of a row that has consequences for its visual presentation. Bike currently supports the following row types:

        • Plain rows
        • Heading rows, formatted in boldface
        • Note rows, formatted in gray italics
        • Quote rows, formatted with a vertical bar to their left
        • Ordered rows, formatted with an (automatically chosen) numeral to their left
        • Unordered rows, formatted with a bullet to their left
        • Task rows, formatted with a checkbox to their left

        Figure. A Bike outline for conference refereeing [0086]

        This Bike outline contains the skeleton of a referee report for a major computer science conference. Bike’s row types are used extensively: for example, sections are formatted as headings, and prompts are formatted as notes