Portable, Repeatable (, and Reproducible) Notebooks

Jupyter Notebooks is arguably the most common and prevalent notebook system out there, with many features. It supports many features such as different "kernels" to execute different languages, integrated documentation and code, and inline plotting. However, the underlying format is cumbersome and difficult to use in collaboration with other developers using traditional tools such as Git. For a larger list of issues with notebooks in general, see the Caveats section below.

As an alternative, Org Mode (with Emacs) offers all the desired features of notebooks, but similarly gets around some of the limitations typical notebook implementations impose. For example, the notebook is manipulated directly in your editor of choice(™); all creature comforts included. Moreover, Org Babel offers the expected seamless integration between documentation and code blocks and more. For example, the code blocks within a notebook can be "tangled" out and executed in a more typical command-line driven style. Multi-kernel notebooks are not common outside of a few extensions, such a feature is built directly into Org Mode's operation. Moreover, computations from one snippet can be input to another, in a different language. For example, see the Meta Programming Language documentation.

GNU Guix provides the necessary reproducibility. Specifically, using envrc and direnv, a manifest is loaded and shimmed into the current environment within Emacs. In this way, Emacs and other shell processes are using a specific environment which includes the necessary tools and libraries for the project. The same manifest is used to create a "really reproducible" package/archive which is deployed to an HPC cluster. This is all done using a channels file to version pin the entire dependency tree (down to the C compiler and GLibC).

For a recent conference paper, I needed to repeatedly run a few thousand different experiments and then the aggregation and analysis of those experiments. This was accomplished with an Org mode notebook. The major techniques discovered in the process are accessible properties, named example blocks, and liberal use of header arguments.

The following are series of exerts directly from the notebook highlighting each of these features.

The file was structured into several different headings, each for different parts of the process. The heading of the notebook had a few variables which are used throughout the notebook:

* notebook
:PROPERTIES:
:REMOTE: borah.boisestate.edu
:REMOTE-DIR: /bsuhome/kennyballou/scratch
:END:

Everything that interacts with the HPC cluster is then relative to these two variables. Another user can change these two variables and easily execute the notebook. For example, the following is a batch script which is "tangled" to the HPC cluster. Notice, the :tangle property uses these variables:

#+begin_src bash :tangle (concat "/ssh:" (org-entry-get nil "REMOTE" t) ":" (org-entry-get nil "REMOTE-DIR" t) "/run-intervals-analysis.sh")
#!/usr/bin/env bash

#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 4
#SBATCH --mem=64G
#SBATCH -t 0-03:00:00
#SBATCH -p bsudfq

module purge

RESULTS_PATH=${1}
CLASS_NAME=${2}
METHOD_ID=${3}

exec ~/DFA/bin/dfa interval-numerical \
     --classpath ~/DFA/artifacts.jar \
     --output "${RESULTS_PATH}" \
     "${CLASS_NAME}" \
     "${METHOD_ID}"
#+end_src

Relative to the current notebook, we also have data files. Many code blocks need to refer to these data files throughout the experimental process. To accomplish this, they are named using example blocks:

#+NAME: METHODS
#+begin_example
./in/methods.txt
#+end_example

#+NAME: DOMAINS
#+begin_example
./in/domains.txt
#+end_example

#+NAME: COMPARISONS
#+begin_example
./in/comparisons.txt
#+end_example

#+NAME: REPORTS
#+begin_example
./in/reports.txt
#+end_example

To take this a step further, the contents of the files themselves could be included in the notebook directly and tangled out to the named paths.

Finally, to uniquely identify different experiment runs, the following variable is generated for each invocation:

#+begin_src bash :eval query
echo "#+NAME: UUID"
echo -n $(uuidgen --time)
#+end_src

#+RESULTS:
:results:
#+NAME: UUID
4cf8581e-e2c2-11ed-aab2-8cf8c5ed93dd
:end:

Using this variable, we can create a relative path variable which is referenced for the experiment and analysis output:

#+NAME: OUTPUT_PREFIX
#+begin_src bash :var UUID=(org-sbe UUID) :results silent
echo -n "./out/${UUID}"
#+end_src

Using a relative path for OUTPUT_PREFIX allows for the prefix to be used on the remote servers and locally when processed data files are copied to the local machine for exploratory analysis.

The (org-sbe ...) pattern is used frequently because it inherently removes any newlines that may be introduced either by the literal text or if the variable is computed from a previous source block.

Before jobs are submitted, the remote directory tree needs to be populated:

#+begin_src bash :session *on-borah* :dir (concat "/ssh:" (org-entry-get nil "REMOTE" t) ":" (org-entry-get nil "REMOTE-DIR" t)) :results silent
mkdir -p ${OUTPUT_PREFIX}/{joblogs,intervals}
#+end_src

#+begin_src bash :results silent
scp -q "${METHODS}" ${remote}:${remote_dir}/${OUTPUT_PREFIX}/methods.txt
scp -q "${DOMAINS}" ${remote}:${remote_dir}/${OUTPUT_PREFIX}/domains.txt
scp -q "${COMPARISONS}" ${remote}:${remote_dir}/${OUTPUT_PREFIX}/comparisons.txt
scp -q "${REPORTS}" ${remote}:${remote_dir}/${OUTPUT_PREFIX}/reports.txt
rsync --archive ExperimentData/domains ${remote}:${remote_dir}/${OUTPUT_PREFIX}/.
#+end_src

To keep the source blocks simple and reduce copying, we use header-args to apply certain variables to all code blocks of a particular section. For example, "Job Initialization" has the following header arguments:

**** Job Initialization
:PROPERTIES:
:ID:       7e8302d4-38b7-4a3b-aed4-b329c81b43ce
:header-args:bash: :var OUTPUT_PREFIX=(org-sbe OUTPUT_PREFIX)
:header-args:bash+: :var METHODS=(org-sbe METHODS)
:header-args:bash+: :var DOMAINS=(org-sbe DOMAINS)
:header-args:bash+: :var COMPARISONS=(org-sbe COMPARISONS)
:header-args:bash+: :var REPORTS=(org-sbe REPORTS)
:header-args:bash+: :var remote=(org-entry-get nil "REMOTE" t)
:header-args:bash+: :var remote_dir=(org-entry-get nil "REMOTE-DIR" t)
:END:

Finally, to execute a series of analyses, we use GNU Parallel to produce a cross-product of our input parameters and submit the jobs against the remote machine:

#+name: intervals
#+begin_src bash
parallel --colsep '\t' \
         --shuf \
         --jobs=25% \
         --delay 1s \
         ssh -q ${remote} \
         sbatch --chdir="${remote_dir}" \
         --job-name="intervals-{1}_{2}" \
         --output="${OUTPUT_PREFIX}/joblogs/%x.out" \
         --error="${OUTPUT_PREFIX}/joblogs/%x.err" \
         run-intervals-analysis.sh "${OUTPUT_PREFIX}/intervals" "{1}" "{2}" \
         :::: "${METHODS}"
#+end_src

Once the jobs are complete, we can download the results and begin the analysis process. However, that is essentially the same set of ideas repeated.