Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vikasrawal/orgpaper
Reproducible Research Papers using Org-mode and R: A Guide
https://github.com/vikasrawal/orgpaper
Last synced: about 1 month ago
JSON representation
Reproducible Research Papers using Org-mode and R: A Guide
- Host: GitHub
- URL: https://github.com/vikasrawal/orgpaper
- Owner: vikasrawal
- Created: 2014-05-04T08:56:41.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2024-04-11T12:48:42.000Z (8 months ago)
- Last Synced: 2024-08-02T16:45:15.425Z (4 months ago)
- Language: Emacs Lisp
- Size: 714 KB
- Stars: 346
- Watchers: 36
- Forks: 66
- Open Issues: 2
-
Metadata Files:
- Readme: README.org
Awesome Lists containing this project
- my-awesome-github-stars - vikasrawal/orgpaper - Reproducible Research Papers using Org-mode and R: A Guide (Emacs Lisp)
README
#+STARTUP: hidestars
#+TITLE: Using Emacs, Org-mode and R for Research Writing in Social Sciences
#+SUBTITLE: A Toolkit for Writing Reproducible Research Papers and Monographs
#+AUTHOR: Vikas Rawal
#+COLUMNS: %25ITEM %TAGS %PRIORITY %T
#+OPTIONS: H:4 toc:2 num:2 broken-links:mark
#+LaTeX_CLASS: vmemoir
#+LATEX_CLASS_OPTIONS: [oneside,openany,strict,extrafontsizes,article,garamond]
#+INCLUDE: default_packages.org
#+LATEX_HEADER: \linespread{1.3}
#+LATEX_HEADER: \renewcommand{\TPTminimum}{\linewidth}
#+LATEX_HEADER: \usepackage{tabularx,siunitx,booktabs}
#+LATEX_HEADER: \newcolumntype{C}{>{\centering\arraybackslash}X}
#+LATEX_HEADER: \newcommand{\mconx}[1]{\multicolumn{1}{C}{#1}}
#+LATEX_HEADER: \newcommand{\mcx}[2]{\multicolumn{#1}{>{\hsize=\dimexpr #1\hsize +\numexpr2*#1-2\relax\tabcolsep \relax}C}{#2}}
#+LATEX_HEADER: \newcolumntype{d}[1]{D{.}{\cdot}{#1}}
#+LATEX_HEADER: \newcommand{\mcone}[1]{\multicolumn{1}{C}{#1}}
#+MACRO: M @@latex:\mcone{$1}@@
#+MACRO: MCD @@latex:\multicolumn{1}{c}{$1}@@
* Introduction
This guide introduces an open-source toolkit for academic research and
writing. The main features of this toolkit centered around Emacs and
Org-mode are:
+ embedded R code in the document that allows for statistical results
to be revised and reproduced,
+ bibliographic citations from a personal bibliographic database,
+ formatting using well defined styles with minimal markup,
+ support for production of final output as pdf, odt, docx, html and
many other formats.
** Will this guide be useful for you?
This guide will be useful for you if you are writing a research paper,
a dissertation or an academic book. It would be useful if your writing
involves one or more of the following:
+ Citing existing literature in your area
+ Presenting results of statistical analyses (in tabular form and/or
graphically)
+ Using mathematical equations
Following this guide would need some investment of time but benefits
far outweigh the investment you make.
** What is our goal?
What are the most interesting features of the writing platform that
you will set up using this guide?
+ With easy style specifications that you provide, the document will
be almost-entirely automatically formatted by the software.
- Complicated LaTeX-style markup is a pain and Openoffice/MS-Word
documents require too much manual formatting. Basic Org-mode
mark-up is extremely simple, and can be mastered in very little
time.
- Org-mode can produce well formatted output in LaTeX, pdf, odt,
docx, html and many other formats.
+ Instead of including statistical results (tables, graphs, etc), we
would embed appropriate R programs in the document, so that when the
formatted output is produced, all programs are run to generate the
results. Advantages of doing this are:
- Any changes in the data being used can be accommodated just by
publishing the document again.
- Any modifications in statistical analysis are easily made by
modifying the programs that are embedded in the file itself.
- Anyone who has the org file, can reproduce your results. You can
also extract all R programs from the org file and distribute those
for reproduction of your results.
+ The document will be integrated with a citation manager, so that
bibliographic information will be pulled automatically from a
central database to create a fully formatted bibliography.
- You will maintain a bibliographic database in BibTeX format, that
you can build over time, adding bibliographic information for
works that you cite.
- Many websites (including Google Scholar) provide bibliographic
information directly in BibTeX format, and we will have integrated
tools that will allow us to pull this information directly into
our local database.
** Acknowledgements
In my adaptation of org, I have benefited immensely from the great
community of org-mode developers and users. The [[http://orgmode.org/manual/][Org-mode manual]], [[http://orgmode.org/worg/][Worg]],
and archives of the [[http://orgmode.org/community.html][Org-mode mailing list]] have been the most important
resources. In addition, I have greatly benefited from solutions
provided by various people to my specific queries on the Org-mode
mailing list. What I present in this document is essentially a
synthesis of solutions provided by various people. The community has
been extremely generous in providing these.
I would particularly like to thank
+ Carsten Dominik, the author of Org-mode.
+ Bastien Guerry, who has been a great maintainer of Org-mode, after
Carsten passed on the baton to him.
+ Nicolas Goaziou, who wrote the brilliant new exporter framework. The
amount of code Nicolas has contributed to Org over the last two
years or so is incredible. Nicolas very kindly responded to several
of my queries.
+ Eric Schulte, the main author of Babel, which gave Org mode the
ability to execute code. I used to use Org-mode as a task manager
and for taking notes. I discovered org-babel in the summer of 2010,
when I was doing fieldwork in villages in eastern India. This
discovery completely changed my work flow, and Org-mode became
central to all my academic work.
+ John Kitchin, the author of scimax, org-ref and org-ref-cite, with
whose use of org-mode I have the greatest overlap. His documentation
and videos are a great resource.
+ In addition to the above, Suvayu Ali, for responses to several of my
queries on the mailing list.
* Installing necessary software
This set up will work with any operating system. I have tested it on
GNU/Linux and Mac OS-X, but it should work on Windows as well. For
this setup, you need to install Emacs (Version 24 along with a few
additional Emacs packages), Texlive, R (along with whatever
additional R packages you want to use) and Pandoc.
** Emacs
+ GNU/Linux
Emacs can be installed using package managers of all GNU/Linux
distributions. Latest versions of most common distributions provide
version 24. I strongly recommend using the latest version of Emacs.
+ Mac OS-X
The built-in Emacs on OS-X is an older version, and it would be a
good idea to install the latest version instead.
The best option is to install it via homebrew. I like the version
available from railwaycat/emacsmacport tap
(https://github.com/railwaycat/emacs-mac-port).
After installing homebrew, or if you already have it installed, just
do the following from the terminal
~$ brew tap railwaycat/emacsmacport~
~$ brew install emacs-mac~
+ Microsoft Windows
Download the latest version of Emacs from
http://ftp.gnu.org/gnu/emacs/windows/, and install.
** Texlive
+ GNU/Linux
Texlive can also be installed from package managers in most
GNU/Linux distribution.
+ Mac OS-X
For OS-X, install MacTeX from http://www.tug.org/mactex/
+ Microsoft Windows
For Windows, download Texlive and follow instructions from
https://www.tug.org/texlive/doc.html
** R
In this guide, I assume that you are familiar with R
(http://www.r-project.org). I will not cover R programming in this
guide.
For GNU/Linux, R can be installed from native package managers (look
for r-base in debian and debian-based distributions). For Mac OS-X and
Windows, download and see installation instructions at
http://www.r-project.org
** Pandoc
Pandoc (http://johnmacfarlane.net/pandoc/) is an extremely powerful
converter, which can translate one markup into another. It supports
conversion between many file formats, and supports "syntax for
footnotes, tables, flexible ordered lists, definition lists, fenced
code blocks, superscript, subscript, strikeout, title blocks,
automatic tables of contents, embedded LaTeX math, citations, and
markdown inside HTML block elements." That is pretty much everything I
use.
We shall use pandoc to convert our file from LaTeX to odt/docx/html
formats.
* Emacs basics
GNU Emacs is an extensible platform. Although its primary function is
as an editor, it can be extended to do almost anything that you would
want your computer to do. Now, that really is not an overstatement. It
is a worthwhile aim to slowly shift an increasing number of tasks you
do on your computer to emacs-based solutions. For each major task you
do on your computer, ask if it can be done using emacs. For almost
everything, the answer is yes, and in most cases, emacs does it better
than other software you are used to. Many emacs users have learnt
emacs by shifting, one-by-one, all major tasks that they
do on the computer to emacs.
I am not going to give a detailed guide to use of emacs. A few tasks
for which I use Emacs include
+ File management (copying files, moving files, creating directories)
+ Reading and writing e-mails
+ Reading RSS feeds
+ Calender, scheduler, planner
+ Calculator
+ Statistical work (by hooking Emacs to R)
+ And, of course, as an editor (including for writing research papers)
In this guide, I will just provide a minimal set of basic commands in
emacs to get you started. This is a minimal but sufficient set to be
able to work. I expect that you would learn more commands as you start
using emacs.
** Notations
In emacs, a buffer is equivalent to a tab in a web browser. It is
normal to have several buffers open at the same time. Each file opens
in emacs as a buffer. Buffers could also have processes like R running
in them. Emacs displays any messages for you in a separate buffer.
Most commands in emacs are given using the Control (ctrl) or the Meta
(often mapped to alt) keys.[fn:3] Control key is usually referred to as ~C-~
and the Meta key as ~M-~. So a command ~C-c~ means pressing Control
and c together. Command ~M-x~ means pressing Meta and x
together. Everything is case-sensitive. So ~M-X~ would mean, pressing
Meta, Shift and x together. ~C-c M-x l~ would mean pressing C-c,
release, then M-x, release, and then l.
** Basic commands
Table [[essential-emacs-commands]] gives the commands that are the most
important. This is a minimal set, commands that you should aim to
learn as soon as possible. There are many more, which you will learn
as you start using emacs.
All commands have a verbose version that can be used by pressing ~M-x~
and writing the command. For example, ~M-x find-file~ to open a file.
All major commands are also mapped to a shortcut. For example, instead
of typing ~M-x find-file~ to open a file, you can say ~C-x C-f~. I
remember shortcuts for commands that I use most frequently. For
others, I use the verbose versions. Over time, one learns more
shortcuts and starts using them instead of the verbose versions.
#+caption: Essential emacs commands
#+name: essential-emacs-commands
#+attr_latex: :environment longtable :align @{}p{7cm}ll@{} :booktabs t
| Description | Verbose command | Shortcut |
| | =M-x= followed by | |
|---------------------------------------------------------------------------------------------------------------+---------------------------+--------------|
| /*Opening files, saving and closing*/ | | |
| /Open a file/ | =find-file= | =C-x C-f= |
| /Save the buffer/file/ | =save-buffer= | =C-x C-s= |
| /Save as: prompts for a new filename and saves the buffer into it/ | =write-named-file= | =C-x C-w= |
| /Save all buffers and quit emacs/ | =save-buffers-kill-emacs= | =C-x C-c= |
| /*Copy, Cut and Delete Commands*/ | | |
| /Delete the rest of the current line/ | =kill-line= | =C-k= |
| /To select text, press this at the beginning of the region and then take the cursor to the end/ | =set-mark-command= | =C-spacebar= |
| /Cut the selected region/ | =kill-region= | =C-w= |
| /Copy the selected region/ | =copy-region-as-kill= | =M-w= |
| /Paste or insert at current cursor location/ | =yank= | =C-y= |
| /*Search Commands*/ | | |
| /prompts for text string and then searches from the current cursor position forwards in the buffer/ | =isearch-forward= | =C-s= |
| /Find-and-replace: replaces one string with another, one by one, asking for each occurrence of search string/ | =query-replace= | =M-%= |
| /Find-and-replace: replaces all occurrences of one string with another/ | =replace-string= | |
| /*Other commands*/ | | |
| Divide a long sentence into multiple lines, each smaller than the maximum width specified | =fill-paragraph= | =M-q= |
| /*Window and Buffer Commands*/ | | |
| /Switch to another buffer/ | =switch-to-buffer= | =C-x b= |
| /List all buffers/ | =list-buffers= | =C-x C-b= |
| /Split current window into two windows; each window can show same or different buffers/ | =double-window= | =C-x 2= |
| /Remove the split/ | =zero-window= | =C-x 0= |
| /When you have two or more windows, move the cursor to the next window/ | =other-window= | =C-x o= |
| /*Canceling and undoing*/ | | |
| /Abort the command in progress/ | =keyboard-quit= | =C-g= |
| /Undo/ | =undo= | =C-_= |
* Customising Emacs
:PROPERTIES:
:CUSTOM_ID: customemacs
:END:
Emacs is highly customisable. We will need a well customised emacs
installation with a number of additional emacs packages. There are
many configuration frameworks available for emacs (including
[[https://www.spacemacs.org/][spacemacs]], [[https://github.com/hlissner/doom-emacs][doom]], [[https://github.com/bbatsov/prelude][prelude]], and [[https://github.com/jkitchin/scimax][scimax]]). You may try these and choose
whichever you like. In most cases, you would need to do some
additional customisation to suit your needs.
My current emacs configuration does not use any of these
frameworks. Instead, I have a rather modular setup primarily based on
use-package for loading the additional packages I need. Packages are
installed mainly using package.el, the most popular package manager
for emacs. A few packages, including org-mode, are directly installed
from their git repositories to take advantage of the latest features
available.
*** Clone the git repository
My configuration directory can be cloned from github using the following commands:
~$ mv .emacs.d .emacs.d.old~
~$ git clone --recurse-submodules -j8 https://github.com/ep624/emacs.d.2021.git ~/.emacs.d~
~$ cd ~/.emacs.d/org-mode && make~
The first command will move any existing emacs configuration to
.emacs.d.old, and the second command will install my configuration
instead. The third command compiles org-mode.
*** Install additional packages
Starting version 24, Emacs includes a package-manager. You can
install/update add-on packages using the package manager. To use the
package manager, press M-x in emacs, and then type
package-list-packages and press return. This would bring up a list of
packages.
To mark a package for installation, take the cursor to the item and
press i. Once you have marked the packages you want to install, press
x to execute the installation.
The following emacs commands will install all the packages that I
currently have in my emacs installation. You many not need some of
them. You may also want some others depending upon your use. You can
always delete the ones you do not need, and add any other packages
that you need.
#+begin_src emacs-lisp
(setq package-selected-packages '(ag anzu async auto-complete avy
bibretrieve auctex cdlatex
citeproc coffee-mode consult
counsel-projectile
counsel-tramp counsel
dired-subtree dired-hacks-utils
dirtree ebib edit-server
elfeed-goodies ace-jump-mode
elfeed-score elfeed emacsql
embark erc-hl-nicks erc-image
eshell-git-prompt
ess-R-data-view ctable
ess-r-insert-obj
ess-smart-equals
ess-smart-underscore ess-view
ess-view-data csv-mode ess
flx-ido flx flycheck
git-gutter+ git-gutter
highlight-indentation htmlize
ido-completing-read+
ido-vertical-mode iedit
ivy-bibtex bibtex-completion
biblio biblio-core
ivy-prescient key-chord keycast
kurecolor magit git-commit
magit-section memoize move-text
multi-web-mode multiple-cursors
nameless noflet notmuch-labeler
notmuch org-gcal alert log4e
gntp org-sidebar org-ql f
dash-functional
org-sticky-header
org-super-agenda ht ov parsebib
pdf-tools peg persist
persistent-scratch pinentry
popup popwin powerline
prescient pretty-hydra hydra lv
projectile pkg-info epl
quelpa-use-package quelpa queue
rainbow-delimiters rainbow-mode
remember-last-theme
request-deferred request
deferred simple-httpd
smartparens string-inflection
super-save swiper ivy tblui
tablist magit-popup transient
tree-mode ts s dash use-package
bind-key visual-fill-column
which-key windata with-editor
yasnippet-snippets yasnippet
zenburn-theme))
(package-install-selected-packages)
#+end_src
You can copy the above lines, paste them in an empty buffer (e.g.,
\*scratch\*), select them, and do M-x eval-region.
* Speed up with Yasnippet templates
You can considerably speed up your work in emacs by using
yasnippets. Yasnippets are chunks of text -- forms, templates, lines
of code -- that can be inserted in a buffer using a keyword. It allows
you to insert text that needs to be used repeatedly by using a short
keyword. If you have cloned my emacs configuration, a whole bunch of
snippets would be in $~/.emacs.d/snippets$ directory. I will
illustrate their use in the sections on org-mode.
* Org-mode basics
:PROPERTIES:
:CUSTOM_ID: orgmodebasics
:END:
** Preamble
An Org file has a few special lines at the top that set up the
environment. Following lines are an example of the minimal set of
lines that we shall use.
#+BEGIN_SRC org
,#+title: Using Emacs, Org-mode and R for Research Writing in Social Sciences
,#+subtitle: A Toolkit for Writing Reproducible Research Papers and Monographs
,#+author: Vikas Rawal
,#+date: May 4, 2014
,#+options: toc:2 H:3 num:2
#+end_src
As you can see, each line starts with a keyword, and the values for
this keyword are specified after the colon.
Table [[special-lines]] gives details of a few major special lines that we shall use. The table also gives snippet keywords that can used to create the keyword if you have got the yasnippets from my emacs configuration.
#+name: special-lines
#+caption: Main special lines to be used at the top of an Org buffer
#+attr_latex: :environment tabularx :width \textwidth :align llX
| Keyword | Snippet | Purpose |
|-------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ~#+title~ | ~= | Move the column left |
| =M-= | Move the column right |
| =M-S-= | Delete the current column |
| =M-S-= | Insert a new column to the left of the cursor position |
| =M-= | Move row up |
| =M-= | Move row down |
| =M-S-= | Delete the current row or horizontal line |
| =M-S-= | Insert a new row above the current row |
For more commands for manipulating tables, see [[http://www.orgmode.org/manual/Tables.html][this section of the Org
manual]]. In particular, you may want to look at spreadsheet-like
functions of the table editor.
If you have snippets included with my emacs configuration, you can
type ~nct1~, ~nct2~, ~nct3~ or ~nct4~ and press the tab key to create
blank tables with 1-4 columns.
Note that we will directly create only those tables in org that are
not produced as a result of some statistical analysis. For tables that
are a result of some statistical analysis, we will embed R programs
rather than the tables themselves. This is discussed in Section
[[#orgmodeandr]] of this guide.
** Images
You can insert images in documents as follows
: [[file:path-to-file/a.jpg]]
You should do this for images that you already have, and you just want
to insert them in the document. If you have snippets included in my
emacs configuration, you can use yasnippet ~ncf~ to help in inserting
a named figure. For graphs produced by R, we will embed the code
instead, so that the graph is generated and inserted automatically.
** Captions and cross-references
We would like to give a title to our tables and images. And we would
like to be able to refer to them from the text. These are achieved by
adding two lines above every table and image.
+ A line starting with ~#+caption:~ placed just above a table or a
figure adds a title to it. All Tables and Figures titles are
automatically numbered.
+ For referring to these Tables and Figures in the text, we shall name
each table and figure in a line starting with ~#+name:~ as below.
To illustrate, for inserting an image, with a caption and a name, this
is what we shall do.
#+BEGIN_SRC org
,#+name: literacy-rate
,#+caption: Percentage of literate men and women, by country (per cent)
[[file:a.jpg]]
#+end_src
Similarly, a table will be inserted as follows.
#+BEGIN_SRC org
,#+name: literacy-rate-table
,#+caption: Percentage of literate men and women, by country (per cent)
| Country | Men | Women |
|------------+-----+-------|
| India | 75 | 43 |
| Bangladesh | 83 | 63 |
| Rwanda | 77 | 60 |
#+end_src
To refer to the Table above in the text, write Table
~[[literacy-rate-table]]~. As an illustration, see the following sentence.
#+BEGIN_SRC org
Tables [[literacy-rate-table]] and [[health-table]], and Figure
[[literacy-figure]], show the level of underdevelopment.
#+end_src
By default, all objects with captions are numbered, and names are used
to anchor cross-references. When the formatted output is produced, all
the references would be automatically converted to appropriate
numbers. If new objects are inserted in the paper, numbering will be
adjusted automatically when you create the formatted output.
* Org-mode and R
:PROPERTIES:
:CUSTOM_ID: orgmodeandr
:END:
** Configuration
Following code in .emacs.d/_configs/use-org.el enables Org to run different types
of code. If you are using my emacs configuration, these are already enabled.
I have included here the languages that I commonly use. See Org
manual, if you would like to add any more.
#+BEGIN_SRC emacs-lisp
(org-babel-do-load-languages
'org-babel-load-languages
'((R . t)
(org . t)
(ditaa . t)
(latex . t)
(dot . t)
(emacs-lisp . t)
(gnuplot . t)
(screen . nil)
(shell . t)
(sql . nil)
(sqlite . t)))
#+END_SRC
** Embedding R code in an Org document
Org uses ESS (emacs-speaks-statistics) to provide a fully functional,
syntax-aware, development environment to write R code. R code is
embedded into Org as a source block. The basic syntax is
#+BEGIN_SRC org
,#+name: name_of_code_block
,#+BEGIN_SRC R
,#+END_SRC
#+END_SRC
This is how source blocks are created.
+ First write the lines starting with ~#+NAME~, ~#+BEGIN_SRC~ and
~#+END_SRC~. We will use different snippets to quickly insert these
lines for different types of code blocks.
+ Then with your cursor in between the ~BEGIN_SRC~ and the ~END_SRC~
lines, give the command C-c ' (that is, press Ctrl-C, release, and
press ').
- This would open a new buffer using ESS mode. If you type your code
in this buffer, you will see that ESS is syntax-aware and nicely
highlights R code.
- ESS also allows you to run (evaluate) the code that you write, to
test what your code is doing. ~C-return~ or ~C-c C-n~ can be used
to evaluate a line/region of code and move to the next line. Or
you can use ~C-c C-j~ for evaluating a single line of code, ~C-c
C-b~ for evaluating the entire ess buffer, or ~C-c C-r~ for a
marked region within the ess buffer.
+ Once you have finished writing a code block and tested it, press
~C-c '~ again to come back to your Org buffer.
+ In your Org buffer, with your cursor in a source-block, press ~C-c
C-c~ to evaluate the whole code block and have the results included
in your document.
+ You can always edit your source code by opening a temporary ESS
buffer using C-c'
** Code blocks that read data and load functions for later use in the document without any immediate output
I normally have one or two code blocks that read the data I am going
to use, call the libraries that I use, and define a few functions of
my own that I plan to use. I want this code block to be evaluated, so
that these data, libraries and functions become available in my R
environment. But no output from such code blocks is expected to be
included into the document.[fn:4]
Code block [[readdata-code]] is an example of such a code block. Note
~:results value silent~ switch used in the ~#+begin_src~ line.
#+name: readdata-code
#+BEGIN_SRC org
,#+name: readdata-code
,#+BEGIN_SRC R :results silent :exports none
read.data("datafile1.csv",sep=",",
header=T)->mydata1
,#+END_SRC
#+END_SRC
** Code blocks that produce results in the form of an Org-mode table
The output of a lot of R code I write is presented in some kind of table.
There are two approaches for producing formatted, publication-ready tables.
1. We get R to produce bare tables, add Org-mode markup to those, and
get Org-mode mode to export these into nice looking LaTeX
tables.
2. Alternatively, we can get R to produce nicely formatted
LaTeX tables, and let Org just export the LaTeX code as it is.
I will discuss the first approach in this section.
The code block may use data and functions made available by previous
code blocks, read some new data and may load some new functions. The
code block does some statistical processing. The last command of the
code block produces an object (for example, a data frame) that is
included in the document as a Table.
For example, the code block [[r-code-table]] below uses mydata1 read in
the previous code block, reads a new dataset, and processes them to
create a table that shows average BMI by country.
#+name: r-code-table
#+BEGIN_SRC org
,#+name: bmi-table-code
,#+BEGIN_SRC R :results value :exports results :colnames yes :hline
aggregate(height~Country,data=mydata1,mean)->a1
read.data("datafile2.csv",sep=",",header=T)->mydata2
aggregate(weight~Country,data=mydata2,mean)->a2
merge(a1,a2,by="Country")->a1
a1$weight/a1$height->a1$BMI
subset(a1,select=c("Country","BMI"))
,#+END_SRC
#+END_SRC
If you are using my emacs configuration, you can use the snippet
~srct~ to insert a blank code block of this kind. You can then use
~C-c '~ to go into a temporary ess buffer and write the code.
You can evaluate the code block using ~C-c C-c~. When you do that, it
produces the output, and places it immediately below the code block.
The results display the output of the code under a line that looks
like below
: #+RESULTS: bmi-table-code
Note that the results are tied to the code block using the name of the
code block. Every time you go to the source code block and press ~C-c
C-c~, the code is evaluated again and the results are updated.
On top of the line starting with ~#+RESULTS:~, we shall add two more
lines, to give the table a caption and a name. Note that the code
block and the result of the code block have separate names.
#+BEGIN_SRC org
,#+name: bmi-table-output
,#+caption: Average BMI, by country
,#+RESULTS: bmi-table-code
#+END_SRC
Like any Org table, you can cross-refer to this table using
~[[bmi-table-output]]~.
Section [[#tableformatting]] discusses how to format tables for LaTeX
export. All that can be done on tables created by source code blocks.
** Code blocks that produce a formatted LaTeX table
An alternative approach would be to use various R packages designed
for producing formatted tables. R has excellent libraries for
producing tables formatted for LaTeX, html, RTF and docx
exports. Particularly noteworthy are [[https://cran.r-project.org/web/packages/xtable/index.html][xtable]], [[https://gt.rstudio.com/articles/gt.html][gt]]/[[https://jthomasmock.github.io/gtExtras/][gtExtras]], [[https://haozhu233.github.io/kableExtra/][kableExtra]],
and [[https://github.com/turbanisch/tabularray][tabularray]] packages. It is beyond the scope of the present
document to describe each of these. Of these, [[https://cran.r-project.org/web/packages/xtable/index.html][xtable]] is most versatile
but complex. [[https://cran.r-project.org/web/packages/xtable/index.html][xtable]] does not work out of the box with threeparttable
for producing table notes. [[https://gt.rstudio.com/articles/gt.html][gt]]/[[https://jthomasmock.github.io/gtExtras/][gtExtras]] are excellent but are mainly
aim at producing html tables. [[https://haozhu233.github.io/kableExtra/][kableExtra]] produces tables that work
with tabular, longtable and tabu LaTeX environments to produce tables,
while [[https://github.com/turbanisch/tabularray][tabularray]] produces output for tabularray LaTeX environment.
[[https://haozhu233.github.io/kableExtra/][kableExtra]] works well with threeparttable.
The code block [[r-code-kable-formatted]] shows code that uses kableExtra
to produce a formatted LaTeX table. Similarly, code block
[[r-code-tabularray-formatted]] shows code that produces a table formatted
for tabularray LaTeX package.
A disadvantage of this approach would be that the results of the code
block will not contain an Org-mode table, which is a visually
appealing and functionally useful representation while working in
Org-mode.
One could, of course, use a combination of two approaches by creating
two separate code blocks: the first producing an Org-mode table that
is not meant for export (~:exports none~), and another that reads the
Org-mode table, and produces LaTeX code meant for export.
#+name: r-code-kable-formatted
#+BEGIN_SRC org
,#+NAME: code1
,#+BEGIN_SRC R :results value verbatim latex :exports results
library(kableExtra)
dt <- mtcars[1:5, 1:4]
kbl(dt, format="latex", booktabs = T, caption = "Demo Table") |>
kable_styling(latex_options = c("striped", "hold_position"),
full_width = F) |>
add_header_above(c(" ", "Group 1" = 2, "Group 2[note]" = 2)) |>
add_header_above(c(" ", "Group 3" = 4)) |>
footnote(c("1. table footnote","2. another footnote"))->t
,#+end_src
#+END_SRC
#+name: r-code-tabularray-formatted
#+BEGIN_SRC org
,#+NAME: code2
,#+BEGIN_SRC R :results value verbatim latex :exports results
library(tabularray)
library(dplyr)
df <- starwars |>
filter(homeworld == "Tatooine") |>
select(name, height, mass, sex, birth_year) |>
arrange(desc(birth_year))
df |>
mutate(sex = stringr::str_to_title(sex)) |>
group_by(sex) |>
tblr(type = "float", caption = "Starwars Creatures from Tatooine") |>
set_source_notes(
Note = "Entry C3PO altered to test characters that have a special meaning in LaTeX.",
Source = "R package \\texttt{dplyr}"
) |>
set_alignment(height:birth_year ~ "X[r]") |>
set_column_labels(
name = "",
height = "Height",
mass = "Mass",
birth_year = "Birth Year"
) |>
set_theme(row_group_style = "panel") |>
set_interface(width = "0.7\\linewidth") |>
set_column_spanner(
c(height, mass) ~ "Group 1",
birth_year ~ "Group 2"
) |>
set_column_spanner(!name ~ "All my vars")->t
t
,#+end_src
#+END_SRC
** Code blocks that produce a graph to be included in the document
These code blocks can have a series of commands. The last command
produces a graph that we would like to be included in the document.
The following code shows an example of a code block that produces a
graph.
#+begin_footnotesize
#+BEGIN_SRC org :results replace drawer
,#+name: mygraph-code
,#+BEGIN_SRC R :results file graphics :exports results :file bmi2.png :width 825 :height 1050 :fonts serif :res 300
library(ggplot2)
data.frame(var1=c(1:10),var2=c(1:10)^2)->data
ggplot(data,aes(x=var1,y=var2))->p
p+geom_line()
,#+END_SRC
#+END_SRC
#+end_footnotesize
You can use snippet ~9.5 to be able
to use this new citation syntax.
Users are advised to read the discussion on citation handling in the
[[https://orgmode.org/manual/Citation-handling.html#Citation-handling][org-mode manual]] and a very useful [[https://blog.tecosaur.com/tmio/2021-07-31-citations.html#fn.3][summary article]] by Timothy.
If you are using my emacs configuration, you should have the latest
version of org-mode. My configuration of functions related to
citations and bibliography resides in ~_configs/use-oc.el~. It is
loaded if sym-linked to ~_activated/use-uc.el~.
The tasks related to citations and bibliography can be divided into
two parts: (i) maintaining a bibliographic database and (ii) inserting
citations in org-mode documents and processing them to create
formatted citations and bibliography at the time of export.
The new syntax of citations divides the process of inserting and
processing citations in org-mode into four components: insert, follow,
activate and export. The core functionality is designed to allow
development and use of different tools for each of these. John Kitchin
and Bruce D'Arcus have built useful extensions on top of the core
functionality.
** Building and maintaining your bibliographic database
The citation processors available for org-mode allow the use of
bibliographic databases maintained in bibtex, biblatex or
citeproc/json formats.
I use a biblatex database because biblatex provide tools that are
highly customisable, to format bibliographies in any
style. Citeproc/json has become increasingly popular in recent years
because this is the native format used by pandoc and other
applications in the markdown ecosystem.
We shall use a master bibliographic database to contain bibliographic
records for the literature that we cite. The database, in Biblatex or
BibTex format, is stored in a text file with .bib extension.
In a BibTeX/BibLaTeX database, each bibliographic record is given a
unique key, which is used to cite it. Each record is classified as one
among various categories of publications (journal article, book,
chapter, etc.), and for the given publication type, the record
specifies values for various fields (author, title, volume, publisher,
etc). BibLaTeX extends the BibTeX specification to cover a wider
variety of publication types and fields. Given that, it is more
versatile than BibTeX.
Bibliographic information in BibTeX/BibLaTeX format is available from
many online sources, including journal/publisher websites, and Google
Scholar. Many applications/tools allow you to search download
bibliographic records from these sources directly. Please note that
you would often need to clean/correct the downloaded entries. And, when
the bibliographic information in BibTeX/BibLaTeX format is not available
from any existing database, you may have to enter the information
yourself.
To start with, you may wish to use a GUI application like JabRef
(cross-platform, http://jabref.sourceforge.net/) or BibDesk (OS-X
only, http://bibdesk.sourceforge.net/) to build and maintain your
database. In my experience, most of these tend to add additional junk
to stamp the entries that a particular application has been used to
create the database.
It is much cleaner, and efficient, to use excellent tools available
for maintaining the database in emacs itself. Ebib gives a nice
interface to manually enter the records. Bibretrieve and org-ref give
you commands that can download BibTeX records directly from online
databases.
Eventually, you should use bibretrieve from within Emacs to
add entries to your database. org-ref.el provided by John Kitchin
(https://github.com/jkitchin/jmax) also has some useful functions.
As a sample, my own bibliographic database is available from
https://github.com/indianstatistics/bibliobase/blob/master/bibliobase.bib.
** Using biblatex with Org (under revision)
*** Setup
Using biblatex with Org requires some customisation of variables. These
are done in ~_config/use-oc.el~ and ~_config/use-org-contrib.el~ files. Both the files are sym-linked to files with the same names in ~_activated~ directory).
Please note the configuration of ivy-bibtex in ~_config/use-oc.el~. You may want to modify the values of variables bibtex-completion-bibliography (default bibliographic database), bibtex-completion-notes-path (directory where you may optionally keep your notes for each source), bibtex-completion-library-path (the directory where you optionally keep pdf files for each source) and bibtex-completion-pdf-open-function (the application that should be used to open the pdfs).
: (use-package ivy-bibtex
: :ensure t
: :init
: (setq bibtex-completion-bibliography '("~/bibliobase/bibliobase.bib")
: bibtex-completion-notes-path "~/bibliobase/notes/"
: bibtex-completion-notes-template-multiple-files "#+TITLE: Notes on: ${author-or-editor} (${year}): ${title}\n\nSee [cite/t:@${=key=}]\n"
: bibtex-completion-library-path '("~/pdfbibliobase/")
: bibtex-completion-additional-search-fields '(keywords)
: bibtex-completion-display-formats
: '((article . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*} ${journal:40}")
: (inbook . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*} Chapter ${chapter:32}")
: (book . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*}")
: (incollection . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*} ${booktitle:40}")
: (inproceedings . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*} ${booktitle:40}")
: (t . "${=has-pdf=:1}${=has-note=:1} ${author:36} ${year:4} ${title:*}")))
: (setq bibtex-completion-pdf-open-function
: (lambda (fpath)
: (call-process "evince" nil 0 nil fpath)))
: )
:
The operative part in the ~_config/use-org-contrib.el~ file is the following:
: (setq org-latex-pdf-process
: '("xelatex -interaction nonstopmode -output-directory %o %f"
: "biber %b"
: "xelatex -interaction nonstopmode -output-directory %o %f"
: "xelatex -interaction nonstopmode -output-directory %o %f"))
Every time you export the document to pdf via latex, it runs xelatex,
then runs Biber and then runs xelatex twice again. This is necessary
to get the citations in the pdf file.
*** Document configuration
Three sets of things have to be specified for each document.
First, the following line in the header of the org file calls biblatex. You may want to modify the options used here.
: #+LATEX_HEADER: \usepackage[hyperref=true,maxcitenames=3,doi=false,url=true,
: backend=biber,natbib=true, maxbibnames=99,uniquename=false,uniquelist=false,
: indexing=cite,sorting=nyt,mergedate=compact,innamebeforetitle=true,articlein=false]{biblatex}
Second, the following line in the header of the org file specifies the biblatex citation and bibliographic styles to be used.
: #+cite_export: biblatex authoryear/authoryear-comp
Third, the following line specifies the bibliographic database to be used for this document.
: #+bibliography: bibliobase/bibliobase.bib
Where I have the choice, I also customise the bibliography in the way
I like. These customisations are in the file [[https://github.com/indianstatistics/bibliobase/blob/master/vikas-bibstyle.org][vikas-bibstyle.org]] in
[[https://github.com/indianstatistics/bibliobase.git][indianstatistics/bibliobase]] repository. This file can be included in
the document as follows:
: #+INCLUDE: PATH-TO-FILE/vikas-bibstyle.org
*** Inserting citations
Let us first understand the general syntax of citations in org. To
insert a citation at any point in the document, you need to use the
following syntax in square brackets.
: [cite/style/variant: common_prefix; prefix@key1 pageno suffix; prefix@key2 pageno suffix; common_suffix]
Let us understand this syntax. There are four parts in the syntax.
1. cite/style/variant: :: First, you specify the citation style. Table [[citation-commands]] lists citations styles and variants available with the biblatex citation processor in org. You can use abbreviations provided in the parentheses rather than the full notation for the citation styles and variants. This part ends in a colon.
2. common_prefix: :: You may optionally use some arbitrary text as prefix. This would be ~see~ in a citation that you want to export as ~(see Seema, 2010 and Nancy, 2011 as examples of this)~. This part ends in a semicolon.
3. citations: :: Then you specify the sources that you want to cite. Each source that you want to cite at this point is separated by a semicolon. The source is cited using the citation key from your bibliographic database. So, if the citation key for ~Seema, 2010~ in the database is ~seema2010~, you would cite it as ~@seema2010~. The citation key may be preceded by a source-specific prefix, and followed by a location identifier such as page number and citation-specific suffix. Prefix, pageno and suffix are optional.
4. Finally, you may optionally use some arbitrary text as a common suffix. This would be ~as examples of this~ in the citation to be exported ~(see Seema, 2010 and Nancy, 2011 as examples of this)~.
#+caption: Important citation styles and variants provided by the biblatex processor in org
#+name: citation-commands
#+attr_latex: :environment tabularx :width \textwidth :align lXXX
| Style (abbreviation) | Variant (abbreviation) | Biblatex command | Example output |
|----------------------+------------------------+------------------+------------------------------------------|
| author(a) | caps(c) | Citeauthor* | Seema et al. |
| author(a) | full(f) | citeauthor | Seema, K and David, P |
| author(a) | caps-full(cf) | Citeauthor | Seema, K and David, P |
| author(a) | nil | citeauthor* | Seema et al. |
| locators(l) | bare(b) | notecite | p. 56 |
| locators(l) | caps(c) | Pnotecite | (P. 56) |
| locators(l) | bare-caps(bc) | Notecite | P. 56 |
| locators(l) | nil | pnotecite | (p. 56) |
| noauthor(na) | bare(b) | cite* | 2010 |
| noauthor(na) | nil | autocite* | (2010) |
| nocite(n) | nil | nocite | No citation but included in bibliography |
| text(t) | caps(c) | Textcite | Seema and David (2010) |
| text(t) | nil | textcite | Seema and David (2010) |
| nil | bare(b) | cite | (Seema and David, 2010) |
| nil | caps(c) | Autocite | (Seema and David, 2010) |
| nil | bare-caps(bc) | Cite | Seema and David, 2010 |
| nil | nil | autocite | (Seema and David, 2010) |
Insert processors provide convenient shortcuts for inserting the
citations and this does not need to be done manually. I use the insert
processor from the [[https://github.com/jkitchin/org-ref-cite][org-ref-cite]] repository created by John
Kitchin. Although this insert processor is created for use with
bibtex/natbib, it works reasonably well for biblatex as well. ~C-c \~
is bound to org-cite-insert. It would open your database in a
convenient form using Ivy completion framework, where you can type
keywords to select the source you want to cite, and press enter to
insert it. Please look at [[https://github.com/jkitchin/org-ref-cite][org-ref-cite]] for additional details on how
to modify citation styles, insert multiple citations, and insert
prefixes and suffixes.
[[org-ref-cite]] also provides other useful facilities. If you click on
the citation, a menu opens which gives you several options including
to go to the entry in the biblatex database (in case you want to edit
it), to look at your notes about the source or to open the pdf file of
the source document. org-ref-cite also colours the citations
differently including colouring them differently (red) if the citation
key does not match with any valid entry in the database. Please note
that since the fontification is designed for bibtex/natbib, it may
give some false alerts for a biblatex database.
*** Creating a bibliography
To insert the bibliography, add the following line where you want to
insert the bibliography (usually, at the end of your paper, but before
the Footnotes)
: #+print_bibliography:
* Producing a formatted LaTeX, pdf, odt, docx or html file
From Org, we can get a well-formatted document as a LaTeX, PDF, odt, docx
or html file. To produce a formatted output, we shall use the built-in
exporters provided with Org, and for some file types, use Pandoc for
further conversion.
Built-in exporters can be called in Org using ~C-c C-e~ or ~M-x
org-export-dispatch~.
** Formatting tables for LaTeX/PDF export
:PROPERTIES:
:CUSTOM_ID: tableformatting
:END:
There are many LaTeX packages that can be used to format tables. They
provide different options for formatting tables, and using them
involves different degrees of complexity. We consider four different
libraries for creating tables.
The choice of LaTeX environment and other associated options are
specified in orgmode using lines starting with [[http://orgmode.org/manual/LaTeX-specific-attributes.html][~#+attr_latex~]]. These
lines generate required LaTeX commands on export. We will also use org
[[http://orgmode.org/manual/Special-blocks.html][special blocks]] for advanced table layouts. ~#+attr_latex~ lines affect
only the object (table, image or a special block like begin_table or
begin_tablenotes) that follows immediately after. These must,
therefore, be placed immediately before the object they are supposed
to be applied on. Note that there should not be even a blank line
between ~#+attr_latex~ lines and the object.
Some key attributes that need to be specified using ~#+attr_latex~
lines are as follows:
+ ~:environment:~ :: ~:environment~ specifies the LaTeX package to be
used for a particular table. This is the simplest of all but allows
limited possibilities for formatting columns. We will use tabulary,
tabularx and tabularray packages for most of our tables. Other
packages that you might want to look at are tabu, longtable and
sidewaystable. However, the packages used in this guide will suffice
for most needs.
+ ~:align~ :: ~:align~ specifies how to render each columns by using
one letter (l,L,r,R,c,C or J) for each column. Each column type is
represented by a character. For example, in most packages, c stands
for center aligned columns, l for left aligned columns and r for
right aligned columns. The number of letters should exactly match
the number of columns in your table. For example, for a four column
table, you would specify something like ~:align lccr~, which means
that the first column is left aligned, next two columns are center
aligned and the last column is right aligned. A ~|~ anywhere in this
string implies a vertical line. So in a table formatted with ~:align
l|ccr~, there would be a vertical line after the first column. When
you have many columns, you can use expressions like ~*{3}{c}~ in
place of ~ccc~. This says, use column ~c~ three times. Normally,
tables are slightly indented on the left and the right. To remove
this indentation, you can add ~@{}~ before and after the alignment
expression (for example, ~:align @{}l*{3}{c}r@{}~).
+ ~:width~ :: ~:width~ is used to specify the width of the table that
the table can take [it may be specified as ~\textwidth~, implying
full text width, as ~0.8\textwidth~, implying a fraction (0.8 here)
of the text width, in centimeters (like, 10cm) or in inches (like,
5in)]. Note that, different environments use the information on
width differently. In particular, if the contents of your table
columns do not require the width specified by you, some packages
would only use the minimum width required while others would widen
the table to the total width specified by you. The total width is
also divided between different columns differently by various
packages.
+ ~:center~ :: This option followed by ~t~ or ~nil~ would center align
or left align the table horizontally.
+ ~:float~ :: This option specifies whether this object (table, figure
or any other special block) is a floating object which can be
optimally placed by LaTeX. In most cases, you would want these
objects to float so that LaTeX can optimally place them.
+ ~:booktabs~ :: This option followed by ~t~ gives us nice horizontal
lines. In most cases, when using ~booktabs~, one should not use
vertical lines in the table (that is, ~:align~ should not have any
~|~). Booktabs automatically inserts a top line (~\toprule~ in LaTeX),
a middle line just below the header row (~\midrule~) and a line at the
bottom of the table (~\bottomrule~).
*** Multicolumn cells and horizontal rules
One limitation of Org is lack of support for merging of cells in a
Table. However, Eric Schulte has created useful functions that can be
used to overcome this limitation while exporting to LaTeX or HTML. I
have extended those functions to also provide facilities for creating
horizontal lines (~\midrule~ and ~\cmidrule~). These functions
(org-export-midrule-filter-latex and org-export-cmidrule-filter-latex)
are already included in my emacs configuration in
(~_configs/use-org-contrib.el~).
The method of creating multicolumn (and multirow) cells in ~tabularay~
is different from other tabular environments and will be discussed
separately when I discuss ~tabularray~ environment. For all other
~tabular~ environments discussed here, the content of any cell can be
preceded by a string such as <3colc> to make those contents span and
centred across 3 columns to the right.
In addition, placed in a cell draws a ~\midrule~ at that
position across the entire width of the table. Other cells in that row
will be ignored.
A string such as <2cid4> can be used to say that a horizontal line be
drawn to span second and fourth columns.
The table below illustrates the use of <2colc> to write text that
spans two columns, to draw a horizontal line across the width of
the table, and <2cid3> to draw a line spanning second and third columns.
A particular advantage of these functions is that they can be inserted
in a table by the code block that is generating the tables.
#+name: multicolumns-in
#+BEGIN_SRC org :results replace drawer
,#+name: tabulary-yield-out
,#+caption: Table formatted using tabulary package
| State | <2colc>Two Column Text | | Variable4 |
| <2cid3> | | | |
| | Yield | Income | |
| | | | |
| Madhya Pradesh | 669 | 13121.25 | 123 |
| Haryana | 300 | 2532.30 | 22 |
| Punjab | 260 | 35232.45 | 324 |
#+End_SRC
*** Formatting tables using tabulary package
Package ~tabulary~ extends the ~tabular~ environment by providing
three additional column types. It is relatively easy to integrate this
package into orgmode.
Table [[tabulary-column-types]] shows different types of columns available
in ~tabulary~ package.
#+name: tabulary-column-types
#+caption: Types of columns in LaTeX tabulary package
#+attr_latex: :environment tabulary :width \textwidth :align lL :booktabs t
| Type | Description |
|------+------------------------------|
| l | Left aligned, no wrapping |
| L | Left aligned with wrapping |
| r | Right aligned, no wrapping |
| R | Right aligned with wrapping |
| c | Centre aligned, no wrapping |
| C | Centre aligned with wrapping |
| J | Justified and wrapped |
Load tabulary and booktabs using the following line in the preamble.
~#+LATEX_HEADER: \usepackage{tabulary,booktabs}~
Following ~#+attr_latex~ lines illustrate the way of declaring that
the table should be constructed using ~tabulary~ and passing various
options to it:
: #+attr_latex: :environment tabulary :width 0.8\textwidth :align @{}L|llR@{}
: #+attr_latex: :center :booktabs t
The following code uses tabulary to create a table. The output of the code is shown in Table [[tabulary-yield-out]].
#+name: tabulary-in
#+BEGIN_SRC org :results replace drawer
,#+name: tabulary-yield-out
,#+caption: Table formatted using tabulary package
,#+ATTR_LATEX: :environment tabulary :width \textwidth :align @{}lRR@{}
,#+ATTR_LATEX: :center t :booktabs t :float t
| State | Average yield | Average income |
|----------------+---------------+----------------|
| Madhya Pradesh | 669 | 13121.25 |
| Haryana | 300 | 2532.30 |
| Punjab | 260 | 35232.45 |
#+End_SRC
#+RESULTS: tabulary-in
:RESULTS:
#+name: tabulary-yield-out
#+caption: Table formatted using tabulary package
#+ATTR_LATEX: :environment tabulary :width \textwidth :align @{}lRR@{}
#+ATTR_LATEX: :center t :booktabs t :float t
| State | Average yield | Average income |
|----------------+---------------+----------------|
| Madhya Pradesh | 669 | 13121.25 |
| Haryana | 300 | 2532.30 |
| Punjab | 260 | 35232.45 |
:END:
*** Using tabularx and siunitx packages
Package tabularx provides a flexible tabular environment. In addition,
siunitx provides an additional column type S which can be used to
align numbers currently on the decimal. Packages tabularx, siunitx and
booktabs are included in the Memoir class, and do not need to be
called if you use vmemoir or varticle LaTeX classes defined in my
emacs configuration. If you are using article (may be the default) or
any other class, you may need to add the following line in the
preamble of your org file.
~#+LATEX_HEADER: \usepackage{tabularx,booktabs}~
Following lines need to be added to the header of the org file to set
the alignment of numeric columns correctly (using siunitx LaTeX
package) and to properly align headings of table columns.
: #+LATEX_HEADER: \usepackage[add-decimal-zero = true,add-integer-zero = true,
: round-integer-to-decimal,round-mode = places,
: round-precision=1]{siunitx}
: #+LATEX_HEADER: \newcolumntype{C}{>{\centering\arraybackslash}X}
: #+MACRO: M @@latex:\multicolumn{1}{C}{$1}@@
With this, you are set to create nicely formatted tables in LaTeX/PDF files.
The following code uses tabularx and siunitx to create a table. The
output of the code is shown in Table [[tabularx-yield-out]].
In the ~:align~ specification for ~tabularx~ tables (with ~siunitx~),
you can use following types of columns:
+ ~l~ specifies a left aligned column with no wrapping.
+ ~r~ specifies a right aligned column with no wrapping.
+ ~X~ specifies a left aligned column with wrapping.
+ ~S~ specifies a numeric column aligned to the decimal point.
+ Use option table-format=n.m to specify the maximum n digits and m
decimals allowed in the column. Numbers will be rounded off to m
decimals.
+ Use option round-mode=off to turn the rounding-mode off (since we
turned it on while loading siunitx). This is needed in columns
that consist only integers.
You are advised to look at the tabularx and siunitx manuals for the
many options that they provide.[fn:1]
For the ~S~ columns, the column headings should be wrapped in macro M
as shown in the example below to ensure that these are properly
aligned in the center of the columns.
#+name: tabularx-yield-in
#+BEGIN_SRC org :results replace drawer
,#+name: tabularx-yield-out
,#+caption: Table formatted using tabularx and siunitx packages
,#+ATTR_LATEX: :environment tabularx :width 0.8\textwidth
,#+ATTR_LATEX: :align @{}l{S[table-format=3.0, round-mode=off]}{S[table-format=5.2]}@{}
,#+ATTR_LATEX: :center t :booktabs t :float t
| State | {{{M(Average yield)}}} | {{{M(Average income)}}} |
|----------------+------------------------+-------------------------+
| Madhya Pradesh | 669 | 13121.25 |
| Haryana | 300 | 2532.30 |
| Punjab | 260 | 35232.45 |
#+End_SRC
#+name: tabularx-yield-out
#+caption: Table formatted using tabularx and siunitx packages
#+ATTR_LATEX: :environment tabularx :width 0.8\textwidth
#+ATTR_LATEX: :align @{}l{S[table-format=3.0]}{S[table-format=5.2]}@{}
#+ATTR_LATEX: :center t :booktabs t :float
| State | {{{M(Average yield)}}} | {{{M(Average income)}}} |
|----------------+------------------------+-------------------------|
| Madhya Pradesh | 669 | 13121.25 |
| Haryana | 300 | 2532.30 |
| Punjab | 260 | 35232.45 |
*** Using threeparttable to write notes below tables
LaTeX package =threeparttable= is used for including notes below the
table. For using =threeparttable= you need to call the package. In
addition, it is a good idea to include the following special line for
better formatting of notes below the table
: #+LATEX_HEADER: \renewcommand{\TPTminimum}{\linewidth}
The following code produces a table (Table [[threeparttable-table-yield]])
with notes below.
#+name: org-threeparttable-in
#+BEGIN_SRC org
,#+name: threeparttable-table-yield
,#+caption: Table created using tabularx, siunitx and threeparttable
,#+begin_table
,#+ATTR_LATEX: :float t :options [hb]
,#+begin_threeparttable
,#+ATTR_LATEX: :environment tabularx :width 0.8\textwidth
,#+ATTR_LATEX: :align @{}l{S[table-format=3.0]}{S[table-format=5.2]}@{}
,#+ATTR_LATEX: :booktabs t
| State | {{{M(Average yield)}}} | {{{M(Average income)}}} |
|----------------+------------------------+-------------------------|
| Madhya Pradesh | 669 | 13121.25 |
| Haryana^{a} | 300 | 2532.30 |
| Punjab | 260 | 35232.45 |
,#+attr_latex: :options [flushleft]
,#+begin_tablenotes
,#+begin_footnotesize
+ Notes: :: \mbox{}
1. This table is very nice. This note is very long. But the long
note wraps nicely under the table.
2. This is the second note. But this is not very wide.
+ We can use bullets.
+ a :: Or use description lists to refer to footnote markets in
the table
+ Source: :: https://www.indianstatistics.org
,#+end_footnotesize
,#+end_tablenotes
,#+end_threeparttable
,#+end_table
#+End_SRC
Org lists can be used to format the notes properly. ~\mbox{}~ is used
to leave white space in the given example since the description list
items ~+Notes:~ and ~Source:~ do not have any description in the same
line.
I have used footnotesize to render the notes in a slightly smaller
font. Option [flushleft] is used to align the notes to the left.
*** Using tabularray package for making tables
~tabularray~ is a new package that is very versatile and provides
flexible ways of creating multi-row and multi-column cells, adding
table notes and using colours in the tables. You can also create
multi-page tables with tabularray.
You are strongly advised to go through the [[http://mirrors.ctan.org/macros/latex/contrib/tabularray/tabularray.pdf][manual]] of tabularray
package to understand various options. The focus here is to illustrate
how to use the package in org documents.
There are three environments provided by ~tabularray~:
+ tblr: :: Basic tabularray environment with excellent support for merging
cells across columns and rows.
+ talltblr: :: In addition to features of ~tblr~, this also allows for
notes below the tables.
+ longtblr: :: For tables running across multiple pages. All the
features of ~tblr~ and ~talltblr~ are available here.
To use ~siunitx~, ~booktabs~ and ~diagbox~ along with ~tabularray~,
you need to add these lines to the preamble of the orgmode file.
: #+LATEX_HEADER: \usepackage{comment,multirow,tabularray}
: #+latex_header: \UseTblrLibrary{booktabs}
: #+latex_header: \UseTblrLibrary{siunitx}
: #+latex_header: \UseTblrLibrary{diagbox}
A few additional hacks are required for using these environments in
orgmode.
1. :align
Both the table width and column specifications will be provided as
value to the :align keyword. Note that a separate :width keyword
does not work with ~tabularray~.
2. Macro to center align column headings for siunitx columns
For siunitx(S) columns, we would like to align column headings
differently from alignment of cells containing numbers. For this,
column headings must be wrapped in triple curly braces. However,
org uses triple curly braces to trigger macros. To deal with this,
we define a macro (mc below) which just replaces itself with the
triple curly braces. So, column headings will be wrapped in the mc
macro as: ~{{{mc(colheading)}}}~, and on export, this would create
~{{{colheading}}}~.
We will add this macro specification to the header.
: #+MACRO: mc @@latex:{{{$1}}}@@
You can also optionally create macros such as this to conveniently
add colour/highlights to specific cells:
: #+MACRO: HG @@latex:\SetCell{bg=gray9} $1@@
3. Some of the options -- such as specification of footnotes -- are
provided in ~tabularray~ as an optional argument in square braces
when invoking the ~tabularray~ environments. Juan Manuel Macias
recently provided an elisp function that allows for the use of an
~:options~ keyword to specify such arguments. This is included in
~_configs/use-org-contrib.el~ in my emacs configuration.
4. ~\cmidrules~
The LaTeX syntax for ~\cmidrules~ in ~tabularray~ is slightly
different from other tabular environments. Just as we use <2cid3>
to declare that we want a line going from second to third column,
we can use <2cd3> with ~tabularray~ environment. This feature is
provided by the function
org-export-tabularray-cmidrule-filter-latex defined in
~_configs/use-org-contrib.el~.
#+name: tabularray-yield-in
#+BEGIN_SRC org :results replace drawer
,#+name: tabularrray-yield-out
,#+caption: Table formatted using tabularray and siunitx packages
,#+ATTR_LATEX: :environment talltblr
,#+ATTR_LATEX: :align width=0.8\textwidth,colspec={lS[table-format=3.0,round-precision=0]S[table-format=5.2,round-mode=places,round-precision=true,alignment-mode=marker]S[table-format=3.0,round-precision=0]}
,#+ATTR_LaTeX: :options remark{Notes} = {This note goes below the table},
,#+ATTR_LaTeX: :options remark{Source} = {Second note goes below the first note}
,#+ATTR_LATEX: :center :booktabs t :font \small
| State | <2colc>Two Column Text | | {{{mc(Variable4)}}} |
| <2cd3> | | | |
| | {{{mc(Average yield)}}} | {{{mc(Average income)}}} | |
| | | | |
| Madhya Pradesh | 669 | 13121.25 | 123 |
| Haryana | 300 | 2532.30 | 22 |
| Punjab | 260 | 35232.45 | 324 |
#+End_SRC
#+caption: Table formatted using tabularray and siunitx packages
#+name: tabularrray-yield-out
#+ATTR_LATEX: :environment talltblr
#+ATTR_LATEX: :align width=0.8\textwidth, colspec={lSSS}
#+ATTR_LaTeX: :options remark{Notes} = {This note goes below the table},
#+ATTR_LaTeX: :options remark{Source} = {Second note goes below the first note}
#+ATTR_LATEX: :center :booktabs t :font \small
| State | <2colc>Two Column Text | | {{{M(Variable4)}}} |
| <2cd3> | | | |
| | {{{M(Average yield)}}} | {{{M(Average income)}}} | |
| | | | |
| Madhya Pradesh | 669 | 13121.25 | 123 |
| Haryana | 300 | 2532.30 | 22 |
| Punjab | 260 | 35232.45 | 324 |
** Creating LaTeX and/or PDF files
The file [[file:default_packages.org]] lists a set of LaTeX packages that
I normally use. Please modify this as you please, save it in a
convenient location, and call it using a line of the following kind.
~#+INCLUDE: "path/to/default_packages.org"~
For producing LaTeX and/or PDF files, use ~C-c C-e~ to call the Org export dispatcher.
+ Press l to select LaTeX, and then chose one of the following options.
+ Press l again, if you just want to create a LaTeX file
+ Press p, if you want to create a pdf file. This will first create
a latex file, then use pdflatex and Biber to create a pdf file.
+ Press o, if you want to create pdf and have it opened in the
default pdf viewing application.
** Creating odt or docx files
There is a built-in odt exporter in Org. While it works well for most
situations, there are two components of the setup proposed here that
it does not support. It does not support biblatex and it does not
support LaTeX-specific solution we have for Notes under Tables and
Images.[fn:6]
Fortunately, [[http://johnmacfarlane.net/pandoc/][Pandoc]] provides an excellent solution for converting
LaTeX output to odt or docx documents. Pandoc supports all the LaTeX
syntax that Org produces from our files, and you can get a very well
formatted output.
Use ~C-c C-e l l~ to create a LaTeX file. Then, from the terminal, use
Pandoc as follows to create an odt or a docx file.
: pandoc --bibliography=biblidatabase.bib --filter pandoc-citeproc \
: latexfile.tex -o outputfile.odt
: pandoc --bibliography=biblidatabase.bib --filter pandoc-citeproc \
: latexfile.tex -o outputfile.docx
If you want, you can use --template to specify an ott or a .dotx
template file, so that the fonts and other formatting attributes are
to your liking.
** HTML
For html as well, there is a built-in exporter in Org. The built-in
exporter is very good, and the way to go if you are planning to
maintain a website using Org (as I do for
http://www.indianstatistics.org).
The built-in exporter can support BibTex citations using ox-BibTex.el,
which is including in Org, and will be loaded if you have installed
[[https://github.com/vikasrawal/orgpaper/blob/master/research-toolkit.org][research-toolkit.org]]. You may need to install BibTex2html separately to make it
work.
However, ox-BibTex.el uses BibTex2html for converting citations and
bibliography to html. BibTex2html provides limited support for
citation and bibliography styles.
If you want full support for bibliography and citation styles, as well
as for other LaTeX components like Table notes explained in this
document, you can use Pandoc for converting LaTeX to html.
* Additional tips and tricks
This section points some additional solutions that you may like to
use. Some of these may come handy when you start using Org for
documenting your research.
** Defining LaTeX classes
If are using my emacs configuration, you will have access to three
custom classes: vreport, vmemoir and varticle. Of these, vmemoir and
varticle are based on the [[https://ctan.org/pkg/memoir?lang=en][Memoir]] class, a very versatile LaTeX package
that allows for many customisations.
While vmemoir and varticle classes should work for books and articles,
you will benefit by looking at [[https://github.com/ep624/emacs.d.2021/blob/main/_configs/use-org-contrib.el][~/.emacs.d/_configs/use-org-contrib.el]]
and customising them for your need.
LaTeX class is specified in the org header using a line as follows
~#+LATEX_CLASS: vmemoir#~
Additional options can be provided by using the keyword ~latex_class_options~:
~#+latex_class_options: [11pt,twoside,openany,strict,extrafontsizes,article]~
Those using my emacs configuration can call the snippet latex-class,
taken from Scimax, using keyword ~lc~ to insert this line. The snippet
shows LaTeX classes that are defined in your installation of
org. Additional options for any latex class can be specified by
calling another snippet using ~lco~.
** Evaluating code during export
By default, Org evaluates source code at the time of exporting. If
your code involves a lot of computation, this can slow down exporting.
You can block evaluation of a source code block at the time of export
by using ~:eval never-export~ in the header arguments of the
block. Such code blocks will have to be evaluated manually using ~C-c
C-c~. To prevent all blocks from being evaluated, set it buffer-wide
using:
: #+PROPERTY: header-args :eval never-export~
If your buffer has this line, the source code is not evaluated at the
time of export, and whatever already exists in ~#+RESULTS~ block is
exported.
** Fonts
I like to use the Garamond font. If you do too, add this special line at
the top:
: #+LaTeX_CLASS_OPTIONS: [garamond]
** TODO Page size and margins
In LaTeX, package ~geometry~ allows you to modify page margins. The
following line in [[https://github.com/vikasrawal/orgpaper/blob/master/research-toolkit.org][research-toolkit.org]] sets the margins. You can tweak
this to define the margins as you like.
: ("innermargin=1.5in,outermargin=1.25in,vmargin=1.25in" "geometry" t)
If you would like to do it for each document separately, remove the
above line, and add the following special line at the top in your
documents.
: #+LaTeX_HEADER: \usepackage[innermargin=1.5in,outermargin=1.25in,vmargin=3cm]{geometry}
*** TODO Add instructions for memoir
** Line spacing
Use the following line at the top. Modify the number to whatever suits
you.
: #+LATEX_HEADER: \linespread{1.3}
** Acknowledgements in footnote
When writing a research paper, it is common to put acknowledgements in
a special footnote to names of authors. It is conventional to use * as
the symbol for this footnote, and to keep this footnote out of the
list of numbered footnotes that the paper may have.
This is achieved as follows.
+ As illustrated in the example below, add acknowledgements in the
special line that specifies authors of the paper.
: #+AUTHOR: Vikas Rawal\footnote{Write your acknowledgements here...}
+ Then, before your first headline, add the following text.
#+BEGIN_SRC org
#+BEGIN_LaTeX
{% begin group
\renewcommand{\thefootnote}{\fnsymbol{footnote}}% set smybols
\setcounter{footnote}{0}% set footnote counter back to 0
}% end group
#+END_LaTeX
#+END_SRC
** Restricting location of tables and images in LaTeX export
LaTeX has a very sophisticated algorithm for determining the location
of Tables and Images in a document. If, however, you want to add a
restriction that the Tables and Images should not cross section
boundaries, or a particular boundary, this can be done using command
~\FloatBarier~ provided by [[http://www.ualberta.ca/afs/ualberta.ca/sunsite/ftp/pub/Mirror/CTAN/help/Catalogue/entries/placeins.html][placeins]] package in LaTeX.
You can put any number of ~\FloatBarrier~ commands, each in a line by
itself, in the document. Tables and Images before such a barrier will
be placed before the barrier.
You can use the following special line at the top to restrict all
Tables and Images within their own sections.
: #+LATEX_HEADER: \usepackage[section]{placeins}
An extension to placeins package, [[http://lexfridman.com/blogs/research/2011/03/06/prevent-figures-from-floating-outside-sections-in-latex/][extraplaceins]] can be used if you
want to restrict the Tables and Images within subsections.[fn:7]
** Customising Biblatex style
I like to use authoryear bibliography style. However, I need some
customisations. The file [[https://raw.githubusercontent.com/vikasrawal/orgpaper/master/vikas-bibstyle.org][vikas-bibstyle.org]] contains all my
customisations.
Download the file, adjust the path to the file in the line below and
add it to your org file.
~#+INCLUDE: /path-to-the-file/vikas-bibstyle.org~
** Index
* Important Resources
+ [[http://orgmode.org/manual/][Org-mode manual]]
+ [[http://orgmode.org/worg/][Worg]]
+ [[http://orgmode.org/community.html][Org-mode mailing list]]
+ [[http://www.gnu.org/software/emacs/manual/emacs.html][Emacs manual]]
+ [[http://www.r-project.org][R website]]
+ [[http://johnmacfarlane.net/pandoc/][Pandoc]]
+ E. Schulte, D. Davison, T. Dye, and C. Dominik. A multi-language
computing environment for literate programming and reproducible
research. Journal of Statistical Software, 46(3):1–24, 1 2012.[[http://www.jstatsoft.org/v46/i03][http://www.jstatsoft.org/v46/i03]]
+ [[http://draketo.de/english/emacs/writing-papers-in-org-mode-acpd][Tutorial: Writing scientific papers for ACPD using emacs org-mode, http://draketo.de/english/emacs/writing-papers-in-org-mode-acpd]]
+ [[http://nakkaya.com/2010/09/07/writing-papers-using-org-mode/][Writing papers Using org-mode, http://nakkaya.com/2010/09/07/writing-papers-using-org-mode]]
* Footnotes
[fn:1] These are available at http://mirrors.ctan.org/macros/latex/required/tools/tabularx.pdf and http://mirrors.ctan.org/macros/latex/contrib/siunitx/siunitx.pdf.
[fn:3] Depending on the keyboard and the default configuration of the
flavour of emacs you have installed, Meta may instead be mapped to a
different key (for example, Windows key, or Option or Command key in
Apple computers.
[fn:4] For libraries and functions that you need to call, it is even
better to include them in a .Rprofile file in your working
directory. These libraries and functions would then be called when R
is started, and not each time you evaluate code blocks in your
document.
[fn:5] Of various image formats, I find that png files are most
versatile. png files support transparency, and are rendered well both
on the web and in print. You can also specify jpeg or pdf files. pdf
files for images work very well if you are only going to produce a pdf
document.
[fn:6] Author of the odt exporter has chosen to develop the exporter
outside Org-mode. He has developed a JabRef exporter to integrate
citations into odt exports, but that is not a part of Org-mode and
needs to be installed separately. In any case, since our toolkit
primarily uses LaTeX, using Pandoc to create odt or docx files from
LaTeX export works better.
[fn:7] See http://lexfridman.com/blogs/research/2011/03/06/prevent-figures-from-floating-outside-sections-in-latex/