Very powerful data analysis environment – org mode with ob-ipython

You can embed embeded source code and evaluate it with C-c C-c. Results of evaluation of your source code are appended after the source code block. Result can be text (including org table) or image (charts).

What’s more You can have separate org file and ipython console open side by side. With ipython, reading python docstrings and code completion works well. See my screenshot.

Since ob-ipython uses jupyter, you can get the same environment for anything that have jupyter kernel, including matlab, Scala, Spark or R and many more.

This blog post is just an export of org mode file via org2blog. All code examples have been written in org mode using workflow described in this post.

Exporting works to formats like html, latex (native and beamer), markdown, jira, odt (than can be imported to google docs and word), wiki formats and many more.

Syntax highlighting can be preserved for some exports, like html or latex.

You can just learn one way to edit documents and presentations than can be exported to majority of formats on earth.

Emacs org mode with org babel is a full fledged literate programming environment. Some people have published whole books or research papers as a large executable document in org. There is an even Research paper about it.

Python computations in science and engineering book supports org mode and it’s far better book reading experience than anything I ever experienced before. I can tweak and re-run code examples, link from my other notes, tag or bookmark interesting sections, jump between sections and many more.

When writing some latex in college, I recall situations when I am half way through writing latex document. I would came up with the idea of some parameter tweak, and suddenly I have to re-generate all charts.

With org mode, the document is generated pragmatically. Not only you can easily re-generate it, but readers of your document can tweak parameters or supply their own data set and re-generate the whole document.

Another example is training machine model. You can define your model parameters as org constants. You can tweak some model parameter and have separate org mode headings for things like “performance statistics”, “top miss-classified cross validation samples”, etc. Added benefit is that you can commit all this to git.

As soon as you learn org mode all of it is easy and seamless.

Sometimes just “manually” editing the data is the most productive thing to do. You can do it with org mode spreadsheet capabilities on org tables.

The added benefit is that formulas are written in lisp, that is cooler and more powerful language than Visual basic. http://orgmode.org/manual/Translator-functions.html

Similar functionality is offered by beaker notebook.

I found out that org mode as intermediate format for data sometimes works better for me.

Since intermediate format for a data frame is the org table, I can import data frame to org, edit it as spreadsheet and export it back. See Pass data directly between languages in examples section.

Org mode outline view is very handy for organizing your work. When working on some larger problem, I am only focusing on small subset of it. Org mode lets me just expand sections that are currently relevant.

I also find adding embedding TODO items in the tree quite handy. When I encounter some problem I mark a subtree as TODO, and I can later inspect just subtree headlines with TODO items with them. See:

You can link to your existing codebase with org-ctags. It seems possible to provide ide-like navigation between code defined in org src buffers, but I didn’t configure it yet.

You don’t have to use all features offered by org mode.

Although I don’t use it, I recommend Spacemacs, pre-configured emacs distribution, like “Ubuntu” of Emacs.

If you don’t run those, you may run into troubles.

pip install --upgrade pip
pip install --upgrade ipython
pip install --upgrade pyzmq
pip install --upgrade jupyter

org mode should be bundled with your emacs installation. If you are new to emacs, you can install packages using M-x package-install.

Add to your Emacs config:

(require ‘org)
(require ‘ob-ipython)

;; don’t prompt me to confirm everytime I want to evaluate a block
(setq org-confirm-babel-evaluate nil)

;;; display/update images in the buffer after I evaluate
(add-hook ‘org-babel-after-execute-hook ‘org-display-inline-images ‘append)

(ob-ipython-kill-kernel)

(toggle-debug-on-error)

If you want to run some code in each ipython block you can add it to ~/.ipython/profile_default/startup. Foe example, to avoid adding %matplotlib inline to each source code block:

echo "%matplotlib inline" >> ~/.ipython/profile_default/startup/66-matplot.py

ob-ipython docs suggest yasnippet for editing code. So far I have been using custom elisp code, but a few things can be nicer about yasnippet.

# -*- mode: snippet -*-
# name: ipython block
# key: py
# —
#+BEGIN_SRC ipython :session ${1::file ${2:$$(let ((temporary-file-directory "./")) (make-temp-file "py" nil ".png"))} }:exports ${3:both}
$0
#+END_SRC

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
%matplotlib inline

df = pd.DataFrame(table[1:], columns=table[0])
df.plot()

You have to write small reusable snippet to print pandas to org format. You can add it to your builtin ipython code snippets. You also need to tell src block to interpret results directly with :results output raw drawer :noweb yes.

def arr_to_org(arr):
line = "|".join(str(item) for item in arr)
return "|{}|".format(line)

def df_to_org(df):
return "\n".join([arr_to_org(df.columns)] +
[arr_to_org(row) for row in df.values])

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
%matplotlib inline

df = pd.DataFrame(table[1:], columns=table[0])
df.y = df.y.apply(lambda y: y * 2)
print df_to_org(df)

Afterwards, you may assign result table to variable, edit it with org spreadsheet capabilities and use in other python script.

Since all code is executed within the same ipython kernel, it’s enough to put blocks one after another.

constant = 30

def some_function(x):
return constant * x

print some_function(30)

I added support of connecting to existing ipython kernel in https://github.com/gregsexton/ob-ipython/pull/71/files.

You can start an ipython kernel on a server with lots of ram and cpu and connect it to a local lightweight machine running emacs.

Create kernel using (outside of the org mode, as it blocks):

#!/usr/bin/env python
import os
from ipykernel.kernelapp import IPKernelApp

app = IPKernelApp.instance()
app.initialize([])
kernel = app.kernel
kernel.shell.push({'print_me': 'Running in previously started kernel.'})

app.start()

It will give you a connection json file name. Pass it as a session name.

#+BEGIN_SRC ipython :session kernel-8520.json
  print print_me
#+END_SRC

Running in previously started kernel.

Create my example based on http://minimallysufficient.github.io/2015/10/24/org-mode-as-an-alternative-to-knitr.html

This should work:

#+BEGIN_SRC ipython :session :kernel clojure
(+ 1 2)
#+END_SRC

#+RESULTS:
: 3

C-c C-c block to open org file directly in Emacs:

(browse-url-emacs
"https://raw.githubusercontent.com/dfeich/org-babel-examples/master/python/pythonbabel.org")

(browse-url-emacs
"https://raw.githubusercontent.com/dfeich/org-babel-examples/master/python/ipython-babel.org")

Currently it opens a separate buffer. I would prefer a popup.

Currently, I have code completion only working in ipython buffer. It seems doable to configure it in the edit source block as well.

To avoid manual copying between ipython buffer and source code block, I could implement an ob-ipython-capture function, that would add last executed command in the ipython console to the src block. Keyboard macros can work cross-buffer, so this could be simple keyboard macro, but I didn’t try it out yet.

In order to make a svg graphic rather than png, you may specify the output format globally to IPython.

%config InlineBackend.figure_format = 'svg'

Open org directly in Emacs:

(browse-url-emacs
"https://raw.githubusercontent.com/gregsexton/ob-ipython/master/README.org")

Search by DOI 10.1145/2723872.2723881 on sci hub.

Interesting case of Chemical Engineering department on CMU managed by John R. Kitchin, ran mostly using org mode, with papers, assignments and books written in org mode.

“Literate devops”

https://github.com/seth/my-emacs-dot-d/blob/master/emacs-init.org http://mescal.imag.fr/membres/arnaud.legrand/misc/init.php http://emacs.stackexchange.com/questions/3143/can-i-use-org-mode-to-structure-my-emacs-or-other-el-configuration-file