Run IPython Notebook on a HPC Cluster via PBS


December 18, 2013

The IPython notebook is a great tool for data exploration and visualization. It is suitable in particular for analyzing a large amount of data remotely on a computing node of a HPC cluster and visualize it in a browser that runs on a local machine. In this configuration, the interface is local, it is very responsive, but the amount of memory and CPU horsepower is provided by a HPC computing node.

Also, it is possible to keep the notebook server running, disconnect and reconnect later from another machine to the same session.

I created a script which is very general and can be used on most HPC cluster and published it on Github:

Once the script is running, it is possible to connect to localhost:PORT and visualize the IPython notebook, see the following screenshot of Chromium running locally on my machine connected to a IPython notebook running on a Gordon computing node:

IPython notebook on Gordon