High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Setup two factor authentication for UCSD, and Lastpass
Starting at the end of January 2019 UCSD requires every employee to have activated two factor authentication. Go over to to register your devices and to read more details.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Advanced pandas with astrophysics example notebook
Taught a lesson today on advanced python and pandas based on an example application in Astrophysics with simulations of data from the Planck Satellite), features also a Binder button to run it yourself. Jupyter Notebook available at: under CC-BY
-
Bring your computing to the San Diego Supercomputer Center
Note (2026 update): This post was written in 2018. Comet has been retired and XSEDE has been replaced by ACCESS. SDSC's current flagship system is Expanse. For updated information about the HPC@UC program for UC researchers, see my 2026 post on requesting an HPC@UC allocation on Expanse.
-
Deploy JupyterHub on Kubernetes deployment on Jetstream created with Kubespray 3/3
All of the following assumes you are logged in to the master node of the Kubernetes cluster deployed with kubespray and checked out the repository:
-
Explore a Kubernetes deployment on Jetstream with Kubespray 2/3
This is the second part of the tutorial on deploying Kubernetes with kubespray and JupyterHub on Jetstream. In the first part, we installed Kubernetes on Jetstream with kubespray.
-
Deploy Kubernetes on Jetstream with Kubespray 1/3
Note: Jetstream 1 has been retired. For current Kubernetes deployments, see Jetstream 2 documentation. This tutorial is obsolete, check the updated version of the tutorial
-
PEARC18 paper on deploying Jupyterhub at scale on XSEDE
Bob Sinkovits and I are presenting a paper at PEARC18 about: "Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops"
-
Updated Singularity images for Comet
Back in January 2017 I wrote a blog post about running Singularity on Comet. I recently needed to update all my container images to the latest scientific python packages, so I also took the opportunity to create both a Docker auto-build repository on DockerHub and a SingularityHub image.
-
Create DockerHub auto build
It is very convenient to create Autobuild repositories on DockerHub linked to a Github repository with a Dockerfile. Then every time you commit to Github, Dockerhub is going to build the image on their service and make it available on and can quickly be pulled to any other system that supports Docker or Singularity.
-
How to organize code and data for simulations at NERSC
I recently improved my strategy for organizing code and data for simulations run at NERSC, I'll write it here for reference.
-
Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to launch a set of dask workers as containers running inside Kubernetes itself and use them for distributed computing.
-
How to post a PEARC18 paper pre-print to Arxiv
Follows the step-by-step version: Why upload a pre-print to arXiv
-
Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream
Let's assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a Jupyter Notebook, we would also like to provide them more computing power for their interactive data exploration.