High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Deploy Kubernetes and JupyterHub on Jetstream with Magnum
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub.
-
Webinar about distributed computing with Python
Recording available of the webinar I gave about "Distributed computing with Python": Threads vs Processes, GIL
-
Kubernetes monitoring with Prometheus and Grafana
See the updated version of this tutorial Updated September 2020
-
Inherit group permission in folder
I have googled this so many times... On shared systems, like Supercomputers, you often belong to many different Unix groups, and that membership allows you to access data from specific projects you are working on and you can share data with your collaborators.
-
Kubernetes monitoring with Dashboard, Prometheus, and Grafana
Install with helm: https://github.com/helm/charts/tree/master/stable/prometheus-operator
-
Scale Kubernetes manually on Jetstream
We would like to modify the number of Openstack virtual machines available to Kubernetes. Ideally we would like to do this automatically based on the load on JupyterHub, that is the target. For now we will increase and decrease the size manually.
-
Deploy Kubernetes with Kubespray 2.8.2 and JupyterHub with helm recipe 0.8 on Jetstream
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. Back in September 2018 I published a tutorial to deploy Kubernetes on Jetstream using Kubernetes.
-
Use the distributed file format Zarr on Jetstream Swift object storage, 2019
This is an updated version of the 2018 edition Zarr
-
Deploy Pangeo on Kubernetes deployment on Jetstream created with Kubespray
The Pangeo collaboration for Big Data Geoscience maintains a helm chart with a prefigured JupyterHub deployment on Kubernetes which also supports launching private dask workers.
-
Setup two factor authentication for UCSD, and Lastpass
Starting at the end of January 2019 UCSD requires every employee to have activated two factor authentication. Go over to to register your devices and to read more details.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Advanced pandas with astrophysics example notebook
Taught a lesson today on advanced python and pandas based on an example application in Astrophysics with simulations of data from the Planck Satellite), features also a Binder button to run it yourself. Jupyter Notebook available at: under CC-BY
-
Bring your computing to the San Diego Supercomputer Center
Note (2026 update): This post was written in 2018. Comet has been retired and XSEDE has been replaced by ACCESS. SDSC's current flagship system is Expanse. For updated information about the HPC@UC program for UC researchers, see my 2026 post on requesting an HPC@UC allocation on Expanse.
-
Deploy JupyterHub on Kubernetes deployment on Jetstream created with Kubespray 3/3
All of the following assumes you are logged in to the master node of the Kubernetes cluster deployed with kubespray and checked out the repository: