High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Closures in Numba
Closures in Numba
-
Install the JupyROOT Python kernel in JupyterHub
After installing ROOT's conda package, for example with micromamba: micromamba create -n root -c conda-forge root python==3.8 matplotlib
-
Singularity on Expanse tutorial
As one of my last tasks in XSEDE, I updated and improved the tutorial about running Singularity containers in the HPC system Espanse at my institution the San Diego Supercomputer Center.
-
Migrate from fastpages to quarto preserving git history
Most of my blog posts are in Markdown...however, there are cases where plotting and code are very important, and nothing beats a Jupyter Notebook for that.
-
Deploy MariaDB on Jetstream 2 on top of Kubernetes
In this tutorial we will install a MariaDB instance backed by a persistent volume on Jetstream 2. It will be in the jhub namespace, so that it can be accessed by the JupyterHub users and from no other namespace.
-
Remove unique cell id from Jupyter Notebooks
I know! Jupyter is littering your git diff with randomly generated cell ids and nbstripout doesn't remove them, (I'm sure they are useful for some reason).
-
Access running GitHub Action with SSH
Sometimes Github actions are failing and it is difficult to reproduce the error locally, in particular if you have a different OS.
-
Monitor Restic backups on Kubernetes
For one of my production JupyterHub deployments on Kubernetes, I have setup an automated system to perform nightly backup of the user data, see the full tutorial on how to set it up.
-
Jetstream2 SU calculator
Jupyter Notebook to compute daily, monthly, yearly consumption of SU based on the number and type of Virtual Machines:
-
Custos authentication for JupyterHub
Custos is a security middleware used to authenticate users to Airavata-based Science Gateways. It is relevant to the Science Gateways community to unify authentication and also authenticate users to JupyterHub using the same framework.
-
Science Gateway with Dask and Zarr
This material was presented on April 2022 at the MiniGateways 2022 conference organized by the wonderful Science Gateways Community Institute (SGCI).
-
Use the distributed file format Zarr on Jetstream 2 object storage
Zarr is a file format designed for cloud computing, see documentation. Zarr is also supported by dask, the parallel computing framework for Python, and the Dask team implemented storage backends for Google Cloud Storage and Amazon S3.
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial obsolete, see the new version of the tutorial Updated 28 April 2022: switched to Dask Gateway 2022.4.0
-
Deploy JupyterHub on Jetstream 2 on top of Kubernetes
This tutorial is a followup to: Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0, so I'll assume Kubernetes is already deployed with a default storageclass.
-
Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0
Obsolete: please use the updated release of this tutorial. Updated in August 2022 to add automatic jetstream-cloud.org subdomains