High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Deploy a NFS server to share data between JupyterHub users on Jetstream
This tutorial is a minor update of . Also consider that a more robust and low-maintenance way of providing shared data volumes is to rely on Manila shares provided by Jetstream 2, see the tutorial
-
Work on a LaTeX document with GitHub and Overleaf
In the past I have written about coordinating a large document using multiple overleaf documents and Github repositories.
-
Deploy Kubernetes on Jetstream 2 with GPU support
This tutorial is obsolete, see the new version of the tutorial. The Jetstream 2 cloud includes 90 GPU nodes with 4 NVIDIA A100 each{target=\blank}. If we want to leverage the GPUs inside Kubernetes pods, for example JupyterHub users, we both need to have a GPU-enabled ContainerD runtime and a compatible Docker image based off NVIDIA images.
-
Share data between JupyterHub users with Manila shares
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website .
-
Compare WebSky radio galaxies maps from PySM to Planck
I need to check that the amplitude of the maps of Radio Galaxies produced by WebSky are reasonable if compared to experimental results. The main worry is that being point sources, when we apply a beam using Spherical Harmonics transform we get spurious results due to the input maps not being band-limited.
-
Deploy BinderHub on top of Kubernetes on Jetstream 2
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website .
-
Setup a conda environment at NERSC
Obsolete: see the new version NERSC recommends to use the "Global Common" to store Conda environments, because it is optimized to store lots of small files. Consider that it is mounted read-only on computing nodes. The filesystem is organized by groups, so choose one of your groups:
-
Build and host Singularity containers on GitHub
I present a demo repository configured to build Singularity containers using Github actions and then host them using the Github container registry:
-
Generate Click command line options dynamically from class arguments
In this application I need to use command line options to create class objects. In order not to repeat all class arguments in the Click configuration, I created a simple function to dynamically create Click options. See the source notebook of this page on Github
-
Closures in Numba
Closures in Numba
-
Install the JupyROOT Python kernel in JupyterHub
After installing ROOT's conda package, for example with micromamba: micromamba create -n root -c conda-forge root python==3.8 matplotlib
-
Singularity on Expanse tutorial
As one of my last tasks in XSEDE, I updated and improved the tutorial about running Singularity containers in the HPC system Espanse at my institution the San Diego Supercomputer Center.
-
Migrate from fastpages to quarto preserving git history
Most of my blog posts are in Markdown...however, there are cases where plotting and code are very important, and nothing beats a Jupyter Notebook for that.
-
Deploy MariaDB on Jetstream 2 on top of Kubernetes
In this tutorial we will install a MariaDB instance backed by a persistent volume on Jetstream 2. It will be in the jhub namespace, so that it can be accessed by the JupyterHub users and from no other namespace.
-
Remove unique cell id from Jupyter Notebooks
I know! Jupyter is littering your git diff with randomly generated cell ids and nbstripout doesn't remove them, (I'm sure they are useful for some reason).