High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Get a Letsencrypt certificate for Dreamhost with certbot
Dreamhost has discontinued offering free Letsecrypt certificated in their shared hosting. However, they support pasting a manually created certificate, so we can generate one on a different machine with the manual mode of certbot and then deploy it.
-
Update OpenStack credentials in Kubernetes
If you are deploying Kubernetes on top of Openstack, the Openstack External cloud provider stores the ID and Secret necessary to authenticate with the cloud infrastructure in a Secret.
-
Work on a LaTeX document in GitHub Codespaces
I was exploring using Github Codespaces instead of Overleaf to work on a Latex document. The most important missing component is the Latex editor, and the other downside is that it is free only for 90 hours per month (with a Pro academic account). However, the other useful features of Overleaf are present:
-
Gzipping or not gzipping HEALPix maps
HEALPix maps are generally stored in FITS format, they can be Gzipped to save disk space. Reading gzipped maps is natively supported in healpy, however, it takes longer because maps are uncompressed on the fly.
-
Share data from NERSC publicly via web
NERSC allows to make data which is on their Community File System publicly available via a web interface. This is handled via "Project folders", so the requirement is to be part of a "project", for example most scientists working on Cosmic Microwave Background are part of the cmb project (and of the cmb Unix group at NERSC):
-
Showyourwork! Globus demo
Showyourwork! is a great project to make scientific articles reproducible. I participated to its first Hack Day, I focused on building a simple recipe in SnakeMake (the workflow engine behind Showyourwork!) to interface with data stored on Globus.
-
Deploy a NFS server to share data between JupyterHub users on Jetstream
This tutorial is a minor update of . Also consider that a more robust and low-maintenance way of providing shared data volumes is to rely on Manila shares provided by Jetstream 2, see the tutorial
-
Work on a LaTeX document with GitHub and Overleaf
In the past I have written about coordinating a large document using multiple overleaf documents and Github repositories.
-
Deploy Kubernetes on Jetstream 2 with GPU support
This tutorial is obsolete, see the new version of the tutorial. The Jetstream 2 cloud includes 90 GPU nodes with 4 NVIDIA A100 each{target=\blank}. If we want to leverage the GPUs inside Kubernetes pods, for example JupyterHub users, we both need to have a GPU-enabled ContainerD runtime and a compatible Docker image based off NVIDIA images.
-
Share data between JupyterHub users with Manila shares
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website .
-
Compare WebSky radio galaxies maps from PySM to Planck
I need to check that the amplitude of the maps of Radio Galaxies produced by WebSky are reasonable if compared to experimental results. The main worry is that being point sources, when we apply a beam using Spherical Harmonics transform we get spurious results due to the input maps not being band-limited.
-
Deploy BinderHub on top of Kubernetes on Jetstream 2
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website .
-
Setup a conda environment at NERSC
Obsolete: see the new version NERSC recommends to use the "Global Common" to store Conda environments, because it is optimized to store lots of small files. Consider that it is mounted read-only on computing nodes. The filesystem is organized by groups, so choose one of your groups:
-
Build and host Singularity containers on GitHub
I present a demo repository configured to build Singularity containers using Github actions and then host them using the Github container registry:
-
Generate Click command line options dynamically from class arguments
In this application I need to use command line options to create class objects. In order not to repeat all class arguments in the Click configuration, I created a simple function to dynamically create Click options. See the source notebook of this page on Github