High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Buy US Treasury bonds on Fidelity
The Fidelity interface is complicated, here a description of the steps necessary to buy newly issued US Treasury bonds at the next available auction. Currently auctions are every Thursday.
-
Gateways 2023 tutorial about Dask and JupyterHub on Kubernetes on Jetstream
Gateways 2023 tutorial about Dask and JupyterHub on Kubernetes on Jetstream
-
Deploy Dask Gateway with JupyterHub on Kubernetes
In this tutorial we will install Dask Gateway, currently version 2023.9.0, on Kubernetes and configure JupyterHub so Jupyter Notebook users can launch private Dask cluster and connect to them.
-
Deploy GitHub Authenticator in JupyterHub
Updated June 2024: added more options to config file Quick reference on how to deploy the Github Authenticator in JupyterHub, the main reference is the Zero to JupyterHub docs.
-
Setup HTTPS on Kubernetes with cert-manager
Update March 2024: the routing issue that force cert-manager pods to run on the control-plane are back, see this Github issue, so we had to add back the pinning of cert-manager services on one of the nodes in the control plane.
-
Tutorial on Python for HPC at the SDSC Summer Institute
Each August the San Diego Supercomputer Center organizes a week-long summer school that teaches a large number of topics related to Supercomputing and AI to early career scientists.
-
Deploy Kubernetes on Jetstream 2 with Kubespray 2.21.0
::: {.callout-note} Obsolete Tutorial This tutorial is obsolete. Please refer to the Magnum tutorial to deploy Kubernetes instead. :::
-
Hiring a computational scientist at the San Diego Supercomputer Center
Come work in my group of High Performance Computing "consultants" at the San Diego Supercomputer Center, help scientists develop, scale and run Data Intensive computations on our Supercomputers.
-
Cosmology webinar Hyper Suprime-Cam
Webinar about Cosmology results, in particular tension in $S8$ by the Hyper Supreme Cam collaboration: Video on Youtube
-
Get a Letsencrypt certificate for Dreamhost with certbot
Dreamhost has discontinued offering free Letsecrypt certificated in their shared hosting. However, they support pasting a manually created certificate, so we can generate one on a different machine with the manual mode of certbot and then deploy it.
-
Update OpenStack credentials in Kubernetes
If you are deploying Kubernetes on top of Openstack, the Openstack External cloud provider stores the ID and Secret necessary to authenticate with the cloud infrastructure in a Secret.
-
Work on a LaTeX document in GitHub Codespaces
I was exploring using Github Codespaces instead of Overleaf to work on a Latex document. The most important missing component is the Latex editor, and the other downside is that it is free only for 90 hours per month (with a Pro academic account). However, the other useful features of Overleaf are present:
-
Gzipping or not gzipping HEALPix maps
HEALPix maps are generally stored in FITS format, they can be Gzipped to save disk space. Reading gzipped maps is natively supported in healpy, however, it takes longer because maps are uncompressed on the fly.
-
Share data from NERSC publicly via web
NERSC allows to make data which is on their Community File System publicly available via a web interface. This is handled via "Project folders", so the requirement is to be part of a "project", for example most scientists working on Cosmic Microwave Background are part of the cmb project (and of the cmb Unix group at NERSC):
-
Showyourwork! Globus demo
Showyourwork! is a great project to make scientific articles reproducible. I participated to its first Hack Day, I focused on building a simple recipe in SnakeMake (the workflow engine behind Showyourwork!) to interface with data stored on Globus.