High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Access running GitHub Action with SSH
Sometimes Github actions are failing and it is difficult to reproduce the error locally, in particular if you have a different OS.
-
Monitor Restic backups on Kubernetes
For one of my production JupyterHub deployments on Kubernetes, I have setup an automated system to perform nightly backup of the user data, see the full tutorial on how to set it up.
-
Jetstream2 SU calculator
Jupyter Notebook to compute daily, monthly, yearly consumption of SU based on the number and type of Virtual Machines:
-
Custos authentication for JupyterHub
Custos is a security middleware used to authenticate users to Airavata-based Science Gateways. It is relevant to the Science Gateways community to unify authentication and also authenticate users to JupyterHub using the same framework.
-
Science Gateway with Dask and Zarr
This material was presented on April 2022 at the MiniGateways 2022 conference organized by the wonderful Science Gateways Community Institute (SGCI).
-
Use the distributed file format Zarr on Jetstream 2 object storage
Zarr is a file format designed for cloud computing, see documentation. Zarr is also supported by dask, the parallel computing framework for Python, and the Dask team implemented storage backends for Google Cloud Storage and Amazon S3.
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial obsolete, see the new version of the tutorial Updated 28 April 2022: switched to Dask Gateway 2022.4.0
-
Deploy JupyterHub on Jetstream 2 on top of Kubernetes
This tutorial is a followup to: Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0, so I'll assume Kubernetes is already deployed with a default storageclass.
-
Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0
Obsolete: please use the updated release of this tutorial. Updated in August 2022 to add automatic jetstream-cloud.org subdomains
-
Workflow for Jupyter Notebooks under version control
I'll present here my strategy for keeping Jupyter Notebooks under version control. TLDR:
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial OBSOLETE Please check the updated version of this tutorial. This tutorial follows the work by the Pangeo collaboration, the main difference is that I prefer to keep JupyterHub and the Dask infrastructure in 2 separate Helm recipes.
-
Make for repeated commands
I often find myself repeat the same commands in a project, for example in Openstack/Kubernetes, but also when I need to interface with Docker (build/push image).
-
Fund healpy via GitHub Sponsors
Fund healpy via GitHub Sponsors
-
Kubernetes certifications CKA and CKAD
I recently pursued 2 Kubernetes certifications by Linux Foundation: Certified Kubernetes Administrator (CKA)
-
Compare HEALPix and Gauss-Legendre pixelizations for sky emission modelling
import healpy as hp import numpy as np import matplotlib.pyplot as plt from time import time from astropy import units as u