High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Workflow for Jupyter Notebooks under version control
I'll present here my strategy for keeping Jupyter Notebooks under version control. TLDR:
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial OBSOLETE Please check the updated version of this tutorial. This tutorial follows the work by the Pangeo collaboration, the main difference is that I prefer to keep JupyterHub and the Dask infrastructure in 2 separate Helm recipes.
-
Make for repeated commands
I often find myself repeat the same commands in a project, for example in Openstack/Kubernetes, but also when I need to interface with Docker (build/push image).
-
Fund healpy via GitHub Sponsors
Fund healpy via GitHub Sponsors
-
Kubernetes certifications CKA and CKAD
I recently pursued 2 Kubernetes certifications by Linux Foundation: Certified Kubernetes Administrator (CKA)
-
Compare HEALPix and Gauss-Legendre pixelizations for sky emission modelling
import healpy as hp import numpy as np import matplotlib.pyplot as plt from time import time from astropy import units as u
-
Deploy Hadoop on Kubernetes on Jetstream
We are deploying the good old Hadoop on top of Kubernetes on Jetstream. Don't ask why. As usual we start with a full-fledged Kubernetes deployment on Jetstream (1) deployed via Kubespray
-
Plot HEALPix pixel boundaries
Some time ago I created a Jupyter Notebook based on plotly to create a 3D visualization of the boundaries of HEALPix pixels on the sphere.
-
Monitor traffic on Github repositories
Github only shows traffic data for the last 2 weeks. Through the API is possible to gather those data every 2 weeks and save them for later collection and reduction.
-
Manage Globus groups with the Python SDK
Globus is the best tool to transfer quickly Terabytes data between Supercomputers because it automatically parallelizes the transfer to saturate the network (yeah I know I always simplify too much).
-
New paper about PySM 3
Update November 2021 Paper was published on JOSS:
-
Upload a JOSS paper to arXiv
If you are in Astrophysics, you probably want to have your JOSS paper published to the Arxiv. Unfortunately some authors don't get it accepted due to the paper being too short for the Arxiv standard.
-
Configure dreamhost email
Configuration to read Dreamhost email from Gmail: Receive
-
healpy 1.15.0 released
Just released healpy 1.15.0, this is a large release with many important changes, please test and open issues at New viz function projview based on plain matplotlib, with lon/lat labels and more
-
Investigate rotation of masks in healpy
Investigating a question about healpy on Stackoverflow python import healpy as hp import numpy as np %matplotlib inline