High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Deploy Hadoop on Kubernetes on Jetstream
We are deploying the good old Hadoop on top of Kubernetes on Jetstream. Don't ask why. As usual we start with a full-fledged Kubernetes deployment on Jetstream (1) deployed via Kubespray
-
Plot HEALPix pixel boundaries
Some time ago I created a Jupyter Notebook based on plotly to create a 3D visualization of the boundaries of HEALPix pixels on the sphere.
-
Monitor traffic on Github repositories
Github only shows traffic data for the last 2 weeks. Through the API is possible to gather those data every 2 weeks and save them for later collection and reduction.
-
Manage Globus groups with the Python SDK
Globus is the best tool to transfer quickly Terabytes data between Supercomputers because it automatically parallelizes the transfer to saturate the network (yeah I know I always simplify too much).
-
New paper about PySM 3
Update November 2021 Paper was published on JOSS:
-
Upload a JOSS paper to arXiv
If you are in Astrophysics, you probably want to have your JOSS paper published to the Arxiv. Unfortunately some authors don't get it accepted due to the paper being too short for the Arxiv standard.
-
Configure dreamhost email
Configuration to read Dreamhost email from Gmail: Receive
-
healpy 1.15.0 released
Just released healpy 1.15.0, this is a large release with many important changes, please test and open issues at New viz function projview based on plain matplotlib, with lon/lat labels and more
-
Investigate rotation of masks in healpy
Investigating a question about healpy on Stackoverflow python import healpy as hp import numpy as np %matplotlib inline
-
Learn Italian in San Diego
My wife Maura D'Andrea recently opened a new Italian language school in San Diego. All instructors are mother-tongue Italian, they teach to kids and adults.
-
Migrate from Google Docs to Overleaf and GitHub
Once a document on Google Docs becomes too big and complicated to make sure it is consistent, it is a good idea to migrate it to Latex. In particular we can support both Google Docs style editing via Overleaf and Pull Request reviewing via Github.
-
Correct a power spectrum by the instrument beam with healpy
If you are analyzing a map from an instrument with a specific beam width, you can correct the power spectrum by the smoothing factor caused by that beam and obtain a better approximation of the power spectrum of the orignal sky.
-
Backup Kubernetes volumes to OpenStorageNetwork object store
In my specific scenario, I have users running JupyterHub on top of Kubernetes on the Jetstream XSEDE Cloud resouce. Each user has a persistent volume as their home folder of a few GB.
-
Science Gateways Tech blog - Kubernetes and JupyterHub on Jetstream
This is cross-posted from the Science Gateways Community institute tech blog, thanks to Marlon Pierce and Katherine Lawrence for their feeback.
-
Stream video from object store on Jetstream
In this tutorial we will serve video stored in Jetstream's object store (like Amazon S3) as HTTP live streaming to a user' browser. This makes it difficult for the user to save the whole video as it is served in chunks.