High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Deploy Kubernetes and JupyterHub on Jetstream with Magnum
This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub. Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials.
-
White noise NET in radio-astronomy and cosmology
Create a white noise map and compare with power spectrum expected from the NET
-
ECSS Symposium introduction to cloud computing with Jetstream and deployment of Kubernetes
Note: XSEDE has been replaced by ACCESS. ECSS Symposium, April 2020, Web presentation to the XSEDE Extended Collaborative Support Services.
-
Use the Jetstream object store
I plan to collect here notes about using the Openstack object store on Jetstream. Get Amazon-style credentials
-
Raise and check a flag array with numpy
Handle flag array bits with numpy bitwise operations
-
Setup HTTPS on Kubernetes with Let's Encrypt
This tutorial is obsolete since September 2023, see the updated tutorial. Updated in August 2022: Add patching script to run on master node
-
Kill Jupyter Notebook servers
Just today I learned how to properly stop previously running Jupyter Notebook servers, here for future reference: jupyter notebook stop
-
Migrate from Pelican to Fastpages
I have been using the Pelican static website generator for a few years, hosting the content on Github, automatically build on push via Travis-CI and deploy on Github pages to zonca.github.io.
-
Deploy CVMFS on Kubernetes
CVMFS is a software distribution service, it is used by High Energy Physics experiments at CERN to synchronize software environments across the whole collaborations.
-
Organize calendars for a large scientific collaboration
Many scientific collaborations have a central calendar, often hosted on Google Calendar, to coordinate Teleconferences, meetings and events across timezones.
-
Simulate users on JupyterHub
Updated January 2021 I currently have 2 different strategies to deploy JupyterHub on top of Kubernetes on Jetstream:
-
Execute Jupyter Notebooks not interactively
Over the years, I have explored how to scale up easily computation through Jupyter Notebooks by executing them not-interactively, possibily parametrized and remotely. This is mostly for reference.
-
Deploy Cluster Autoscaler for Kubernetes on Jetstream
The Kubernetes Cluster Autoscaler is a service that runs within a Kubernetes cluster and when there are not enough resources to accomodate the pods that are queued to run, it contacts the API of the cloud provider to create more Virtual Machines to join the Kubernetes Cluster.
-
Create a Github account for your research group with free private repositories
Github allows a research group to create their own webpage where they can host, share and develop their software using the git version control system and the powerful Github online issue-tracking interface.
-
Ship large files with Python packages
It is often useful to ship large data files together with a Python package, a couple of scenarios are: data necessary to the functionality provided by the package, for example images, any binary or large text dataset, they could be either required just for a subset of the functionality of the package or for all of it