High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Organize calendars for a large scientific collaboration
Many scientific collaborations have a central calendar, often hosted on Google Calendar, to coordinate Teleconferences, meetings and events across timezones.
-
Simulate users on JupyterHub
Updated January 2021 I currently have 2 different strategies to deploy JupyterHub on top of Kubernetes on Jetstream:
-
Execute Jupyter Notebooks not interactively
Over the years, I have explored how to scale up easily computation through Jupyter Notebooks by executing them not-interactively, possibily parametrized and remotely. This is mostly for reference.
-
Deploy Cluster Autoscaler for Kubernetes on Jetstream
The Kubernetes Cluster Autoscaler is a service that runs within a Kubernetes cluster and when there are not enough resources to accomodate the pods that are queued to run, it contacts the API of the cloud provider to create more Virtual Machines to join the Kubernetes Cluster.
-
Create a Github account for your research group with free private repositories
Github allows a research group to create their own webpage where they can host, share and develop their software using the git version control system and the powerful Github online issue-tracking interface.
-
Ship large files with Python packages
It is often useful to ship large data files together with a Python package, a couple of scenarios are: data necessary to the functionality provided by the package, for example images, any binary or large text dataset, they could be either required just for a subset of the functionality of the package or for all of it
-
Deploy Kubernetes and JupyterHub on Jetstream with Magnum
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub.
-
Webinar about distributed computing with Python
Recording available of the webinar I gave about "Distributed computing with Python": Threads vs Processes, GIL
-
Kubernetes monitoring with Prometheus and Grafana
See the updated version of this tutorial Updated September 2020
-
Inherit group permission in folder
I have googled this so many times... On shared systems, like Supercomputers, you often belong to many different Unix groups, and that membership allows you to access data from specific projects you are working on and you can share data with your collaborators.
-
Kubernetes monitoring with Dashboard, Prometheus, and Grafana
Install with helm: https://github.com/helm/charts/tree/master/stable/prometheus-operator
-
Scale Kubernetes manually on Jetstream
We would like to modify the number of Openstack virtual machines available to Kubernetes. Ideally we would like to do this automatically based on the load on JupyterHub, that is the target. For now we will increase and decrease the size manually.
-
Deploy Kubernetes with Kubespray 2.8.2 and JupyterHub with helm recipe 0.8 on Jetstream
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. Back in September 2018 I published a tutorial to deploy Kubernetes on Jetstream using Kubernetes.
-
Use the distributed file format Zarr on Jetstream Swift object storage, 2019
This is an updated version of the 2018 edition Zarr
-
Deploy Pangeo on Kubernetes deployment on Jetstream created with Kubespray
The Pangeo collaboration for Big Data Geoscience maintains a helm chart with a prefigured JupyterHub deployment on Kubernetes which also supports launching private dask workers.