High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Explore a Kubernetes deployment on Jetstream with Kubespray 2/3
This is the second part of the tutorial on deploying Kubernetes with kubespray and JupyterHub on Jetstream. In the first part, we installed Kubernetes on Jetstream with kubespray.
-
Deploy Kubernetes on Jetstream with Kubespray 1/3
Note: Jetstream 1 has been retired. For current Kubernetes deployments, see Jetstream 2 documentation. This tutorial is obsolete, check the updated version of the tutorial
-
PEARC18 paper on deploying Jupyterhub at scale on XSEDE
Bob Sinkovits and I are presenting a paper at PEARC18 about: "Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops"
-
Updated Singularity images for Comet
Back in January 2017 I wrote a blog post about running Singularity on Comet. I recently needed to update all my container images to the latest scientific python packages, so I also took the opportunity to create both a Docker auto-build repository on DockerHub and a SingularityHub image.
-
Create DockerHub auto build
It is very convenient to create Autobuild repositories on DockerHub linked to a Github repository with a Dockerfile. Then every time you commit to Github, Dockerhub is going to build the image on their service and make it available on and can quickly be pulled to any other system that supports Docker or Singularity.
-
How to organize code and data for simulations at NERSC
I recently improved my strategy for organizing code and data for simulations run at NERSC, I'll write it here for reference.
-
Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to launch a set of dask workers as containers running inside Kubernetes itself and use them for distributed computing.
-
How to post a PEARC18 paper pre-print to Arxiv
Follows the step-by-step version: Why upload a pre-print to arXiv
-
Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream
Let's assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a Jupyter Notebook, we would also like to provide them more computing power for their interactive data exploration.
-
Install a BOINC server on Jetstream
BOINC is the leading platform for volunteer computing. Scientists can create a project on the platform and submit computational jobs that will be executed on computers of volunteers all over the world.
-
Use the distributed file format Zarr on Jetstream Swift object storage
Use the distributed file format Zarr on Jetstream Swift object storage
-
Use the distributed file format Zarr on Jetstream Swift object storage
Zarr is a pretty new file format designed for cloud computing, see documentation and a webinar for more details. Zarr is also supported by dask, the parallel computing framework for Dask, and the Dask team implemented storage backends for Google Cloud Storage and Amazon S3.
-
Install custom Python environment on Jupyter Notebooks at NERSC
NERSC has provided a JupyterHub instance for quite some time to all NERSC users. It is currently running on a dedicated large-memory node on Cori, so now it can access also data on Cori $SCRATCH, not only /project and $HOME. See their documentation
-
ECSS Symposium about Jupyterhub deployments on XSEDE
Note: XSEDE has been replaced by ACCESS. ECSS Symposium, 19 December 2017, Web presentation to the XSEDE Extended Collaborative Support Services.
-
Deploy scalable Jupyterhub with Kubernetes on Jetstream
The best infrastructure available to deploy Jupyterhub at scale is Kubernetes. Kubernetes provides a fault-tolerant system to deploy, manage and scale containers. The Jupyter team released a recipe to deploy Jupyterhub on top of Kubernetes, Zero to Jupyterhub.