jetstream
66 posts — page 4
-
Simulate users on JupyterHub
Updated January 2021 I currently have 2 different strategies to deploy JupyterHub on top of Kubernetes on Jetstream:
-
Deploy Cluster Autoscaler for Kubernetes on Jetstream
The Kubernetes Cluster Autoscaler is a service that runs within a Kubernetes cluster and when there are not enough resources to accomodate the pods that are queued to run, it contacts the API of the cloud provider to create more Virtual Machines to join the Kubernetes Cluster.
-
Deploy Kubernetes and JupyterHub on Jetstream with Magnum
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub.
-
Kubernetes monitoring with Prometheus and Grafana
See the updated version of this tutorial Updated September 2020
-
Kubernetes monitoring with Dashboard, Prometheus, and Grafana
Install with helm: https://github.com/helm/charts/tree/master/stable/prometheus-operator
-
Scale Kubernetes manually on Jetstream
We would like to modify the number of Openstack virtual machines available to Kubernetes. Ideally we would like to do this automatically based on the load on JupyterHub, that is the target. For now we will increase and decrease the size manually.
-
Deploy Kubernetes with Kubespray 2.8.2 and JupyterHub with helm recipe 0.8 on Jetstream
Note: Jetstream 1 has been retired. See Jetstream 2 documentation for current tutorials. Back in September 2018 I published a tutorial to deploy Kubernetes on Jetstream using Kubernetes.
-
Use the distributed file format Zarr on Jetstream Swift object storage, 2019
This is an updated version of the 2018 edition Zarr
-
Deploy Pangeo on Kubernetes deployment on Jetstream created with Kubespray
The Pangeo collaboration for Big Data Geoscience maintains a helm chart with a prefigured JupyterHub deployment on Kubernetes which also supports launching private dask workers.
-
Deploy JupyterHub on Kubernetes deployment on Jetstream created with Kubespray 3/3
All of the following assumes you are logged in to the master node of the Kubernetes cluster deployed with kubespray and checked out the repository:
-
Explore a Kubernetes deployment on Jetstream with Kubespray 2/3
This is the second part of the tutorial on deploying Kubernetes with kubespray and JupyterHub on Jetstream. In the first part, we installed Kubernetes on Jetstream with kubespray.
-
Deploy Kubernetes on Jetstream with Kubespray 1/3
Note: Jetstream 1 has been retired. For current Kubernetes deployments, see Jetstream 2 documentation. This tutorial is obsolete, check the updated version of the tutorial
-
PEARC18 paper on deploying Jupyterhub at scale on XSEDE
Bob Sinkovits and I are presenting a paper at PEARC18 about: "Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops"
-
Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to launch a set of dask workers as containers running inside Kubernetes itself and use them for distributed computing.
-
Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream
Let's assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a Jupyter Notebook, we would also like to provide them more computing power for their interactive data exploration.