dask
11 posts
-
Deploy the Dask Operator for Kubernetes on Jetstream2 and access it from JupyterHub
This post describes how to deploy the Dask Operator for Kubernetes alongside a Helm-based JupyterHub installation on a Jetstream2 Kubernetes cluster (Magnum). The Operator provides a Kubernetes-native way to create and manage Dask clusters via custom resources, simplifying multi-tenant setups.
-
Deploy Dask Gateway with JupyterHub on Kubernetes
In this tutorial we will install Dask Gateway, currently version 2023.9.0, on Kubernetes and configure JupyterHub so Jupyter Notebook users can launch private Dask cluster and connect to them.
-
Science Gateway with Dask and Zarr
This material was presented on April 2022 at the MiniGateways 2022 conference organized by the wonderful Science Gateways Community Institute (SGCI).
-
Use the distributed file format Zarr on Jetstream 2 object storage
Zarr is a file format designed for cloud computing, see documentation. Zarr is also supported by dask, the parallel computing framework for Python, and the Dask team implemented storage backends for Google Cloud Storage and Amazon S3.
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial obsolete, see the new version of the tutorial Updated 28 April 2022: switched to Dask Gateway 2022.4.0
-
Deploy Dask Gateway with JupyterHub on Kubernetes
Tutorial OBSOLETE Please check the updated version of this tutorial. This tutorial follows the work by the Pangeo collaboration, the main difference is that I prefer to keep JupyterHub and the Dask infrastructure in 2 separate Helm recipes.
-
Deploy Dask Gateway with JupyterHub on Kubernetes
This tutorial is obsolete, please follow This tutorial follows the work by the Pangeo collaboration, the main difference is that I prefer to keep JupyterHub and the Dask infrastructure in 2 separate Helm recipes.
-
Dask array rounding
Dask array rounding float32
-
Use the distributed file format Zarr on Jetstream Swift object storage, 2019
This is an updated version of the 2018 edition Zarr
-
Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to launch a set of dask workers as containers running inside Kubernetes itself and use them for distributed computing.
-
Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream
Let's assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a Jupyter Notebook, we would also like to provide them more computing power for their interactive data exploration.