Deploy the Dask Operator for Kubernetes on Jetstream2 and access it from JupyterHub

kubernetes
jetstream2
jupyterhub
dask
python
Author

Andrea Zonca

Published

August 19, 2026

Download sourceContribute

This post describes how to deploy the Dask Operator for Kubernetes alongside a Helm-based JupyterHub installation on a Jetstream2 Kubernetes cluster (Magnum). The Operator provides a Kubernetes-native way to create and manage Dask clusters via custom resources, simplifying multi-tenant setups. Compared to Dask Gateway, it is more integrated into the Kubernetes ecosystem.

These commands target a Jetstream 2 Kubernetes deployment and assume you already have a running JupyterHub installed with the JupyterHub tutorial. The upstream install docs are at https://kubernetes.dask.org/en/latest/installing.html; this is the condensed version aligned with the JupyterHub setup on Jetstream2.

Versions tested

This tutorial was tested (2026-08-19) against:

  • Kubernetes 1.33.2 (Magnum template kubernetes-1-33-jammy-fixed-labels)
  • JupyterHub Helm chart 4.3.3
  • dask-kubernetes Operator & Python client 2026.3.0
  • dask / distributed 2026.7.1
  • jupyter-server-proxy 4.5.0

1. Install the Dask Operator

Install the Operator (CRDs + controller) into its own namespace:

helm install \
  --repo https://helm.dask.org \
  --create-namespace \
  -n dask-operator \
  --generate-name \
  dask-kubernetes-operator

Helm prints the release name (for example, dask-kubernetes-operator-1787187822). Keep it if you plan to upgrade later instead of reinstalling.

2. Validate the controller

Confirm the controller pod comes up cleanly:

kubectl get pods -n dask-operator

If you do not see the pod, verify the CRDs are present:

kubectl get crds | grep dask

You should see daskclusters, daskworkergroups, daskjobs, and daskautoscalers:

daskautoscalers.kubernetes.dask.org
daskclusters.kubernetes.dask.org
daskjobs.kubernetes.dask.org
daskworkergroups.kubernetes.dask.org

Important: apply the JupyterHub RBAC (next section) before starting a user notebook server, otherwise spawning fails with a serviceaccount "dask-sa" not found error.

3. Prepare a service account for JupyterHub

git clone https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream.git
cd jupyterhub-deploy-kubernetes-jetstream
cd dask_operator
kubectl apply -f dask_cluster_role.yaml

This creates a dask-sa ServiceAccount in the jhub namespace, a dask-cluster-role ClusterRole, and a binding so that user pods can create and manage Dask custom resources.

4. Configure JupyterHub

Apply it via the existing Helm deployment script. First edit install_jhub.sh in the deployment repository and add the extra --values dask_operator/jupyterhub_config.yaml flag to the helm upgrade --install line (alongside any other --values files you already use). Also add --values dask_operator/jupyterhub_dask_dashboard_config.yaml to enable the Dask dashboard proxy. Then run:

bash install_jhub.sh

5. Create a Dask cluster from JupyterHub

From JupyterHub, start a new notebook server and open a Python notebook.

In the first cell, install dask-kubernetes (this ensures the Operator Python client is available inside the user environment):

%pip install dask-kubernetes

If you want to avoid running %pip install dask-kubernetes every time a pod restarts, build your JupyterHub spawn image with dask-kubernetes already included so the dependency is available as soon as the notebook starts (see the recommended image below).

In the next cell, create a Dask cluster using the Operator. The namespace must match the JupyterHub namespace so the spawned pods are visible and accessible to the user environment:

from dask_kubernetes.operator import KubeCluster

cluster = KubeCluster(
  name="my-dask-cluster",
  namespace="jhub",                 # important: JupyterHub namespace
  image="ghcr.io/dask/dask:2026.7.1",  # pin a version matching your client
)
cluster.scale(10)

The scheduler/worker image should be the official Dask image ghcr.io/dask/dask and its version should be close to the distributed version in your user environment to avoid a VersionMismatchWarning. Mixing very different versions leads to client/scheduler/worker incompatibilities that are hard to debug.

Tip: If your user image is built on a different Python minor version than the Dask image (e.g., the scipy-notebook image uses Python 3.13 while ghcr.io/dask/dask ships Python 3.10), you may see a benign VersionMismatchWarning listing only the python and tornado rows. As long as dask and distributed match, this is harmless and does not affect correctness.

Finally, display the cluster object to get a realtime view on the number of workers:

cluster

You can see it scale with:

kubectl -n jhub get pods -l dask.org/component

6. Specify worker CPU and memory

Pass a resources dict directly to KubeCluster. This sets requests/limits for both the scheduler and the default worker group. Then scale.

from dask_kubernetes.operator import KubeCluster

cluster = KubeCluster(
  name="my-dask-cluster-resources",
  namespace="jhub",
  resources={
    "requests": {"cpu": "2", "memory": "2Gi"},
    "limits": {"cpu": "2", "memory": "2Gi"},
  },
  n_workers=0,  # start with zero, then scale explicitly
)
cluster.scale(5)  # create 5 workers with 2 CPU / 2Gi each
cluster

Hint: If cluster shows the scheduler as Pending, the worker pods are waiting for nodes. On a fresh cluster the autoscaler needs a few minutes to add them. Check with kubectl -n jhub get pods -l dask.org/cluster-name=<CLUSTER_NAME> — once the scheduler is Running, continue.

Different sizes? Add another worker group with its own resources:

cluster.add_worker_group(
  name="highmem",
  n_workers=2,
  resources={"requests": {"memory": "8Gi"}, "limits": {"memory": "8Gi"}},
)
cluster.scale(4, worker_group="highmem")  # scale that group

Use cluster.scale(N, worker_group="groupname") to change replicas for a specific group later.

Notice that if you have the Kubernetes Autoscaler enabled in your cluster (as in the Kubespray tutorial), it will automatically scale the number of nodes to accommodate the requested resources.

When you are done with the cluster, shut it down cleanly:

cluster.close()

7. Run distributed work and connect with the Client

from distributed import Client
import dask.array as da

client = Client(cluster)

x = da.random.random((10000, 10000), chunks=(1000, 1000))
x.sum().compute()

After you are done, close both the client and the cluster:

client.close()
cluster.close()

8. Access the Dask dashboard

One of the most important features of Dask is its dashboard, which provides real-time insight into distributed calculations. To visualize the Dask dashboard from within JupyterHub, we need to use jupyter-server-proxy.

A critical point is that jupyter-server-proxy must be baked into the single-user Docker image that JupyterHub spawns. This is because the proxy needs to be available when the user’s pod starts. Installing it with %pip install inside a running notebook will not work, as the server-proxy components are not loaded dynamically.

Ensure install_jhub.sh also passes --values dask_operator/jupyterhub_dask_dashboard_config.yaml to the helm upgrade --install command so JupyterHub is configured to expose the Dask dashboard through the proxy.

I maintain an example single-user image derived from Jupyter’s scipy-notebook that includes jupyter-server-proxy, dask, and dask-kubernetes at current versions. The repository is github.com/zonca/jupyterhub-dask-docker-image, and the image is automatically built with GitHub Actions and hosted on the GitHub Container Registry.

The list of available image tags is at github.com/zonca/jupyterhub-dask-docker-image/pkgs/container/jupyterhub-dask-docker-image.

If you want the simplest maintenance flow for your own image, start from the custom template and just edit requirements.txt; a GitHub Actions workflow will rebuild and publish automatically.

To use one of these images in your JupyterHub deployment, update your config_standard_storage.yaml file for the JupyterHub Helm chart:

singleuser:
  image:
    name: ghcr.io/zonca/jupyterhub-dask-docker-image
    tag: "latest"

Pin the tag for reproducibility — replace latest with the dated tag (e.g. 2026-08-19-<sha>) listed in the registry after you verify it works on your deployment.

After updating your JupyterHub configuration to use a compatible image, when you create a KubeCluster and print the cluster object in your notebook, it will include a clickable link to the Dask dashboard. The link has this format:

/user/YOURUSER/proxy/my-dask-cluster-scheduler.jhub:8787/status

This lets you access the dashboard and monitor your Dask cluster’s activity in real time.