Deploy nbgrader on Jetstream with a Manila exchange disk (Kubernetes)

kubernetes
jetstream
jupyterhub
nbgrader
Author

Andrea Zonca

Published

March 3, 2026

Download sourceContribute

This tutorial shows how to deploy nbgrader on Jetstream Kubernetes using a shared Manila disk mounted on all JupyterHub user pods.

This is for the setup where:

We will:

If you want the API-based approach instead of a shared disk, see the previous tutorial: ./2026-02-04-nbgrader-ngshare-jetstream.md

Difference in one line: this tutorial uses nbgrader’s native filesystem exchange on a Manila RWX share, while the previous tutorial uses ngshare (REST API exchange without shared filesystem).

Why this approach

nbgrader’s native exchange expects a shared filesystem visible from both instructor and student pods.
A Manila share mounted as ReadWriteMany provides exactly that.

Prerequisites

  • A running Magnum cluster named k8s.
  • JupyterHub deployed with Helm.
  • A Manila share already created (or created in Step 2 below).
  • OpenStack credentials file (*openrc*.sh) available.
  • A shell environment where required CLIs are installed and configured: openstack, kubectl, and helm.

Step 1: Check out this repository

This tutorial uses config files from this repo (manila/, nbgrader/, config_*.yaml, etc.), so clone it first:

git clone https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream.git
cd jupyterhub-deploy-kubernetes-jetstream

Step 2 (if needed): Create the Manila share

This tutorial uses an existing Jetstream Manila share named nbgraderexchange.

You can create it in Exosphere (recommended), or via OpenStack CLI.

Example CLI flow:

# First ensure your shell already has OpenStack credentials loaded.
# Also ensure openstack/kubectl/helm are available in your current environment.

# Create a 50 GiB CephFS share
openstack share create \
  --name nbgraderexchange \
  --share-type cephfsnativetype \
  CEPHFS \
  50

# Wait until available
while [ "$(openstack share show nbgraderexchange -f value -c status)" != "available" ]; do sleep 5; done

# Create a CephX RW access rule
openstack share access create \
  --access-level rw \
  nbgraderexchange cephx nbgraderexchange-rw

# Verify
openstack share list --name nbgraderexchange
openstack share access list nbgraderexchange

Note: in this workflow, Kubernetes mounts the Manila share through CephFS CSI, but does not create the Manila share itself.

Step 3: Configure access to the Magnum cluster

From the repo root:

# Ensure your current shell already has OpenStack credentials loaded
# and has openstack/kubectl/helm available.

export K8S_CLUSTER_NAME=k8s
bash kubernetes_magnum/configure_kubectl_locally.sh
export KUBECONFIG=$(pwd)/config
kubectl get nodes

You should see nodes from your k8s cluster.

Step 4: Prepare the existing Manila share as a Kubernetes RWX volume

This repo includes templates for Jetstream Manila CephFS:

  • manila/cephfs-csi-values.yaml
  • manila/cephfs-csi-pv.yaml
  • manila/cephfs-csi-pvc.yaml

Get mount info from the existing Exosphere share:

bash manila/generate_mount_command.sh nbgraderexchange

Use the output to fill placeholders in manila/cephfs-csi-values.yaml and manila/cephfs-csi-pv.yaml:

  • <CEPH_FSID> (any stable ID string is fine, but it must match in both files)
  • <ACCESS_RULE_NAME>
  • <ACCESS_KEY>
  • <SHARE_PATH> as path only (/volumes/...), not the full mon1,mon2:/volumes/... string

Install CephFS CSI (if not already installed) and create PV/PVC:

helm repo add ceph-csi https://ceph.github.io/csi-charts/ || true
helm repo update
helm upgrade --install ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --namespace kube-system \
  -f manila/cephfs-csi-values.yaml

kubectl create namespace jhub --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f manila/cephfs-csi-pv.yaml
kubectl apply -f manila/cephfs-csi-pvc.yaml
kubectl -n jhub get pvc manila-cephfs

Important for manila/cephfs-csi-pv.yaml:

  • Add fsName: "cephfs" under volumeAttributes.
  • Keep clusterID aligned with manila/cephfs-csi-values.yaml.
  • Ensure nodeStageSecretRef.namespace matches where csi-cephfs-secret exists.

If the chart created csi-cephfs-secret in kube-system but your PV references default, copy it once:

kubectl -n kube-system get secret csi-cephfs-secret -o yaml \
  | sed 's/namespace: kube-system/namespace: default/' \
  | kubectl apply -f -

Expected PVC status:

NAME            STATUS   VOLUME             CAPACITY   ACCESS MODES
manila-cephfs   Bound    manila-cephfs-pv   50Gi       RWX

Step 5: Mount the Manila share in all JupyterHub user pods

Use the values file already in this repo:

  • manila/jupyterhub_manila.yaml

It mounts the shared PVC at /share for every user pod.

Verify JupyterHub exists in jhub namespace:

helm list -n jhub

If no jhub release is present yet, deploy JupyterHub first, then continue.

Add this values file to install_jhub.sh (before the last line):

--values manila/jupyterhub_manila.yaml \

Re-deploy:

bash install_jhub.sh

Then stop and start existing user servers so they pick up the new mount.

Step 6: Install nbgrader and configure filesystem exchange

Use this file from the repo:

  • nbgrader/jhub-singleuser-nbgrader-filesystem.yaml

Replace COURSE_ID with your course ID (example: course101).

Add this file to install_jhub.sh:

--values nbgrader/jhub-singleuser-nbgrader-filesystem.yaml \

Re-deploy:

bash install_jhub.sh

Then stop and start existing user servers so the postStart hook re-runs and writes the updated nbgrader_config.py.

Step 7: Validate the shared mount and initialize exchange folders

Open a JupyterHub terminal as an instructor/admin user:

python -m pip show nbgrader
mkdir -p /share/nbgrader/exchange
mkdir -p /share/nbgrader/exchange/course101/inbound
mkdir -p /share/nbgrader/exchange/course101/outbound
mkdir -p /share/nbgrader/exchange/course101/feedback
chmod -R 0777 /share/nbgrader/exchange
ls -la /share/nbgrader/exchange/course101

If /share is mounted but writing fails with Permission denied, run a one-time permission bootstrap pod:

kubectl -n jhub apply -f nbgrader/manila-exchange-permissions-pod.yaml
kubectl -n jhub logs manila-exchange-permissions
kubectl -n jhub delete pod manila-exchange-permissions

Then open a different user server and confirm the same files are visible:

ls -la /share/nbgrader/exchange/course101

Step 8: Instructor workflow (create + release)

From this point, the nbgrader workflow is the same as in the previous tutorial:

Manila-specific difference:

  • You do not need ngshare-course-management.
  • Student roster is managed with nbgrader db student add ... on the instructor side.

Step 9: Student workflow (list + fetch + submit)

Use the same commands as linked in Step 8.

Step 10: Instructor workflow (collect + autograde)

Use the same commands as linked in Step 8.

Notes

  • This tutorial uses nbgrader’s filesystem exchange on a shared Manila disk.
  • No ngshare service or ngshare_exchange package is required.
  • The important part is that all user pods mount the same RWX path and permissions allow read/write for instructor and students.

Troubleshooting

If nbgrader fetch_assignment or submit fails with permission errors:

chmod -R 0777 /share/nbgrader/exchange

If user pods don’t see /share, confirm both values files were included in install_jhub.sh and run bash install_jhub.sh again.

If the Manila PVC is not Bound, re-check placeholders in:

  • manila/cephfs-csi-values.yaml
  • manila/cephfs-csi-pv.yaml

If pods are stuck in ContainerCreating with Ceph mount errors:

  • missing required field fsName: add fsName: "cephfs" in manila/cephfs-csi-pv.yaml.
  • mount error 3 = No such process: rootPath is wrong; use only /volumes/... path.
  • failed to find the secret csi-cephfs-secret in namespace default: align nodeStageSecretRef.namespace with the namespace where the secret exists.