Deploy ngshare on Jetstream with a Manila volume (Kubernetes)

kubernetes
jetstream
jupyterhub
nbgrader
Author

Andrea Zonca

Published

September 15, 2026

Download sourceContribute

This tutorial shows how to deploy nbgrader with ngshare on Jetstream Kubernetes where ngshare’s SQLite database lives on a Manila CephFS share instead of a Cinder volume.

It is the storage-backend update of the original ngshare tutorial: Deploy nbgrader on Jetstream with ngshare (Kubernetes).

Why use Manila for ngshare

ngshare keeps its SQLite database on a ReadWriteMany PVC (in case of multiple replicas or shared access). The default Cinder CSI driver on Jetstream registers fsGroupPolicy: ReadWriteOnceWithFSType, so per the CSI spec fsGroup is applied only to ReadWriteOnce claims; RWX volumes are mounted with root ownership, and ngshare fails with:

sqlite3.OperationalError: unable to open database file

The previous workaround was a root initContainer fixing ownership and mode (see the old tutorial’s Troubleshooting section).

Manila CephFS volumes mounted through the CephFS CSI driver register fsGroupPolicy: File, so fsGroup is honored also for RWX mounts. ngshare then works out of the box, without initContainers. I verified this on 2026-09-15 on a fresh Jetstream2 cluster: the ngshare pod starts, runs its alembic migrations and creates the SQLite DB with fsGroup 1000 applied. All manifests below live in the jupyterhub-deploy-kubernetes-jetstream deployment repository, and the reusable test case is at nbgrader/ngshare-standalone-repro-manila/.

Prerequisites

  • A running Magnum cluster on Jetstream (see Deploy Kubernetes on Jetstream2 with Magnum and Cluster API).
  • OpenStack credentials loaded, plus openstack, jq, kubectl, and helm.
  • The deployment repository cloned locally: git clone https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream.git (all commands below run from its root).

Step 1: Create a Manila share

Create a 10 GiB CEPHFS share and a cephx access rule:

openstack share create --name ngshare --share-type cephfsnativetype CEPHFS 10
while [ "$(openstack share show ngshare -f value -c status)" != "available" ]; do sleep 5; done
openstack share access create --access-level rw ngshare cephx ngshare-rw

Extract the mount details (monitors + path, access key):

openstack share show ngshare -f json | jq -r '.export_locations[0].path'
openstack share access list ngshare -f value -c "Access To" -c "Access Key"

The path looks like:

149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/<share-id>/<path>

The Ceph monitors are the comma-separated host:port entries up to the colon that immediately precedes /volumes/...; the share path is everything from /volumes/... onward (use only that part in the PV).

Step 2: Configure access to the cluster

From the deployment repository root, with OpenStack credentials loaded:

export K8S_CLUSTER_NAME=k8s
bash kubernetes_magnum/configure_kubectl_locally.sh
export KUBECONFIG=$(pwd)/config
kubectl get nodes

Step 3: Install the CephFS CSI driver

Copy the values template to a local, gitignored file (never edit the tracked template, it holds the share access key):

cp manila/cephfs-csi-values.yaml manila/cephfs-csi-values.local.yaml

Fill manila/cephfs-csi-values.local.yaml with:

  • <CEPH_FSID>: any stable ID string, e.g. manila-cephfs (it only needs to match the PV, see the next step)
  • <ACCESS_RULE_NAME>: the access rule name (ngshare-rw, no client. prefix)
  • <ACCESS_KEY>: the access key from Step 1
  • the monitors list from the export location

Then install in kube-system (the nodeStageSecretRef of the PV below points there):

helm repo add ceph-csi https://ceph.github.io/csi-charts/
helm repo update
helm upgrade --install ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --namespace kube-system \
  -f manila/cephfs-csi-values.local.yaml

Verify the driver is registered and pods run:

kubectl get csidriver cephfs.csi.ceph.com -o jsonpath='{.spec.fsGroupPolicy}'
kubectl get csidriver cephfs.csi.ceph.com
kubectl -n kube-system get pods | grep ceph-csi-cephfs

Expected fsGroupPolicy: File.

Step 4: Create the StorageClass, PV and PVC

This repo ships templates for the ngshare Manila PV:

  • nbgrader/ngshare-standalone-repro-manila/00-storageclass.yaml: manila-cephfs StorageClass with provisioner: kubernetes.io/no-provisioner
  • nbgrader/ngshare-manila-pv.yaml: static PV (fill <SHARE_ID> and <SHARE_PATH>)

Fill ngshare-manila-pv.yaml:

  • <SHARE_ID>: the Manila share UUID (any unique string is fine as volumeHandle)
  • <SHARE_PATH>: /volumes/_nogroup/... from Step 1
  • clusterID must equal the <CEPH_FSID> used in Step 3

Apply the StorageClass and PV (do not create a PVC here: the ngshare Helm chart creates its own PVC in Step 5, and a static PV can be bound only once):

kubectl create namespace jhub --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f nbgrader/ngshare-standalone-repro-manila/00-storageclass.yaml
kubectl apply -f nbgrader/ngshare-manila-pv.yaml
kubectl get pv ngshare-manila-pv

Expected: PV Available, access modes RWX.

Why a no-provisioner StorageClass: the ngshare Helm chart creates its own PVC and we bind it statically with a selector. A StorageClass with a real provisioner would make the controller attempt dynamic provisioning, and the CephFS provisioner rejects PVC selectors with claim Selector is not supported.

Step 5: Install ngshare

Add the ngshare Helm repository and use the values file nbgrader/ngshare-manila-config.yaml: it points the chart PVC at the static PV via pvc.selector, uses storageClassName: manila-cephfs, and contains no initContainer.

Edit it (token, admins) and install:

helm repo add ngshare https://libretexts.github.io/ngshare-helm-repo/
helm repo update
helm install ngshare ngshare/ngshare \
  --namespace jhub \
  -f nbgrader/ngshare-manila-config.yaml

Verify the PVC bound and the pod running:

kubectl -n jhub get pvc ngshare-pvc
kubectl -n jhub get pods -l app.kubernetes.io/instance=ngshare
kubectl -n jhub logs deploy/ngshare --tail=20

Expected: pod 1/1 Running, log shows alembic migrations (Context impl SQLiteImpl, Running upgrade -> aa00db20c10a, Init) and no permission errors. The SQLite database is created on the Manila volume with fsGroup ownership:

kubectl -n jhub exec deploy/ngshare -- sh -c 'id; ls -ld /srv/ngshare; ls -l /srv/ngshare'

Step 6: Register ngshare in JupyterHub and enable nbgrader

Same as the original tutorial:

  1. Add the ngshare service snippet from nbgrader/jhub-ngshare-service.yaml to your JupyterHub values and redeploy.
  2. Add nbgrader/jhub-singleuser-nbgrader.yaml so user pods have nbgrader and ngshare_exchange.

See steps 2 to 4 of Deploy nbgrader on Jetstream with ngshare (Kubernetes).

Notes

  • No initContainer is required: the Manila CephFS volume honors fsGroup on RWX claims.
  • The ngshare PVC is standalone metadata storage; 10 GiB is plenty for typical classes.
  • The nbgrader/ngshare-standalone-repro-manila/ folder contains a standalone repro (same ngshare runtime conditions) if you want to verify a fresh setup before deploying. It uses its own PV (ngshare-repro-pv) and its own Manila share or subdirectory: never point it at the same share/path as the deployed ngshare volume, because its with-init variant recursively chowns the mount.

Troubleshooting

PVC stuck Pending, event claim Selector is not supported: the StorageClass has a real provisioner. Recreate it with provisioner: kubernetes.io/no-provisioner as in 00-storageclass.yaml.

PVC Pending after the PV was previously bound and released: clear the claimRef on the PV:

kubectl patch pv ngshare-manila-pv --type=merge -p '{"spec":{"claimRef":null}}'

Pod stuck ContainerCreating with Ceph mount errors: check the secret referenced by nodeStageSecretRef exists in kube-system, and that rootPath is only the /volumes/... part of the export location.

Old sqlite error still appears: make sure the ngshare pod uses the Manila PVC and that the volume directory is not a leftover Cinder mount. Delete the ngshare pod so it re-creates the DB on the new volume:

kubectl -n jhub delete pod -l app.kubernetes.io/instance=ngshare