Flatcar Container Linux on Jetstream


April 30, 2024

Virtual Machine images provided by the Jetstream team are fully fledged and therefore quite large.

Julien Chastang proposed the shift to a minimalist distribution to save disk space, resources and shorten deployment time, see the relevant issue discussion on the Jetstream Gitlab organization.

We are mostly interested in replacing Ubuntu as the default image for our Kubespray developments, so we are looking only at OS supported by Kubespray

Flatcar Container Linux

Best features:

  • Minimalist, image is only 450 MB
  • OS specifically for running containers
  • Automatic unsupervised updates

See their docs about deploying on Openstack, uploading to Jetstream worked out of the box using this script.

The name of the image is FlatcarContainerLinux-3815-2-2, it is set as a Community image:

openstack image list --community | grep Flatcar
| 3c4b5192-3dbc-4478-8340-7409da188669 | FlatcarContainerLinux-3815-2-2                   | active |


The instance boots correctly on Exosphere, I can ssh into the instance as exouser, however, on the Dashboard the image is stuck in the “Building” phase and a passphrase is not assigned to the user. Trying to run Docker commands fails for permission issues and without password cannot sudo.


The instance boots fine on Horizon as well, I can ssh with the user core using the keypair injected by Openstack. core by default can execute Docker commands, so can test with:

docker run --rm -p 80:80 -d nginx

and curl localhost.

Occupies minimal space:

df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           596M  556K  595M   1% /run
/dev/sda9        17G  215M   16G   2% /
sysext          1.5G   12K  1.5G   1% /usr
overlay          17G  215M   16G   2% /etc
/dev/sda6       128M  940K  123M   1% /oem
tmpfs           1.5G     0  1.5G   0% /media
tmpfs           1.5G     0  1.5G   0% /tmp
/dev/sda1       127M   59M   68M  47% /boot
tmpfs           298M     0  298M   0% /run/user/500

The update machinery executes without errors:

core@flatcarhorizon ~ $ update_engine_client -update
I0501 05:10:49.327685  1763 update_engine_client.cc:251] Initiating update check and install.
I0501 05:10:49.330926  1763 update_engine_client.cc:256] Waiting for update to complete.
I0501 05:10:54.688055  1763 update_engine_client.cc:194] No update available

Both in Horizon and Exosphere I get a warning in dmesg, this is fine:

[   54.197042] BTRFS warning: duplicate device /dev/sda3 devid 1 generation 33 scanned by (udev-worker) (1659)


The real objective of this test is to deploy Kubernetes on Jetstream using Flatcar, following the latest tutorial with minor modifications.

We can replace Ubuntu with Flatcar on Terraform by modifying in cluster.tfvars:

image = "FlatcarContainerLinux-3815-2-2"
ssh_user = "core"

Then in Ansible, following the Kubespray docs, we need to set:

bin_dir: /opt/bin

at the top of group_vars/all/all.yml.

Deployment time of 1 master node and 2 workers 21 minutes.

Detailed timing from Ansible:

kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS templates -------------------------------- 33.72s
kubernetes-apps/ingress_controller/ingress_nginx : NGINX Ingress Controller | Create manifests -------- 33.61s
kubernetes/kubeadm : Join to cluster ------------------------------------------------------------------ 27.87s
kubernetes-apps/csi_driver/cinder : Cinder CSI Driver | Generate Manifests ---------------------------- 23.79s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------- 22.23s
kubernetes-apps/ingress_controller/ingress_nginx : NGINX Ingress Controller | Apply manifests --------- 16.56s
kubernetes-apps/external_cloud_controller/openstack : External OpenStack Cloud Controller | Generate Manifests -- 13.77s
kubernetes-apps/csi_driver/cinder : Cinder CSI Driver | Apply Manifests ------------------------------- 12.62s
kubernetes/control-plane : kubeadm | Initialize first master ------------------------------------------ 10.90s
kubernetes-apps/csi_driver/csi_crd : CSI CRD | Generate Manifests ------------------------------------- 10.34s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down nodelocaldns Template ---------------------------- 10.23s
download : download_container | Download image if required --------------------------------------------- 9.86s
kubernetes/preinstall : Ensure kube-bench parameters are set ------------------------------------------- 9.70s
etcd : reload etcd ------------------------------------------------------------------------------------- 8.40s
kubernetes/preinstall : Create kubernetes directories -------------------------------------------------- 8.25s
etcd : Check certs | Register ca and etcd admin/member certs on etcd hosts ----------------------------- 8.14s
etcd : Check certs | Register ca and etcd admin/member certs on etcd hosts ----------------------------- 7.86s
kubernetes-apps/snapshots/snapshot-controller : Snapshot Controller | Generate Manifests --------------- 6.93s
network_plugin/flannel : Flannel | Create Flannel manifests -------------------------------------------- 6.91s
kubernetes/control-plane : Renew K8S control plane certificates monthly 1/2 ---------------------------- 6.77s

Nodes running happily with Flatcar:

k get nodes -o wide
NAME                       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP       OS-IMAGE                                             KERNEL-VERSION   CONTAINER-RUNTIME
kubejetstream-1            Ready    control-plane   35m   v1.25.6   149.xxx.xxx.xxx   Flatcar Container Linux by Kinvolk 3815.2.2 (Oklo)   6.1.85-flatcar   containerd://1.6.15
kubejetstream-k8s-node-1   Ready    <none>          33m   v1.25.6   149.xxx.xxx.xxx   Flatcar Container Linux by Kinvolk 3815.2.2 (Oklo)   6.1.85-flatcar   containerd://1.6.15
kubejetstream-k8s-node-2   Ready    <none>          33m   v1.25.6    149.xxx.xxx.xxx   Flatcar Container Linux by Kinvolk 3815.2.2 (Oklo)   6.1.85-flatcar   containerd://1.6.15

Also tested deploying successfully JupyterHub, enjoy!