High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Auto-merge GitHub pull requests after GitHub Actions pass
Pull request auto-merge is a small feature that keeps teams from babysitting green builds. Once every required check is green, GitHub merges the PR for you—no extra clicks, no late-night merges. Here is how to get it running safely with GitHub Actions.
-
Manage calendar of a scientific collaboration using Google Groups
Managing a calendar for a large scientific collaboration can be challenging. Over the years, I have tried different approaches to solve this problem.
-
EZID DOI management with Python
This project offers Python scripts for interacting with the EZID API to create and verify Digital Object Identifiers (DOIs).
-
Building Singularity containers with Conda environments from requirements.txt
This post introduces a method for building Singularity containers that include a Conda environment based on a requirements.txt file. This approach provides a reproducible and portable environment for any Python project.
-
Running Nextflow on Expanse
This tutorial will guide you through setting up Nextflow on Expanse. Nextflow is a powerful and flexible workflow management system that enables scalable and reproducible scientific workflows.
-
PySM 3.4.3 Released
PySM version 3.4.3 is now live. Install it from PyPI or upgrade existing environments with pip install --upgrade pysm3. Full details are available in the GitHub release notes.
-
Deploy a ChatGPT-like LLM on Jetstream with llama.cpp
Tutorial last updated in December 2025 This is a crosspost of the official Jetstream documentation: Deploy a ChatGPT-like LLM service on Jetstream. I built a brand new version of that tutorial that swaps in llama.cpp for vLLM so we can run GGUF quantized models on Jetstream's GPUs without giving up speed or context length.
-
PySM methods paper published in ApJ
The Astrophysical Journal formally published our paper, Full-sky Models of Galactic Microwave Emission and Polarization at Sub-arcminute Scales for the Python Sky Model, on September 12, 2025. I wanted to capture the news here now that the issue is out, following up on my March note about the ArXiv submission.
-
Inspect SMTP sending through UCSD
When configuring email sending through UCSD, it can be surprisingly unclear which SMTP server to use. The official documentation mentions three options: smtp.ucsd.edu, smtp.gmail.com, and smtp.office365.com.
-
OpenStack Unshelver Demo
I recently vibe-coded a lightweight web application, using gpt-5-codex, that revives shelved OpenStack instances on demand. The stack is intentionally minimal: a FastHTML frontend, GitHub for authentication, and the OpenStack SDK orchestrated through a YAML configuration that lists the instances the team cares about.
-
Deploy a 70B LLM to Jetstream
Deploying large language models on Jetstream is getting easier thanks to the official Jetstream LLM guide. Here I follow that walkthrough but scale the hardware and model so we can run something far more capable than the defaults.
-
Timing the unshelving of a Jetstream 70B LLM instance
Following the work documented in Deploy a 70B LLM to Jetstream, the Meta-Llama-3.1-70B-Instruct-GGUF deployment is now running on a g3.xl instance. The goal of this follow-up is to measure how long it takes to unshelve that virtual machine and bring the chat interface back online.
-
Python for HPC
Python is often the first choice for prototyping research ideas, but scaling that prototype to thousands of cores and multi‑node workflows needs a different toolkit.
-
Execute Pegasus jobs on Expanse
Pegasus is a workflow management system that helps scientists and engineers execute complex computational workflows. Pegasus supports different data staging mechanisms, primarily sharedfs and condorio. The sharedfs mode is used when the head node and all worker nodes share a common file system, allowing jobs to directly access data. However, there is currently a bug that prevents sharedfs from working as expected. In contrast, the condorio mode is designed for environments where worker nodes do not share a file system, relying on HTCondor's built-in file transfer capabilities for all data I/O. Since Pegasus 5.0, condorio is the default. Our current setup on Expanse, utilizing HTCondor Annex, operates in condorio mode, leveraging Condor's efficient data transfer for distributed execution.
-
Use VS Code on Expanse
Using VS Code directly on Expanse, or other HPC systems, is generally not recommended due to the high resource usage of the IDE itself. These systems are optimized for computational tasks, not for running graphical applications on login nodes.