High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Allow users to self-invite to a Google Calendar event
Google Calendar is the most popular calendar service, and it is widely used in academia, often Scientific Collaborations maintain one or multiple shared calendars to keep track of events, deadlines, and meetings.
-
Deploy an LLM ChatGPT-like service on Jetstream
In this tutorial we will deploy a LLM Chat-GPT like service on a GPU node on Jetstream. However the same instructions can be used to deploy any other model available on the Hugging Face model hub.
-
Kubernetes monitoring with Prometheus and Grafana
In a production Kubernetes deployment it is necessary to make it easier to monitor the status of the cluster effectively. Kubernetes provides Prometheus to gather data from the different components of Kubernetes and Grafana to access those data and provide real-time plotting and inspection capability.
-
Generate script to mount a Manila share
Jetstream 2 makes creating volumes that are shared across instances as easy as creating standard volumes. This is provided by the Manila service, which creates volumes based on the Ceph file system that can be mounted simultaneously on multiple Jetstream 2 Virtual Machines.
-
Configure ssh-agent on Linux
I keep Googling this again and again. First, ssh-agent does not need to run on the remote server, it is the local agent which is forwarded over ssh to the remote server.
-
Tutorial on Python for HPC at the SDSC Summer Institute 2024
Each August the San Diego Supercomputer Center organizes a week-long summer school that teaches a large number of topics related to Supercomputing and AI to early career scientists.
-
How to use Anki flashcards to help kids learn spelling
Now the kid can study the spelling words using the Anki desktop app or the AnkiWeb website or the AnkiDroid Android or IOS app (the IOS app is paid, all other options are free).
-
Generate sample Cosmology data for Cheap and FAIR data portal with PySM
This notebook generates some figures of Galactic and Extra-Galactic emissions using PySM. Mostly for displaying purposes.
-
Cosmic Microwave Background data licensing
Some thoughts on how to license and attribute data products from the Simons Observatory. Published data products, like papers and software, should have a license, this is for example required by the FAIR Principles.
-
Export a Confluence page to Markdown
I feel the pain, you have a page trapped in Confluence and you want to set it free. Of course the "Export" features of Confluence are useless.
-
Ubuntu 22.04 minimal on Jetstream
Update July 2024: Ubuntu 22.04 Minimal image with Generic Kernel Virtual Machine images provided by the Jetstream team are fully fledged and therefore quite large.
-
PEARC24 paper Cheap and FAIR Data Portal
In this blog post, we summarize the key points from our recent paper presented at PEARC 2024, titled "Cheap and FAIR: A Serverless Research Data Repository for the Next Generation Cosmic Microwave Background Experiment."
-
2024-07-29-pysm-point-source-pixell-healpix
Testing the pixell simobjects functionality to create maps of point sources pre-smoothed with a gaussian beam. The purpose is to include this functionality in PySM to be able to generate on the fly maps of source starting from a catalog.
-
Run (part of) healpy in the browser with pyodide
Thanks to the help of @VeerioSDSC and Rick Wagner, I have been able to run part of healpy in the browser. For now only tested reading FITS maps and plotting with projview.
-
Conditional authentication on a single endpoint with FastAPI
Building on my previous tutorial, I have now implemented a toy FastAPI application which requires authentication based on query parameters.