hpc
44 posts — page 2
-
Run PyTorch with GPU support on Expanse
In this tutorial we will create a Python environment with PyTorch and see how to get Jupyterlab running on a GPU node. First of all we want to create an isolated Python environment, I generally favor micromamba, see the documentation on how to install it
-
Tutorial on Python for HPC at the SDSC Summer Institute
Each August the San Diego Supercomputer Center organizes a week-long summer school that teaches a large number of topics related to Supercomputing and AI to early career scientists.
-
Hiring a computational scientist at the San Diego Supercomputer Center
Come work in my group of High Performance Computing "consultants" at the San Diego Supercomputer Center, help scientists develop, scale and run Data Intensive computations on our Supercomputers.
-
Singularity on Expanse tutorial
As one of my last tasks in XSEDE, I updated and improved the tutorial about running Singularity containers in the HPC system Espanse at my institution the San Diego Supercomputer Center.
-
Manage Globus groups with the Python SDK
Globus is the best tool to transfer quickly Terabytes data between Supercomputers because it automatically parallelizes the transfer to saturate the network (yeah I know I always simplify too much).
-
Install HEALPix and PolSpice in a conda environment
Some notes on how to install HEALPix and PolSpice inside a conda environment, with some details about doing it at NERSC, but most of the tutorial is independent of that.
-
Webinar about distributed computing with Python
Recording available of the webinar I gave about "Distributed computing with Python": Threads vs Processes, GIL
-
Inherit group permission in folder
I have googled this so many times... On shared systems, like Supercomputers, you often belong to many different Unix groups, and that membership allows you to access data from specific projects you are working on and you can share data with your collaborators.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a different strategy this year, in particular relying on the littlest JupyterHub project for the initial deployment.
-
Bring your computing to the San Diego Supercomputer Center
Note (2026 update): This post was written in 2018. Comet has been retired and XSEDE has been replaced by ACCESS. SDSC's current flagship system is Expanse. For updated information about the HPC@UC program for UC researchers, see my 2026 post on requesting an HPC@UC allocation on Expanse.
-
Updated Singularity images for Comet
Back in January 2017 I wrote a blog post about running Singularity on Comet. I recently needed to update all my container images to the latest scientific python packages, so I also took the opportunity to create both a Docker auto-build repository on DockerHub and a SingularityHub image.
-
How to organize code and data for simulations at NERSC
I recently improved my strategy for organizing code and data for simulations run at NERSC, I'll write it here for reference.
-
Install custom Python environment on Jupyter Notebooks at NERSC
NERSC has provided a JupyterHub instance for quite some time to all NERSC users. It is currently running on a dedicated large-memory node on Cori, so now it can access also data on Cori $SCRATCH, not only /project and $HOME. See their documentation
-
How to modify Singularity images on a Supercomputer
Singularity allows to run your own OS within most Supercomputers, see my previous post about Running Ubuntu on Comet via Singularity