High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Store a conda environment inside a Notebook
Last August, during the Container Analysis Environments Workshop held at Urbana-Champaign, we had discussion about reproducibility in the Jupyter Notebooks. There came out the idea of storing all the details about the Python environment inside the Notebook, in the metadata.
-
How to modify Singularity images on a Supercomputer
Singularity allows to run your own OS within most Supercomputers, see my previous post about Running Ubuntu on Comet via Singularity
-
Deploy scalable Jupyterhub on Docker Swarm mode
Jupyterhub genrally requires roughly 500MB per user for light data processing and many GB for heavy data processing, therefore it is often necessary to deploy it across multiple machines to support many users.
-
Setup automated testing on a Github repository with Travis-ci
Start from the CUDA 8 image from NVIDIA: FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04
-
Setup automated testing on a Github repository with Travis-ci
It is good practice in software development to implement extensive testing of the codebase in order to catch quickly any bug introduced into the code when implementing new features.
-
Deployment of Jupyterhub with Globus Auth to spawn notebook on Comet in Singularity containers
Follow the instructions at to build images from the ubuntuanacondajupyterhub.def and centosanacondajupyterhub.def definition files, or use the containers I have already built on Comet:
-
How to create pull requests on Github
Pull Requests are the web-based version of sending software patches via email to code maintainers. They allow a person that has no access to a code repository to submit a code change to the repository administrator for review and 1-click merging.
-
How to create pull requests on Github
Pull Requests are the web-based version of sending software patches via email to code maintainers. In more detail:
-
Deploy Jupyterhub on a supercomputer with SSH authentication
The best way to deploy Jupyterhub with an interface to a Supercomputer is through the use of batchspawner. I have a sample deployment explained in an older blog post:
-
Configure Globus on your local machine for GridFTP with XSEDE authentication
All the commands are executed on your local machine, the purpose of this tutorial is to be able to use globus-url-copy to copy efficiently data back and forth between your local machine and a XSEDE Supercomputer on the command line.
-
Sample deployment of Jupyterhub in HPC on SDSC Comet
I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns single user Jupyter notebooks on Comet computing nodes using batchspawner and then proxies the Notebook back to the user using SSH-tunneling.
-
Customize your Python environment in Jupyterhub
Usecase: You have access to a Jupyterhub server and you would like to install some packages but cannot use pip install and modify the systemwide Python installation.
-
Automated deployment of Jupyterhub with Ansible
Last year I wrote some tutorials on simple deployments of Jupyterhub on Ubuntu 16.04 on the OpenStack deployment SDSC Cloud, even if most of the steps would also be suitable on other resources like Amazon EC2.
-
How to publish your research software to Github
Nowadays many journals require that the software used to produce results described in a scientific paper be made available publicly for other peers to be able to reproduce the results or even just explore the analysis more in detail.
-
Run Ubuntu in HPC with Singularity
If your answer to any of those question is yes, read on! Otherwise, well, still read on, it's awesome! Singularity