High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Configure Globus on your local machine for GridFTP with XSEDE authentication
All the commands are executed on your local machine, the purpose of this tutorial is to be able to use globus-url-copy to copy efficiently data back and forth between your local machine and a XSEDE Supercomputer on the command line.
-
Sample deployment of Jupyterhub in HPC on SDSC Comet
I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns single user Jupyter notebooks on Comet computing nodes using batchspawner and then proxies the Notebook back to the user using SSH-tunneling.
-
Customize your Python environment in Jupyterhub
Usecase: You have access to a Jupyterhub server and you would like to install some packages but cannot use pip install and modify the systemwide Python installation.
-
Automated deployment of Jupyterhub with Ansible
Last year I wrote some tutorials on simple deployments of Jupyterhub on Ubuntu 16.04 on the OpenStack deployment SDSC Cloud, even if most of the steps would also be suitable on other resources like Amazon EC2.
-
How to publish your research software to Github
Nowadays many journals require that the software used to produce results described in a scientific paper be made available publicly for other peers to be able to reproduce the results or even just explore the analysis more in detail.
-
Run Ubuntu in HPC with Singularity
If your answer to any of those question is yes, read on! Otherwise, well, still read on, it's awesome! Singularity
-
Jupyterhub Docker Spawner with GPU support
Docker Spawner allows users of Jupyterhub to run Jupyter Notebook inside isolated Docker Containers. Access to the host NVIDIA GPU was not allowed until NVIDIA release the NVIDIA-docker plugin.
-
Jupyterhub deployment on multiple nodes with Docker Swarm
This post is part of a series on deploying Jupyterhub on OpenStack tailored at workshops, in the previous posts I showed:
-
Quick Jupyterhub deployment for workshops with pre-built image
This tutorial explains how to use a OpenStack image I already built to quickly deploy a Jupyterhub Virtual Machine that can provide a good initial setup for a workshop, providing students access to Python 2/3, Julia, R, file editor and terminal with bash.
-
Deploy Jupyterhub on a virtual machine for a workshop
This tutorial describes the steps to install a Jupyterhub instance on a single machine suitable for hosting a workshop, suitable for having people login with training accounts on Jupyter Notebooks running Python 2/3, R, Julia with also Terminal access on Docker containers. Details about the setup:
-
Use your own Python installation (kernel) in Jupyterhub
Updated February 2017 You have access to a Jupyterhub server but the Python installation provided does not satisfy your needs, how to use your own?
-
IPython/Jupyter notebook setup on NERSC Edison
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Edison at NERSC and forward its port encrypted with SSH to the browser on a local laptop. This setup is a bit more complicated than other supercomputers, i.e. see my tutorial for Comet for 2 reasons:
-
IPython/Jupyter notebook setup on SDSC Comet
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Comet at the San Diego Supercomputer Center and forward the port encrypted with SSH to the browser on a local laptop.
-
Run Jupyterhub on a supercomputer
The IPython (recently renamed Jupyter) Notebook is a powerful tool for analyzing and visualizing data in Python and other programming languages. A key feature is that a single document contains code, figures, text and equations. Everything is saved in a single .ipynb file that can be shared, executed and modified.
-
Accelerate groupby operation on pixels with Numba
Download the original IPython notebook Astrophysics background