hpc
44 posts — page 3
-
Deployment of Jupyterhub with Globus Auth to spawn notebook on Comet in Singularity containers
Follow the instructions at to build images from the ubuntuanacondajupyterhub.def and centosanacondajupyterhub.def definition files, or use the containers I have already built on Comet:
-
Deploy Jupyterhub on a supercomputer with SSH authentication
The best way to deploy Jupyterhub with an interface to a Supercomputer is through the use of batchspawner. I have a sample deployment explained in an older blog post:
-
Sample deployment of Jupyterhub in HPC on SDSC Comet
I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns single user Jupyter notebooks on Comet computing nodes using batchspawner and then proxies the Notebook back to the user using SSH-tunneling.
-
Run Ubuntu in HPC with Singularity
If your answer to any of those question is yes, read on! Otherwise, well, still read on, it's awesome! Singularity
-
IPython/Jupyter notebook setup on NERSC Edison
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Edison at NERSC and forward its port encrypted with SSH to the browser on a local laptop. This setup is a bit more complicated than other supercomputers, i.e. see my tutorial for Comet for 2 reasons:
-
IPython/Jupyter notebook setup on SDSC Comet
This tutorial explains the setup to run an IPython Notebook on a computing node on the supercomputer Comet at the San Diego Supercomputer Center and forward the port encrypted with SSH to the browser on a local laptop.
-
Run Jupyterhub on a supercomputer
The IPython (recently renamed Jupyter) Notebook is a powerful tool for analyzing and visualizing data in Python and other programming languages. A key feature is that a single document contains code, figures, text and equations. Everything is saved in a single .ipynb file that can be shared, executed and modified.
-
Thoughts on a career as a computational scientist
Recently I've been asked what are the prospects of a wannabe computational scientist, both in terms of training and in terms of job opportunities.
-
Machine learning at scale with Python
My talk for the San Diego Data Science meetup: http://www.meetup.com/San-Diego-Data-Science-R-Users-Group/events/170967362/
-
Python on Gordon
Gordon has already a python environment setup which can be activated by loading the python module: module load python add this to .bashrc to load it at every login
-
Run IPython Notebook on a HPC cluster via PBS
The IPython notebook is a great tool for data exploration and visualization. It is suitable in particular for analyzing a large amount of data remotely on a computing node of a HPC cluster and visualize it in a browser that runs on a local machine.
-
Processing sources in Planck maps with Hadoop and Python
Purpose The purpose of this post is to investigate how to process in parallel sources extracted from full sky maps, in this case the maps release by Planck, using Hadoop instead of more traditional MPI-based HPC custom software.
-
How to use the IPython notebook on a small computing cluster
The IPython notebook is a powerful and easy to use interface for using Python and particularly useful when running remotely, because it allows the interface to run locally in your browser, while the computing kernel runs remotely on the cluster.
-
IPython parallell setup on Carver at NERSC
IPython parallel is one of the easiest ways to spawn several Python sessions on a Supercomputing cluster and process jobs in parallel. On Carver, the basic setup is running a controller on the login node, and submit engines to the computing nodes via PBS.