High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
How to log exceptions in Python
Sometimes it is useful to just catch any exception, write details to a log file and continue execution. In the Python standard library, it is possible to use the logging and exceptions modules to achieve this. First of all, we want to catch any exception, but also being able to access all information about it:
-
Google Plus comments plugin for Pelican
There has been recently several discussions about whether comments are any useful on blogs I think it is important to find better ways to connect blogs to social networks. In my opinion the most suitable social network for this is Google+, because there is space for larger discussion, without Twitter's character limit.
-
How to automatically build your Pelican blog and publish it to Github Pages
Something I like a lot about Jekyll, the Github static blog generator, is that you just push commits to your repository and Github takes care of re-building and publishing your website.
-
clviewer, interactive plot of CMB spectra
Today it was HackDay at .Astronomy, so I felt compelled to hack something around myself, creating something I have been thinking for a while after my previous work on Interactive CMB power spectra in the browser
-
Planck CMB map at high resolution
Prompted by a colleague, I created a high-resolution version of the Cosmic Microwave Background map in MollWeide projection released by the Planck collaboration, available on the Planck Data Release Website in FITS format.
-
Run Hadoop Python jobs on Amazon with mrjob
First we need to install mrjob with: pip install mrjob I am starting with a simple example of word counting. MrJob makes the implementation of the reducer particularly simple. Using hadoop-streaming directly, we needed also to first parse back the output of the mapper into python objects, while MrJob does it for you and gives directly the key and the list of count, that we just need to sum.
-
Interactive figures in the browser - CMB power spectra
For a long time I've been curious about trying out d3.js , the javascript plotting library which is becoming the standard for interactive plotting in the browser.
-
Planck CTP angular power spectrum ell binning
Planck released a binning of the angular power spectrum in the Explanatory supplement, unfortunately the file is in PDF format, non easily machine-readable: http://www.sciops.esa.int/wikiSI/planckpla/index.php?title=Frequencymapsangularpowerspectra&instance=PlanckPublicPLA So here is a csv version: https://gist.github.com/zonca/6288439 Follows embedded gist.
-
HEALPix map of the Earth using healpy
HEALPix maps can also be used to create equal-area pixelized maps of the Earth, RGB colors are not supported in healpy, so we need to convert the image to colorscale. The best user case is for using spherical harmonic transforms, e.g. apply a smoothing filter, in this case HEALPix/healpy tools are really efficient.
-
Export google analytics data via API with Python
Fun weekend hacking project: export google analytics data using the google APIs. Clone the latest version of the API client from: https://code.google.com/p/google-api-python-client there is an example for accessing analytics APIs in the samples/analytics folder, but you need to fill in clientsecrets.json.
-
Processing sources in Planck maps with Hadoop and Python
Purpose The purpose of this post is to investigate how to process in parallel sources extracted from full sky maps, in this case the maps release by Planck, using Hadoop instead of more traditional MPI-based HPC custom software.
-
How to use the IPython notebook on a small computing cluster
The IPython notebook is a powerful and easy to use interface for using Python and particularly useful when running remotely, because it allows the interface to run locally in your browser, while the computing kernel runs remotely on the cluster.
-
IPython parallell setup on Carver at NERSC
IPython parallel is one of the easiest ways to spawn several Python sessions on a Supercomputing cluster and process jobs in parallel. On Carver, the basic setup is running a controller on the login node, and submit engines to the computing nodes via PBS.
-
Simple Mixin usage in python
One situation where Mixins are useful in Python is when you need to modify a method of similar classes that you are importing from a package.
-
Noise in spectra and map domain
Spectra NET or $\sigma$ is the standard deviation of the noise, measured in mK/sqrt(Hz), typical values for microwave amplifiers are 0.2-5. This is the natural unit of the amplitude spectra (ASD), therefore the high frequency tail of the ASD should get to the expected value of the NET.