High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
Vim regular expressions
very good reference of the usage of regular expressions in VIM: http://www.softpanorama.org/Editors/Vimorama/vimregularexpressions.shtml
-
set python logging level
often using logging.basicConfig is useless because if the logging module is already configured upfront by one of the imported libraries this is ignored. The solution is to set the level directly in the root logger: logging.root.level = logging.DEBUG
-
pyfits memory leak in new_table
I found a memory leakage issue in pyfits.newtable, data were NOT deleted when the table was deleted, I prepared a test on github, using objgraph , which shows that data are still in memory: https://gist.github.com/884298 the issue was solved by Erik Bray of STSCI on March 28th, 2011 , see bug report: http://trac6.assem
-
ipython and PyTrilinos
start ipcontroller start ipengines: mpiexec -n 4 ipengine --mpi=pytrilinos start ipython 0.11: import PyTrilinos from IPython.kernel import client mec = client.MultiEngineClient() %loadext parallelmagic mec.activate() px import PyTrilinos px comm=PyTrilinos.Epetra.PyComm() px print(comm.NumProc())
-
git make local branch tracking origin
git branch --set-upstream master origin/master you obtain the same result as initial cloning
-
memory map npy files
Mem-map the stored array, and then access the second row directly from disk: X = np.load('/tmp/123.npy', mmapmode='r')
-
force local install of python module
python setup.py install --prefix FOLDER creates lib/python2.6/site-packages, to force a local install you should use: python setup.py install --install-lib FOLDER
-
gnome alt f2 popup launcher
gnome alt f2 popup launcher
-
switch to interactive backend with ipython -pylab
objective: when running ipython without pylab or executing scripts you want to use an image matplotlib backend like Agg just when calling ipython -pylab you want to use an interactive backend like GTKAgg or TKAgg you need first to setup as default backend on .matplotlib/matplotlibrc Agg : backend : Agg then setup you i
-
numpy dtypes and fits keywords
bool: 'L', uint8: 'B', int16: 'I', int32: 'J', int64: 'K', float32: 'E', float64: 'D', complex64: 'C', complex128: 'M'
-
count hits with numpy
I have an array where I record hits a=np.zeros(5) and an array with the indices of the hits, for example I have 2 hits on index 2 hits=np.array([2,2]) so I want to increase index 2 of a by 2 I tried: a[hits]+=1 but it gives array([ 0., 0., 1., 0., 0.]) does someone have a suggestion?
-
change column name in a fits with pyfits
no way to change it manipulating the dtype of the data array. a=pyfits.open('filename.fits') a[1].header.update('TTYPE1','newname') you need to change the header, using the update method of the right TTYPE and then write again the fits file using a.writeto.
-
healpix coordinates
Healpix considers latitude theta from 0 on north pole to pi south pole, so the conversion is: theta = pi/2 - latitude longitude and phi instead are consistently from 0 to 2pi with zero on vernal equinox (for ecliptic ). zero in the direction from Sun to galactic center (for galactic )
-
parallel computing the python way
Note: The original showmedo.com video tutorial site is no longer available. For parallel computing in Python, consider using multiprocessing, dask, or mpi4py as modern alternatives to MPI.
-
quaternions for python
Update: SciPy now includes scipy.spatial.transform.Rotation which provides excellent quaternion support. See the SciPy documentation.