High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
some python resources
python tutorial: https://docs.python.org/3/tutorial/ numpy tutorial [arrays]: https://numpy.org/doc/stable/user/quickstart.html plotting tutorial: https://matplotlib.org/stable/tutorials/pyplot.html free online books: https://diveintopython3.net/ https://python.swaroopch.com/ install python (recommended): https://www.a
-
cfitsio wrapper in python
After several issues with pyfits, and tired of it being so overengineered, I've wrote my own fits I/O package in python, wrapping the C library cfitsio with ctypes. Pretty easy, first version completely developed in 1 day. https://github.com/zonca/pycfitsio
-
unit testing happiness
nosetests -v testallcols (pycfitsio.test.TestPyCfitsIoRead) ... ok testcolnames (pycfitsio.test.TestPyCfitsIoRead) ... ok testmove (pycfitsio.test.TestPyCfitsIoRead) ... ok testopenfile (pycfitsio.test.TestPyCfitsIoRead) ... ok testreadcol (pycfitsio.test.TestPyCfitsIoRead) ...
-
Pink noise (1/f noise) simulations in numpy
Pink noise (1/f noise) simulations in numpy
-
Vim regular expressions
very good reference of the usage of regular expressions in VIM: http://www.softpanorama.org/Editors/Vimorama/vimregularexpressions.shtml
-
set python logging level
often using logging.basicConfig is useless because if the logging module is already configured upfront by one of the imported libraries this is ignored. The solution is to set the level directly in the root logger: logging.root.level = logging.DEBUG
-
pyfits memory leak in new_table
I found a memory leakage issue in pyfits.newtable, data were NOT deleted when the table was deleted, I prepared a test on github, using objgraph , which shows that data are still in memory: https://gist.github.com/884298 the issue was solved by Erik Bray of STSCI on March 28th, 2011 , see bug report: http://trac6.assem
-
ipython and PyTrilinos
start ipcontroller start ipengines: mpiexec -n 4 ipengine --mpi=pytrilinos start ipython 0.11: import PyTrilinos from IPython.kernel import client mec = client.MultiEngineClient() %loadext parallelmagic mec.activate() px import PyTrilinos px comm=PyTrilinos.Epetra.PyComm() px print(comm.NumProc())
-
git make local branch tracking origin
git branch --set-upstream master origin/master you obtain the same result as initial cloning
-
memory map npy files
Mem-map the stored array, and then access the second row directly from disk: X = np.load('/tmp/123.npy', mmapmode='r')
-
force local install of python module
python setup.py install --prefix FOLDER creates lib/python2.6/site-packages, to force a local install you should use: python setup.py install --install-lib FOLDER
-
gnome alt f2 popup launcher
gnome alt f2 popup launcher
-
switch to interactive backend with ipython -pylab
objective: when running ipython without pylab or executing scripts you want to use an image matplotlib backend like Agg just when calling ipython -pylab you want to use an interactive backend like GTKAgg or TKAgg you need first to setup as default backend on .matplotlib/matplotlibrc Agg : backend : Agg then setup you i
-
numpy dtypes and fits keywords
bool: 'L', uint8: 'B', int16: 'I', int32: 'J', int64: 'K', float32: 'E', float64: 'D', complex64: 'C', complex128: 'M'
-
count hits with numpy
I have an array where I record hits a=np.zeros(5) and an array with the indices of the hits, for example I have 2 hits on index 2 hits=np.array([2,2]) so I want to increase index 2 of a by 2 I tried: a[hits]+=1 but it gives array([ 0., 0., 1., 0., 0.]) does someone have a suggestion?