High Performance Computing & AI
Practical notes on Python, JupyterHub, Kubernetes and AI for science — from the San Diego Supercomputer Center.
-
change column name in a fits with pyfits
no way to change it manipulating the dtype of the data array. a=pyfits.open('filename.fits') a[1].header.update('TTYPE1','newname') you need to change the header, using the update method of the right TTYPE and then write again the fits file using a.writeto.
-
healpix coordinates
Healpix considers latitude theta from 0 on north pole to pi south pole, so the conversion is: theta = pi/2 - latitude longitude and phi instead are consistently from 0 to 2pi with zero on vernal equinox (for ecliptic ). zero in the direction from Sun to galactic center (for galactic )
-
parallel computing the python way
Note: The original showmedo.com video tutorial site is no longer available. For parallel computing in Python, consider using multiprocessing, dask, or mpi4py as modern alternatives to MPI.
-
quaternions for python
Update: SciPy now includes scipy.spatial.transform.Rotation which provides excellent quaternion support. See the SciPy documentation.
-
change permission recursively to folders only
change permission recursively to folders only
-
aptitude search 'and'
this is really something really annoying about aptitude, if you run: aptitude search linux headers it will make an 'or' search...to perform a 'and' search, which I need 99.9% of the time, you need quotation marks: aptitude search 'linux headers'
-
using numpy dtype with loadtxt
Let's say you want to read a text file like this: filename start end fdsafda.fits 23143214 23143214 safdsafafds.fits 21423 23423432 you can use dtype to create a custom array, which is very flexible as you can work by row or columns with strings and floats in the same array: dt=np.dtype({'names':['filename','start','end'],'formats':['S100',np.float,np.float]}) [I tried also using np.str instead of S100 without success, anyone knows why?] then give this as input to loadtxt to load the file and create the array.
-
Stop ipcluster from a script
Ipcluster is easy to start but not trivial to stop from a script, after having finished the processing, here's the solution: from IPython.kernel import client mec = client.MultiEngineClient() mec.kill(controller=True)
-
Correlation
Expectation value or first moment of a random variable is the probability weighted sum of the possible values (weighted mean). Expectation value of a 6-dice is 1+2+3+4+5+6 / 6 = 3.5 Covariance of 2 random variables is: COV(X,Y)=E[(X-E(X))(Y-E(Y))]=E(XY) - E(X)E(Y) i.e.
-
execute bash script remotely with ssh
a bash script launched remotely via ssh does not load the environment, if this is an issue it is necessary to specify --login when calling bash: ssh user@remoteserver.com 'bash --login lifeom/cronodproc' | mail your@email.com -s cronodproc
-
lock pin hold a package using apt on ubuntu
set hold: echo packagename hold | dpkg --set-selections check, should be hi : dpkg -l packagename unset hold: echo packagename install | dpkg --set-selections
-
load arrays from a text file with numpy
space separated text file with 5 arrays in columns: [sourcecode language="python"] ods,rings,gains,offsets,rparams = np.loadtxt(filename,unpack=True) [/sourcecode] quite impressive...
-
Latest Maxima and WxMaxima for Ubuntu Karmic
Note: The original third-party repository for Maxima packages is no longer available. On the maxima mailing lists they suggested to install the sbcl built, so I first installed sbcl from the Ubuntu repositories and then maxima and wxmaxima from the third-party repository.
-
number of files in a folder and subfolders
folders are not counted find . -type f | wc -l
-
forcefully unmount a disk partition
check which processes are accessing a partition: [sourcecode language="python"]lsof | grep '/opt'[/sourcecode] kill all the processes accessing the partition (check what you're killing, you could loose data): [sourcecode language="python"]fuser -km /mnt[/sourcecode] try to unmount now: [sourcecode language="python"]umo