Arnaud Beck Laboratoire Leprince-Ringuet, École .

DataAnalysisWith PythonA. BeckIntroductionData Analysis With PythonUsingPythonBasicPythonArnaud BeckScipyData I/OVisualizationLaboratoire Leprince-Ringuet, École Polytechnique, CNRS/IN2P3Space Science Training Week

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

Why come to Python ?DataAnalysisWith PythonShould I use low-level,compiled language or an interpreted language ?Commercial or open source ?A. BeckC/C IntroductionUsingPythonBasicPythonScipyData I/OVisualizationEasy and flexiblePerformancesXFree and available on any systemXMatlabPythonXXX

Why stick to Python ?DataAnalysisWith PythonPython is distinguished by its large and active scientific computing community.There are people developing “libraries” for virtually anything.A. BeckGlue to other es to interface other languages (C/C /Fortran).with the same performances ! !Critical part of codes are written in a lower level language.ScipyData I/OParallelizationVisualizationMPIOpenMPGPUData management and visualizationIO data in any format (HDF5, VTK, .)Data management dedicated libraries (scipy, pandas)Direct visualization or interfaces with other softwares (Paraview, Mayavi)

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

Getting Python for data analysisDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonBasic Python distributionAvailable on any Linux or Mac OS.Critical for data analysisScipyData I/OModules : Scipy, MatplotlibVisualizationApplication specificModules : mpi4py, VTK, pytable, etc.It is possible to install fully pre-built scientific Python environment : “EnthoughtPython Distribution” or “Python(x,y)” for Windows.

Running PythonDataAnalysisWith PythonInteractive mode in a Python shellA. BeckIntroductionUsingPythonBasicPythonScipyData I/OUse of a scriptVisualizationTurn your python script into a unix scriptYou can compile scripts into binary .pyc files. Mostly for developers.

IPython : a convenient and comfortable Python shellDataAnalysisWith PythonA. BeckIntroductionInteresting featuresUsingPythonCommand historyBasicPythonAny Xterm command accessible via ’ !’ScipyCommands auto-completionData I/OQuick help through the use of “ ?”VisualizationInline and interactive graphicsTiming and profiling toolsMany many more .Best tool for exploring, debugging or work interactively. Have a look !

IPython exampleDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

Python is an object oriented languageDataAnalysisWith PythonA. Beck“In Python, we do things with stuff !”Introductionthings operationsstuff objectsUsingPythonBasicPythonScipyData I/OVisualizationTypeExampleNumbers128, 3.14, 4 5jStrings'Rony', ng",2.45)Strings, Lists and Tuples are sequences.Strings and Tuples are “immutable”.

NumbersDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

StringsOrdered collection (or sequence) of charactersDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

String MethodsDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

ListsSequence of any objectsDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

SlicesManipulating sequencesDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

Importing modulesDataAnalysisWith PythonA. BeckModules define new object types and Data I/OVisualizationThe large and growing Python users community provides an increasing numberof modules that already do what you need.

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

The Scipy moduleDataAnalysisWith PythonA. BeckIntroductionUsingPythonScipy is a collection of powerful , high level functions for mathematics and datamanagement. It is based on the numpy.ndarray object type and vectorizedoperations. The operations are optimized and coded in C to deliver highperformances.BasicPythonScipyData I/OVisualizationIf you are using a for loop, you are probably doing something wrong !

Creating an ndarrayDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

Manipulating ndarraysDataAnalysisWith PythonA. BeckIntroductionSlicing is still the basis of array manipulation.UsingPythonReshape – Change number and size of dimensions of the array.BasicPythonSort – Quite self explanatory.ScipyDelete, insert, append – Remove or add parts of the array.Data I/OSqueeze, flatten, ravel – More ways to control dimensionality of the array.VisualizationTranspose,swapaxes, rollaxis – More ways to arange the dimensions asyou wantThese functions are important because a well aranged data is a quicklyprocessed data.

Extracting information from your dataDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualizationIntersection (convenient for filtering)Histograms (perfect for distribution functions)ConvolutionIntegrationInterpolationName it .

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

Reading dataDataAnalysisWith PythonA. BeckIntroductionUsingPythonThe whole game is to fit your data in a ndarray.BasicPythonScipydata scipy.fromfile("file",dtype ’float32’,count -1,sep " ")Data I/OVisualizationWorks with raw binary files and ASCII files but not very flexible.data scipy.loadtxt("file",skiprows 0,delimiter ",")More flexible but works only with text files.

The file objectDataAnalysisWith PythonA. BeckThe file object is a basic python type. It is created byIntroductionUsingPythonBasicPythonScipyData I/Ofid open("filename","r")"r" for read, "w" for write.Visualizationfid.readline() – reads a line in a stringfid.readlines() – reads all line in a list of stringsfid.tell() – returns the file’s current position (in byte)fid.seek(n) – goes to position nfid.read() – reads all file in a stringfid.close()

Manipulating a fileDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

Quick words about reading HDF5 filesDataAnalysisWith PythonReading HDF5 files is module dependant. You can use either “tables” or “h5py”for instance.A. BeckIntroductionThese modules coexist well with Scipy and load data directly into ndarray.UsingPythonBasicPythonScipyData I/OVisualizationtables example

Writing dataDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualizationscipy.save("file",ndarray) and scipy.load("file") in order to usethe binary scipy format to store arrays.ndarray.tofile() in order to store an array in a text file or raw binary.fileobject.write("any string") to write a string in a text file.The h5py and tables modules are used to write HDF5 files.VTK script

OutlineDataAnalysisWith PythonA. BeckIntroduction1Introduction2Using Python3Basic Python4Scipy5Data I/O6VisualizationUsingPythonBasicPythonScipyData I/OVisualization

Visualization workflowDataAnalysisWith PythonA. BeckIntroductionUsingPythonPythonRaw DataPythonPost processed dataFormated data fileBasicPythonScipyData I/OPython, matplotlibVisualizationVisualization softwareParaview, Visit, Mayavi .PlotVisualization

Matplotlib : the figure objectfig figure([options])DataAnalysisWith PythonOptions include :Size in inchesUsingPythonDpiBasicPythonFace and edge colorsScipyFrame layoutData I/OVisualizationOperations include :Title and axis labelsfig.xlabel("string")Axis ticks and extentfig.ticks(ndarray)1.00.8Normalized to maximumA. BeckIntroduction0.60.4Injected ChargeDisplay a colorbarfig.colorbar()Bubble Size0.2Laser AmplitudeDisplay a legendfig.legend()Save figure (png or eps)fig.savefig()0.00510Propagation length [mm]15

Matplotlib : Simple plotsDataAnalysisWith Pythonplot(x,y,[options])A. BeckIntroductionIf x is omitted, default is x range(len(y)).UsingPythonBasicPythonScipyData I/OVisualizationAll typical options are here : lines (style, color, width .), markers (size, shape,colors .), labels for legend, antialiasing, transparency, many more .

Matplotlib 2D plots : imshow and pcolorDataAnalysisWith PythonA. BeckIntroduction2Dar rand((100,100))imshow(2Dar,[options])2Dar sicPython1.01001.0Scipy0.900.9Data 801000.0

2D plots with a little bit of tuningDataAnalysisWith PythonA. Data I/OVisualizationy [µm]Scipy0 201.0 40 600.0 60 40x 20 ct [µm]0

Other features of matplotlibDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualization

Matplotlib has native LATEX renderingDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualizationlabel r" Math \LaTex code "

The futur of visualization in PythonDataAnalysisWith PythonA. BeckIntroductionUsingPythonBasicPythonScipyData I/OVisualizationIt is an extremely vast, active and changing domain.New modules are emerging : Chaco, MayaVi, Bokeh, stressing interactivity anddynamic data visualizations in web browsers and in 3D.What you saw today is extremely basic and is only a tiny part of what Python iscapable of.

Python Scipy Data I/O Visualization Python is an object oriented language “In Python, we do things with stuff!” things operations stuff objects Type Example Numbers 128, 3.14, 4 5j Strings ’Rony , "Giovanni’s" Lists [1,"string",2.45] Tuples (1,"string",2.45) Strings, Lists and Tuples are sequences. Strings and Tuples are .