Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hsf-training/PyHEP-resources

Python libraries of interest to particle physicists. This is meant for educational purposes.
https://github.com/hsf-training/PyHEP-resources

documentation hep libraries particle-physics pyhep python python3

Last synced: 14 days ago
JSON representation

Python libraries of interest to particle physicists. This is meant for educational purposes.

Awesome Lists containing this project

README 

        
# Python Libraries of Interest to Particle Physics



[![Join the chat at https://gitter.im/HSF/PyHEP](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/HSF/PyHEP?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1420444.svg)](https://doi.org/10.5281/zenodo.1420444)



Python libraries of interest to particle physicists.  This is meant to be a living document.  Therefore, if you have suggestions, click the edit button then make a pull request with your proposed change(s). This is meant to focus on Python resources for new-ish users; if you just want to look for different libraries for a particular purpose, please visit [the Python section of the awesome-hep](https://github.com/iris-hep/awesome-hep#python) list.



You are more than welcome to join the [HSF/PyHEP Gitter channel](https://gitter.im/HSF/PyHEP)

and contribute to the informal discussions there.

The channel [HSF/PyHEP-newcomers Gitter channel](https://gitter.im/HSF/PyHEP-newcomers) specifically targets, well, newcomers.



See the [HSF](https://hepsoftwarefoundation.org/) PyHEP Working Group [page](https://hepsoftwarefoundation.org/workinggroups/pyhep.html)

for a full list of Gitter channels and resources.



## New to Python



If you are new to Python, the following contains general information on using Python in Science.



| Name         | Use             |

| ------------ | --------------- |

| [Software Carpentry Python Lesson][] | Lesson aimed at people who have never used Python before.  |

| [Scipy tutorials][] | You'll want the beginner courses, the intermediate/advanced courses are actually quite advanced. Setup instructions are linked on the page, videos are [here][Scipy tutorial videos] |

| [Dive into Python 3][] | Very useful for learning python, though it's a bit old and doesn't cover any of the scientific python stuff you really need. |

| [Code academy][] | |

| Many EdX and Coursera courses | Often introductory CS courses which can teach other useful skills (algorithms and datastructures) |

| [The python docs][] | |

| [Level Up Your Python][] | Advanced topics: Debuggers, static typing, logging, decorators, generators and more as interactive Jupyter book |



[Software Carpentry Python Lesson]: http://swcarpentry.github.io/python-novice-inflammation/

[Scipy tutorials]:                  https://scipy2017.scipy.org/ehome/220975/493423/

[Scipy tutorial videos]:            https://www.youtube.com/playlist?list=PLYx7XA2nY5GfdAFycPLBdUDOUtdQIVoMf

[Dive into Python 3]:               http://www.diveintopython3.net/index.html

[Code academy]:                     https://www.codecademy.com/learn/python

[The python docs]:                  https://docs.python.org/3/

[Level Up Your Python]:             https://henryiii.github.io/level-up-your-python/



Otherwise, just google python + description of problem, usually answer is on stackoverflow.



Youtube channels with talks / tutorials:



Pycon, e.g.:

  * [PyCon 2019](https://www.youtube.com/@pycon2019)

  * [PyCon 2018](https://www.youtube.com/@pycon2018)



Some more advanced talks of interested:

  * [Ned Batchelder, "Facts and Myths about Python names and values"](https://www.youtube.com/watch?v=_AEJHKGk9ns)

  * [David Baumgold, "Advanced Git"](https://www.youtube.com/watch?v=4EOZvow1mk4)

  * [Thomas Ballinger, "Finding closure with closures"](https://www.youtube.com/watch?v=E9wS6LdXM8Y)



## Getting Python



| Name         | Use             |

| ------------ | --------------- |

| [Conda][] | Anaconda packages most scientific Python libraries while also living purely in user space.  Therefore, you don't need special permissions to setup. Anaconda is a metapackage of 100 or so scientific Python packages for Conda. |

| [pipenv][] | Very slick all-in-one combination of virtual environments and package installation, can manage Python installs too. |

| [ripa][] | ripa solves the packaging issue by letting you install packages (or requirements.txt) where priority given to conda channels, otherwise fetches from PyPI. (Works but rough) |



[Conda]:  https://conda.io/docs/user-guide/getting-started.html

[pipenv]: https://pipenv.readthedocs.io/en/latest/

[ripa]:   https://github.com/tunnell/ripa



## Scientific Python Stack



The packages that are used in Physics and/or data science within Python grew somewhat organically before forming the current ecosystem.  Detailed information on what exists can be found [here](https://scipy.org/about.html), but we will summarize here.



| Name         | Use             |

| ------------ | --------------- |

| [jupyter notebook][] | Main one way of doing interactive and/or exploratory analysis. |

| [numpy][] | Array and matrix operations (including math operations) at C speeds. |

| [pandas][] | A very elegant way to work with tabular data (i.e. ntuples) with in memory calculations.  Especially good at time series. |

| [xarray][] | Extension of pandas to N-Dim structures. |

| [h5py][] | Simple numpy to HDF5 bindings (backend for Keras saved models). |

| [scipy][] | Various scientific routines like minimization. |

| [matplotlib][] | Main Python plotting library.  Start from [matplotlib gallery][] then adapt to your application. |

| [scikit-learn][] | Very easy to use machine learning routines with great examples. |

| [awkward][] | Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms. |



[jupyter notebook]:   https://jupyter.org

[numpy]:              http://www.numpy.org

[pandas]:             https://pandas.pydata.org

[xarray]:             http://xarray.pydata.org/en/stable/why-xarray.html#core-data-structures

[h5py]:               http://www.h5py.org/

[scipy]:              https://www.scipy.org

[matplotlib]:         https://matplotlib.org

[matplotlib gallery]: https://matplotlib.org/gallery.html

[scikit-learn]:       http://scikit-learn.org/stable/index.html

[awkward]:            https://awkward-array.org



Visualisation:



| Name         | Use             |

| ------------ | --------------- |

| [matplotlib][] | Main Python plotting library.  Start from [matplotlib gallery][] then adapt to your application. |

| [seaborn][] | Easier to use plotting library with some statistical routines. Builds on matplotlib, but annoying to customize. |



[seaborn]: https://seaborn.pydata.org



Machine learning:



| Name         | Use             |

| ------------ | --------------- |

| [scikit-learn][] | Popular package. Very easy to use ML routines, with great examples. |

| [tensorflow][] | By Google, for deep neural networks and more. |

| [pytorch][] | Deep learning framework for fast, flexible experimentation with dynamic computational graphs. |

| [keras][] | Higher level neural network interfaces. |

| [wandb][] | Weights & Biases makes it easy to track your experiments, manage & version your data, and collaborate with your team so you can focus on building the best models. |



[tensorflow]: http://tensorflow.org

[pytorch]: https://pytorch.org

[keras]: https://keras.io

[wandb]: https://wandb.ai/site



General information through talks that may be useful on PyData (various conferences each year):

  *  and other per-conference channels

  * [Scipy conferences](https://www.youtube.com/playlist?list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6)

  * [Enthought](https://www.youtube.com/user/EnthoughtMedia)

  * [Continuum Analytics](https://www.youtube.com/channel/UCND4vKhJssAtK8p1Blfj14Q)



## Data manipulation



| Name         | Use             |

| ------------ | --------------- |

| [boost-histogram][] | Python bindings for the C++14 Boost::Histogram library. |

| [hist][] | Analyst-friendly front-end for boost-histogram. |



[boost-histogram]: https://github.com/scikit-hep/boost-histogram

[hist]: https://github.com/scikit-hep/hist



## Statistical analysis and fitting



| Name         | Use             |

| ------------ | --------------- |

| [iminuit][] | Jupyter-friendly Python frontend to the MINUIT2 C++ library. |

| [uncertainties][] | Calculations with numbers with uncertainties. It also yields the derivatives of any expression. |

| [pyhf][] | Pure python implementation of HistFactory specification with auto-diff enabled backends in tensorflow, pytorch, and MXNet. |

| [hepstats][] | HEP statistics tools and utilities. |

| [zfit][] | Scalable pythonic fitting. |

| [cabinetry][] | Constructing binned template fit models and steering statistical inference using pyhf. |



[iminuit]: https://github.com/scikit-hep/iminuit

[hepstats]: https://github.com/scikit-hep/hepstats

[pyhf]: https://github.com/scikit-hep/pyhf

[uncertainties]: https://github.com/lebigot/uncertainties

[zfit]: https://github.com/zfit/zfit

[cabinetry]: https://github.com/scikit-hep/cabinetry/



## Particle Physics packages

| Name         | Use             |

| ------------ | --------------- |

| [DecayLanguage][] | Describe, manipulate and display particle decays. |

| [hepstats][] | HEP statistics tools and utilities. |

| [hepunits][] | Units and constants in the HEP system of units. |

| [Particle][] | PDG particle data and identification codes. |

| [pylhe][] | Interface to read Les Houches Event (LHE) files. |

| [fastjet][] | Official FastJet bindings to Python and Awkward Array. |

| [vector][] | Arrays of 2D, 3D, and Lorentz vectors. |



[DecayLanguage]: https://github.com/scikit-hep/decaylanguage

[hepunits]: https://github.com/scikit-hep/hepunits

[Particle]:  https://github.com/scikit-hep/particle

[pylhe]:     https://github.com/scikit-hep/pylhe

[fastjet]:     https://github.com/scikit-hep/fastjet

[vector]:     https://github.com/scikit-hep/vector



## ROOT and interoperability with ROOT



For many particle physics experiments, a lot of data is stored within ROOT files.  This means at very least one must have the ability to read ROOT files.  ROOT also serves as a tool suite designed to solve many computational problems encountered in HEP, which means that one may want to access some of this tool suite.  The following packages below are worth knowing for these situations:



| Package name | Use | Pro | Con | Further information |

| ------------ | --- | --- | --- | ------------------- |

| [ostap][] | User-friendly & more intuitive interface to(Py)ROOT | Many decorations to ROOT classes | Requires C++ code compilation |  |

| [uproot][] | Native Python ROOT I/O | Easy to install, fast, no dependence on C++ ROOT | Although it can read all ROOT files, can only write ROOT files with specific objects. | |

| [Conda-Forge ROOT][] | Using ROOT within Anaconda | Full-featured ROOT and PyROOT on Linux and macOS |  |  |

| [PyROOT][] | Official ROOT Python bindings | Good support and many examples | Raw C++ wrapping results in weird Python code (improved in 6.22) |  |

| [rootpy][] | Pythonic ROOT access | More logical for people who know Python | Abandoned, mostly replaced by ROOT 6.22+ new bindings or the uproot family. | [Repository][rootpy repository] |

| [alphatwirl][] | Summarizing ROOT data into categorical data as Pandas' data frames | Small output size. Easy one-function interface with [qtwirl][] | Not for data type conversion | |

| [pyhf][] | statistical analysis / fitting | pure python implementation of HistFactory specification with auto-diff enabled backends in tensorflow, pytorch, and MXNet | not yet interoperable with ROOT-based RooFit models | |



[ostap]:                https://github.com/OstapHEP/ostap

[uproot]:               https://github.com/scikit-hep/uproot4

[Conda-Forge ROOT]:     https://github.com/conda-forge/root-feedstock

[NLeSC: ROOT on conda]: https://nlesc.gitbooks.io/cern-root-conda-recipes/content/installing_root_via_anaconda.html

[PyROOT]:               https://root.cern.ch/pyroot

[rootpy]:               http://www.rootpy.org

[rootpy repository]:    https://github.com/rootpy/rootpy/

[alphatwirl]:           https://github.com/alphatwirl/alphatwirl

[qtwirl]:               https://github.com/alphatwirl/qtwirl



## Jupyter extensions



Jupyter has a wide ecosystem of extensions that can be used to extend the functionality. Some useful extensions for HEP data analysis are summarised here.



| Name         | Use             |

| ------------ | --------------- |

| [nbdime][]   | Simplifies diffing and merging of jupyter notebooks that are stored in version control. |

| [jupytext][] | Splits notebooks into a `.ipynb` and `.py` file for easier version control and to allow them to be run as scripts independently of jupyter. |



[nbdime]:   https://github.com/jupyter/nbdime

[jupytext]: https://github.com/mwouts/jupytext



## Speeding up code



Often, it is not needed anymore to write C++/C routines that get wrapped since there are other ways to speed up your Python code.  Namely:



| Name         | Use             |

| ------------ | --------------- |

| [numba][] | Tight loops are often the slow part of Python code, where this JIT compiles them! |

| [Pythran][] | Whole scripts |

| [numpy][] | Expressing your code as array options means you get native-C speeds per sub-expression. |

| [jax][] | Allows compiling and running NumPy operations on accelerated hardware such as GPU's. Also offers a JIT compiler (less sophisticated than Numba), automatic analytic differentiation (gradient) of Python functions and efficient vectorization. Was developed with the aim of being able to develop efficient machine learning algorithms "from scratch" with NumPy-like code.

| [NumExpr][] | Single pass "mapper" operations (one input → one output). |



[numba]:   https://numba.pydata.org

[Pythran]: https://pythran.readthedocs.io/en/latest/

[numpy]:   http://www.numpy.org

[jax]:     Https://jax.readthedocs.io/en/latest/notebooks/quickstart.html

[NumExpr]: https://numexpr.readthedocs.io/en/latest/user_guide.html



## Binding C/C++ to Python



*Before you read this, realize that this is for existing C++ code.  If you want to write new C/C++ code for speed, see section above.*



Python entered into the particle physics ecosystem since it was useful as a 'glue lanaguage'.  This means that you can get multiple softwares in different languages to work with one another.  Given the large ecosystem of Python packages in the last decade, this is less common now.  However, the situation still does arise that you want to call some existing C/C++ code from Python.



Please be aware that wrapping C code is significantly easier than wrapping C++ code due to details of how function names get garbled in libraries within C++; but the tools below can make wrapping C++ easy as well.



At present, the best summary of how to bind code in HEP applications comes from Henry Schreiner in a [2018 PyHEP talk][Binding Python].



| Package name  | Use | Pro | Con | Further information |

| ------------- | --- | --- | --- | ------------------- |

| [pybind11][]  | Wrapping existing C++ codes  | Small elegant package, simple build. | Young but gaining populatiry quickly. | [Henry's slides][Binding Python] |

| [Cython][]  | Wrapping C++ code  | Widely used, freely mixing Python and C++. | Mix of C and Python is a new language, incomplete coverage of C++. | |

| [SWIG][] | Wrapping C++ code  | Widely used, multiple languages. | Have to write wrapper file, harder to customize, and development is slow/dated. | |

| [Boost::Python][] | Wrapping C++ code | Widely used.  | Giant dependency since Boost does many other things, uses "jam" to build.| |



[pybind11]:       http://pybind11.readthedocs.io

[Binding Python]: https://indico.cern.ch/event/694818/contributions/2985778/attachments/1682465/2703470/PyHEPTalk.pdf

[Cython]:         https://cython.readthedocs.io/en/latest/

[SWIG]:           http://www.swig.org

[Boost::Python]:  https://www.boost.org/doc/libs/1_69_0/libs/python/doc/html/index.html



## Miscellaneous



Packages that do not easily fit in any of the above topics.



| Name         | Use             |

| ------------ | --------------- |

| [comparxiv][] | Compare 2 versions of an arXiv preprint with a single command |



[comparxiv]: https://pypi.org/project/comparxiv/



## Tutorials



See tutorials here and other resources collected by [IML HEP-ML Resources](https://github.com/iml-wg/HEP-ML-Resources#tutorials).



## Experimental codes



Stealing code from other physicists is its own sign of flattery.  Codes that are abandoned more than two years will get struck through.:



|  Name | Collaboration | Use | Further information | Date added to list |

| ----- | ---------- | --- | ------------------- | --------------------- |