{"id":13477341,"url":"https://github.com/hep-lbdl/CaloGAN","last_synced_at":"2025-03-27T05:31:25.512Z","repository":{"id":71445325,"uuid":"82329392","full_name":"hep-lbdl/CaloGAN","owner":"hep-lbdl","description":"Generative Adversarial Networks for High Energy Physics extended to a multi-layer calorimeter simulation","archived":false,"fork":false,"pushed_at":"2024-06-08T16:48:51.000Z","size":17264,"stargazers_count":108,"open_issues_count":8,"forks_count":66,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-12-06T20:19:18.872Z","etag":null,"topics":["atlas","calogan","calorimeter","cern","deep-learning","gan","generative-adversarial-network","hep","high-energy-physics","machine-learning","physics"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hep-lbdl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-17T19:10:23.000Z","updated_at":"2024-11-26T12:36:39.000Z","dependencies_parsed_at":"2024-10-30T10:40:53.665Z","dependency_job_id":null,"html_url":"https://github.com/hep-lbdl/CaloGAN","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hep-lbdl%2FCaloGAN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hep-lbdl%2FCaloGAN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hep-lbdl%2FCaloGAN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hep-lbdl%2FCaloGAN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hep-lbdl","download_url":"https://codeload.github.com/hep-lbdl/CaloGAN/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245791481,"owners_count":20672665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atlas","calogan","calorimeter","cern","deep-learning","gan","generative-adversarial-network","hep","high-energy-physics","machine-learning","physics"],"created_at":"2024-07-31T16:01:41.332Z","updated_at":"2025-03-27T05:31:24.216Z","avatar_url":"https://github.com/hep-lbdl.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# CaloGAN\nSimulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks.\n\nThis repository contains what you'll need to reproduce M. Paganini ([@mickypaganini](https://github.com/mickypaganini)), L. de Oliveira ([@lukedeo](https://github.com/lukedeo)), B. Nachman ([@bnachman](https://github.com/bnachman)), _CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks_ [[`arXiv:1705.02355`](https://arxiv.org/abs/1705.02355)].\n\nYou are more than welcome to use the open data and open-source software provided here for any of your projects, but we kindly ask you that you please cite them using the DOIs provided below:\n\n| Asset  | Location |\n| ------------- | ------------- |\n| Training Data (GEANT4 showers, ⟂ to center) | [![DOI](https://zenodo.org/badge/DOI/10.17632/pvn3xc3wy5.1.svg)](https://doi.org/10.17632/pvn3xc3wy5.1)|\n| Source Code (this repo!) | [![DOI](https://zenodo.org/badge/82329392.svg)](https://zenodo.org/badge/latestdoi/82329392)|\n\n**For any use of paper ideas and results, please cite**\n\n```\n@article{paganini_calogan,\n      author         = \"Paganini, Michela and de Oliveira, Luke and Nachman,\n                        Benjamin\",\n      title          = \"{CaloGAN: Simulating 3D High Energy Particle Showers in\n                        Multi-Layer Electromagnetic Calorimeters with Generative\n                        Adversarial Networks}\",\n      year           = \"2017\",\n      eprint         = \"1705.02355\",\n      archivePrefix  = \"arXiv\",\n      primaryClass   = \"hep-ex\",\n}\n```\n\n## Goal\nThe goal of this project is to help physicists at CERN speed up their simulations by encoding the most computationally expensive portion of the simulation process (i.e., showering in the EM calorimeter) in a deep generative model.\n\nThe challenges come from the fact that this portion of the detector is longitudinally segmented into three layers with heterogeneous granularity. For simplicity, we can visualize the energy depositions of particles passing through the detector as a series of three images per shower, while keeping in mind the sequential nature of their relationship in the generator. \n\nTo download and better understand the training dataset, visit the [Mendeley Data repository](https://data.mendeley.com/datasets/pvn3xc3wy5/1).\n\n![3D shower in the EM calorimeter](/figures/3d.jpg)\n\n## Getting Started\n\nThis repository contains three main folders: `generation`, `models` and `analysis`, which represent the three main stages in this project. You may decide to only engage in a subset of them, so we decided to provide you with all you need to jump right in at the level you are interested in. \n\n`generation` contains the code to build the electromagnetic calorimeter geometry and shoot particles at it with a give energy. This is all based on `Geant4`. For more instructions, go to [Generation on PDSF](#generation-on-pdsf).\n\n`models` contains the core ML code for this project. The file [`train.py`](https://github.com/hep-lbdl/CaloGAN/blob/master/models/train.py) takes care of loading the data and training the GAN. \n\n`analysis` contains Jupyter Notebooks used to evaluate performance and produce all plots for the paper.\n\n## GEANT4 Generation on PDSF\n\nOn [PDSF](http://www.nersc.gov/users/computational-systems/pdsf/), your should run all generation code that outputs big files to a scratch space. To make a scratch environment, run `mkdir /global/projecta/projectdirs/atlas/{your-username}`, and for convenience, link to home via `ln -s /global/projecta/projectdirs/atlas/{your-username} ~/scratch`.\n\nTo build the generation code on PDSF, simply run `source cfg/pdsf-env.sh` from the `generation/` folder in the repository. This loads modules.\n\nNext, you can type `make` which should build an executable called `generate`. Because of how Geant4 works, this executable gets deposited in `$HOME/geant4_workdir/bin/Linux-g++/`, which is in your `$PATH` when the modules from `cfg/pdsf-env.sh` are loaded.\t\n\nTo run the generation script, run `generate -m cfg/run2.mac`. You can change generation parameters inside [`cfg/run2.mac`](https://github.com/hep-lbdl/CaloGAN/blob/master/generation/cfg/run2.mac) (follow [these instructions](https://geant4.web.cern.ch/geant4/UserDocumentation/UsersGuides/ForApplicationDeveloper/html/ch02s07.html)). \n\nThis will output a file called `plz_work_kthxbai.root` with a TTree named `fancy_tree`, which will contain a branch for each calorimeter cell (`cell_#`) with histograms of the energy deposited in that cell across the various shower events. The last three cells (numbered 504, 505, and 506) actually represent the overflow for each calorimeter layer. Finally, a branch called `TotalEnergy` is added for bookkeeping. \n\nWe provide you with a convenient script to **convert the ROOT file into a more manageable HDF5 archive**. The `convert.py` script is located in the `generation/` folder and can be used as follows:\n```\nusage: convert.py [-h] --in-file IN_FILE --out-file OUT_FILE --tree TREE\n\nConvert GEANT4 output files into ML-able HDF5 files\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --in-file IN_FILE, -i IN_FILE\n                        input ROOT file\n  --out-file OUT_FILE, -o OUT_FILE\n                        output HDF5 file\n  --tree TREE, -t TREE  input tree for the ROOT file\n\n```\nSo you can run, for example, `python convert.py --in-file plz_work_kthxbai.root --out-file test.h5 --tree fancy_tree`.\nAssuming you specified `/run/beamOn 1000` in your `run.mac` file, the structure of the output HDF5 should look like this:\n```\nenergy                   Dataset {1000, 1}\nlayer_0                  Dataset {1000, 3, 96}\nlayer_1                  Dataset {1000, 12, 12}\nlayer_2                  Dataset {1000, 12, 6}\noverflow                 Dataset {1000, 3}\n```\nTo launch a batch job on PDSF, simply run `./launch \u003cnum_jobs\u003e`, to launch `\u003cnum_jobs\u003e` concurrent tasks in a job array.\n\n## The CaloGAN Model\nThis work builds on the solution presented in [`arXiv/1701.05927`](https://arxiv.org/abs/1701.05927) which we named LAGAN, or Location-Aware Generative Adversarial Network. This is a Physics specific modification of the more common DCGAN and ACGAN frameworks which is specifically designed to handle the levels of sparsity, the location dependence, and the high dynamic range that characterizes Physics images.\n\n### Generator\n![Generator](figures/caloGAN_gen.jpg)\nThe Generator contains three parallel LAGAN-style streams with an in-painting mechanism to handle the sequential nature of these shower images.\n### Discriminator\n![Discriminator](figures/caloGAN_discr_rev.jpg)\nThe discriminator uses convolutional features combined with ad-hoc notions of sparsity and reconstructed energy to classify real and fake images as well as comparing the reconstructred energy with the condition requested by the user.\n\n## Training the CaloGAN model\nTo begin the training process, create a YAML file with the specification in [`models/particles.yaml`](https://github.com/hep-lbdl/CaloGAN/blob/master/models/particles.yaml) for all the particles you want to train on. **In the paper, we only train one GAN per particle type.** If you specify more than one particle type and dataset in the YAML file, it will train an ACGAN, which we found to not be as performant. Assuming that you have a folder called `data/` in the root directory of the project, and you have a file inside called `eplus.h5` (downloaded from Mendeley, perhaps?), you can train your own CaloGAN by running \n\n```bash\npython -m models.train models/particles.yaml\n```\n\nWe recommend running `python -m models.train -h` at least once to see all the parameters one can change. \n\n## Performance Analysis\n\nPerformance evaluation is done both from a qualitative and a quantitative standpoint. The jupyter notebook available in the `analysis` folder will guide you through our plotting conventions.\n\nFor quick handling, we have pre-extracted the shower shape variables into a pandas dataframe (stored as HDF5) and made it available available [on S3](https://s3-us-west-2.amazonaws.com/lukedeo-data/lbl/calogan/shower-shapes.h5). To load, you can simply do `pd.read_hdf('path/to/shower-shapes.h5')`.\n\n## Docker \n\nRunning in local : \n\n$ `docker run -it --rm -v $PWD/CaloGAN:/home/CaloGAN engineren/calogan-docker python -m models.train models/particles.yaml`\n\n\nRunning naf-ilc-gpu :\n\n$ `singularity pull docker://engineren/calogan-docker:latest`\n\n$ `singularity instance start --bind data:/home/CaloGAN/data --nv calogan-docker_latest.sif caloGAN`\n\n$ `singularity run instance://caloGAN python -m models.train models/particles.yaml`\n\n\n\n## Copyright Notice\n \n“CaloGAN” Copyright (c) 2017, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.\n \nIf you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation \u0026 Partnerships Office at [IPO@lbl.gov](mailto:IPO@lbl.gov).\n \nNOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhep-lbdl%2FCaloGAN","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhep-lbdl%2FCaloGAN","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhep-lbdl%2FCaloGAN/lists"}