{"id":26460845,"url":"https://github.com/libertem/libertem","last_synced_at":"2026-04-09T20:15:53.842Z","repository":{"id":37405822,"uuid":"116963730","full_name":"LiberTEM/LiberTEM","owner":"LiberTEM","description":"Open pixelated STEM framework","archived":false,"fork":false,"pushed_at":"2025-03-31T19:56:22.000Z","size":235130,"stargazers_count":114,"open_issues_count":181,"forks_count":68,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-02T04:15:47.178Z","etag":null,"topics":["data-processing","electron-microscopy","image-processing","python"],"latest_commit_sha":null,"homepage":"https://libertem.github.io/LiberTEM/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LiberTEM.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-10T13:49:16.000Z","updated_at":"2025-03-27T17:15:52.000Z","dependencies_parsed_at":"2023-10-20T21:14:33.666Z","dependency_job_id":"9992a591-a271-488d-91b5-1babe0713503","html_url":"https://github.com/LiberTEM/LiberTEM","commit_stats":{"total_commits":3410,"total_committers":34,"mean_commits":"100.29411764705883","dds":0.5765395894428152,"last_synced_commit":"6092429b367524f2271ac8cc035e863991fec37c"},"previous_names":[],"tags_count":101,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiberTEM%2FLiberTEM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiberTEM%2FLiberTEM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiberTEM%2FLiberTEM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiberTEM%2FLiberTEM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LiberTEM","download_url":"https://codeload.github.com/LiberTEM/LiberTEM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987185,"owners_count":21028891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-processing","electron-microscopy","image-processing","python"],"created_at":"2025-03-19T03:37:37.530Z","updated_at":"2026-02-05T13:02:33.640Z","avatar_url":"https://github.com/LiberTEM.png","language":"Python","readme":"|docs|_ |gitter|_ |azure|_ |github|_ |precommit|_ |joss|_ |zenodo|_ |pypi|_ |condaforge|_\n\n.. |docs| image:: https://img.shields.io/badge/%F0%9F%95%AE-docs-green.svg\n.. _docs: https://libertem.github.io/LiberTEM/\n\n.. |gitter| image:: https://badges.gitter.im/join_chat.svg\n.. _gitter: https://gitter.im/LiberTEM/Lobby\n\n.. |azure| image:: https://dev.azure.com/LiberTEM/LiberTEM/_apis/build/status/LiberTEM.LiberTEM-data?branchName=master\n.. _azure: https://dev.azure.com/LiberTEM/LiberTEM/_build/latest?definitionId=4\u0026branchName=master\n\n.. |zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1477847.svg\n.. _zenodo: https://doi.org/10.5281/zenodo.1477847\n\n.. |github| image:: https://img.shields.io/badge/GitHub-MIT-informational\n.. _github: https://github.com/LiberTEM/LiberTEM/\n\n.. |joss| image:: https://joss.theoj.org/papers/10.21105/joss.02006/status.svg\n.. _joss: https://doi.org/10.21105/joss.02006\n\n.. |precommit| image:: https://results.pre-commit.ci/badge/github/LiberTEM/LiberTEM/master.svg\n.. _precommit: https://results.pre-commit.ci/latest/github/LiberTEM/LiberTEM/master\n\n.. |pypi| image:: https://badge.fury.io/py/libertem.svg\n.. _pypi: https://pypi.org/project/libertem/\n\n.. |condaforge| image:: https://anaconda.org/conda-forge/libertem/badges/version.svg\n.. _condaforge: https://anaconda.org/conda-forge/libertem\n\nLiberTEM is an open source platform for high-throughput distributed processing\nof large-scale binary data sets and live data streams using a modified\n`MapReduce programming model \u003chttps://en.wikipedia.org/wiki/MapReduce\u003e`_. The\ncurrent focus is `pixelated\n\u003chttps://en.wikipedia.org/wiki/Scanning_transmission_electron_microscopy#Universal_detectors\u003e`_\nscanning transmission electron microscopy (`STEM\n\u003chttps://en.wikipedia.org/wiki/Scanning_transmission_electron_microscopy\u003e`_)\n\\[`MacLaren et al. (2016) \u003chttps://doi.org/10.1002/9783527808465.EMC2016.6284\u003e`_,\n`Ophus (2019) \u003chttps://doi.org/10.1017/s1431927619000497\u003e`_\\] and scanning electron\nbeam diffraction data.\n\nMapReduce-like processing allows to specify an algorithm through two functions:\nOne function that is mapped on portions of the input data, and another function\nthat merges (reduces) a partial result from this mapping step into the complete\nresult. A wide range of TEM and 4D STEM processing tasks can be expressed in\nthis fashion, see `Applications`_.\n\nThe UDF interface of LiberTEM offers a standardized, versatile API to decouple\nthe mathematical core of an algorithm from details of data source, parallelism,\nand use of results. Mapping and merging can be performed in any order and with\ndifferent subdivisions of the input data, including running parts of the\ncalculation concurrently. That means the same implementation can be used in a\nwide range of modalities, including massive scaling on clusters. Since each\nmerge step produces an intermediate result, this style of processing is suitable\nfor displaying live results from a running calculation in a GUI application and\nfor `processing live data streams \u003chttps://github.com/LiberTEM/LiberTEM-live\u003e`_.\nA closed-loop feedback between processing and instrument control can be realized\nas well. See `User-defined functions\n\u003chttps://libertem.github.io/LiberTEM/udf.html\u003e`_ for more details on the\nLiberTEM UDF interface.\n\nThe LiberTEM back-end offers `high throughput and scalability\n\u003chttps://libertem.github.io/LiberTEM/architecture.html\u003e`_ on PCs, single server\nnodes, clusters and cloud services. On clusters it can use fast distributed\nlocal storage on high-performance SSDs. That way it achieves `very high\naggregate IO performance\n\u003chttps://libertem.github.io/LiberTEM/performance.html\u003e`_ on a compact and\ncost-efficient system built from stock components. All CPU cores and CUDA\ndevices in a system can be used in parallel.\n\nLiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow\ninstallation of a suitable Python version and the required packages will likely\nwork as well. The GUI is running in a web browser.\n\nInstallation\n------------\n\nThe short version:\n\n.. code-block:: shell\n\n    $ virtualenv -p python3 ~/libertem-venv/\n    $ source ~/libertem-venv/bin/activate\n    (libertem-venv) $ python -m pip install \"libertem[torch]\"\n\n    # optional for GPU support\n    # See also https://docs.cupy.dev/en/stable/install.html\n    (libertem-venv) $ python -m pip install cupy\n\nPlease see `our documentation\n\u003chttps://libertem.github.io/LiberTEM/install.html\u003e`_ for details!\n\nAlternatively, to run the `LiberTEM Docker image\n\u003chttps://libertem.github.io/LiberTEM/deployment/clustercontainer.html\u003e`_:\n\n.. code-block:: shell\n\n    $ docker run -p 127.0.0.1:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro ghcr.io/libertem/libertem\n\nor\n\n.. code-block:: shell\n\n    $ apptainer exec docker://ghcr.io/libertem/libertem /venv/bin/libertem-server\n\nDeployment for offline data processing on a single-node system for a local user\nis thoroughly tested and can be considered stable. Deployment on a cluster is\nexperimental and still requires some additional work, see `Issue #105\n\u003chttps://github.com/LiberTEM/LiberTEM/issues/105\u003e`_. Back-end support for live data processing\nis still experimental as well, see https://github.com/LiberTEM/LiberTEM-live.\n\nApplications\n------------\n\nSince LiberTEM is programmable through `user-defined functions (UDFs)\n\u003chttps://libertem.github.io/LiberTEM/udf.html\u003e`_, it can be used for a wide\nrange of processing tasks on array-like data and data streams. The following\napplications have been implemented already:\n\n- Virtual detectors (virtual bright field, virtual HAADF, center of mass\n  \\[`Krajnak et al. (2016) \u003chttps://doi.org/10.1016/j.ultramic.2016.03.006\u003e`_\\],\n  custom shapes via masks)\n- `Analysis of amorphous materials \u003chttps://libertem.github.io/LiberTEM/app/amorphous.html\u003e`_\n- `Strain mapping \u003chttps://libertem.github.io/LiberTEM-blobfinder/\u003e`_\n- `Off-axis electron holography reconstruction \u003chttps://libertem.github.io/LiberTEM-holo/\u003e`_\n- `Single Side Band ptychography \u003chttps://ptychography-4-0.github.io/ptychography/\u003e`_\n\nSome of these applications are available through an `interactive web GUI\n\u003chttps://libertem.github.io/LiberTEM/usage.html#gui-usage\u003e`_. Please see `the\napplications section \u003chttps://libertem.github.io/LiberTEM/applications.html\u003e`_\nof our documentation for details!\n\nThe Python API and user-defined functions (UDFs) can be used for complex\noperations such as arbitrary linear operations and other features like data\nexport. Example Jupyter notebooks are available in the `examples directory\n\u003chttps://github.com/LiberTEM/LiberTEM/tree/master/examples\u003e`_. If you are having\ntrouble running the examples, please let us know by filing an issue or\nby `joining our Gitter chat \u003chttps://gitter.im/LiberTEM/Lobby\u003e`_.\n\nLiberTEM is suitable as a high-performance processing backend for other\napplications, including live data streams. `Contact us\n\u003chttps://gitter.im/LiberTEM/Lobby\u003e`_ if you are interested!\n\nLiberTEM is evolving rapidly and prioritizes features following user demand and\ncontributions. Currently we are working on `live data processing\n\u003chttps://github.com/LiberTEM/LiberTEM-live\u003e`_, improving application support for sparse\ndata and event-based detectors, performance improvements for GPU processing, and implementing\nanalysis methods for various applications of pixelated\nSTEM and other large-scale detector data. If you like to influence the direction\nthis project is taking, or if you'd like to `contribute\n\u003chttps://libertem.github.io/LiberTEM/contributing.html\u003e`_, please join our\n`gitter chat \u003chttps://gitter.im/LiberTEM/Lobby\u003e`_ and our `general mailing list\n\u003chttps://groups.google.com/forum/#!forum/libertem\u003e`_.\n\nFile formats\n------------\n\nLiberTEM currently opens most file formats used for pixelated STEM. See `our\ngeneral information on loading data\n\u003chttps://libertem.github.io/LiberTEM/formats.html\u003e`_ and `format-specific\ndocumentation\n\u003chttps://libertem.github.io/LiberTEM/reference/dataset.html#formats\u003e`_ for more\ninformation!\n\n- Raw binary files\n- NumPy .npy binary files\n- Thermo Fisher EMPAD detector \\[`Tate et al. (2016) \u003chttps://doi.org/10.1017/S1431927615015664\u003e`_\\] files\n- `Quantum Detectors MIB format \u003chttps://quantumdetectors.com/products/merlinem/\u003e`_\n- Nanomegas .blo block files\n- Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for `DE-Series \u003chttps://directelectron.com/de-series-cameras/\u003e`_ detectors\n- `Gatan K2 IS \u003chttps://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera\u003e`_ raw format\n- Stacks of Gatan DM3 and DM4 files (via `openNCEM \u003chttps://github.com/ercius/openNCEM\u003e`_)\n- Single-file Gatan DM4 scans when saved using C-ordering\n- FRMS6 from PNDetector pnCCD cameras \\[`Simson et al. (2015) \u003chttps://doi.org/10.1017/s1431927615011836\u003e`_\\]\n  (currently alpha, gain correction still needs UI changes)\n- FEI SER files (via `openNCEM \u003chttps://github.com/ercius/openNCEM\u003e`_)\n- MRC (via `openNCEM \u003chttps://github.com/ercius/openNCEM\u003e`_)\n- HDF5-based formats such as HyperSpy files, NeXus and EMD\n- TVIPS binary files\n- Sparse data in Raw CSR (compressed sparse row) format, as is possible\n  to generate from event-based detectors\n- Please contact us if you are interested in support for an additional format!\n\nLive processing and detectors (experimental)\n--------------------------------------------\n\nSee `LiberTEM-live \u003chttps://libertem.github.io/LiberTEM-live/\u003e`_!\n\nLicense\n-------\n\nLiberTEM is licensed under the MIT license.\n\nAcknowledgements\n----------------\n\nWe are very grateful for your continuing support for LiberTEM!\n\nSee `the acknowledgement page\n\u003chttps://libertem.github.io/acknowledgements.html\u003e`_ for a list of authors and\ncontributors to LiberTEM and its subprojects. See also our info on `funding\n\u003chttps://libertem.github.io/#funding\u003e`_ and `industry partners\n\u003chttps://libertem.github.io/#industry-partners\u003e`_.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibertem%2Flibertem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flibertem%2Flibertem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibertem%2Flibertem/lists"}