{"id":23727779,"url":"https://github.com/libratom/libratom","last_synced_at":"2025-04-07T05:15:07.627Z","repository":{"id":34726055,"uuid":"179343530","full_name":"libratom/libratom","owner":"libratom","description":"Python library and supporting utilities to parse and process PST and mbox email sources","archived":false,"fork":false,"pushed_at":"2024-06-18T00:03:11.000Z","size":795,"stargazers_count":106,"open_issues_count":14,"forks_count":17,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-30T21:14:24.721Z","etag":null,"topics":["email","ost","pst","python3"],"latest_commit_sha":null,"homepage":"https://github.com/libratom/libratom/wiki","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/libratom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-03T18:03:28.000Z","updated_at":"2025-03-21T11:03:30.000Z","dependencies_parsed_at":"2023-01-15T08:47:17.388Z","dependency_job_id":"abf8565f-c028-439a-8a17-43c1801020fe","html_url":"https://github.com/libratom/libratom","commit_stats":{"total_commits":672,"total_committers":4,"mean_commits":168.0,"dds":0.06994047619047616,"last_synced_commit":"6d1b54d4e6972786b2150b0872b895c1bedc54e1"},"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libratom%2Flibratom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libratom%2Flibratom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libratom%2Flibratom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libratom%2Flibratom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/libratom","download_url":"https://codeload.github.com/libratom/libratom/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247595335,"owners_count":20963943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["email","ost","pst","python3"],"created_at":"2024-12-31T01:30:15.395Z","updated_at":"2025-04-07T05:15:07.583Z","avatar_url":"https://github.com/libratom.png","language":"Python","funding_links":[],"categories":["Sending"],"sub_categories":["Inbound - Mail Parser"],"readme":"![Logo](https://github.com/libratom/ratom-logos/raw/main/basic_variations/RATOM_Vector_Logo_v1_300px.png)\n\n# libratom\n\n[![PyPI version](https://badge.fury.io/py/libratom.svg)](https://badge.fury.io/py/libratom)\n[![Build Status](https://github.com/libratom/libratom/actions/workflows/test_suite.yml/badge.svg?branch=main)](https://github.com/libratom/libratom/actions/workflows/test_suite.yml?query=branch%3Amain)\n[![codecov](https://codecov.io/gh/libratom/libratom/branch/main/graph/badge.svg)](https://codecov.io/gh/libratom/libratom)\n[![Maintainability](https://api.codeclimate.com/v1/badges/da1ce430d2d5fb3b548a/maintainability)](https://codeclimate.com/github/libratom/libratom/maintainability)\n[![Twitter Follow](https://img.shields.io/twitter/follow/RATOM_project.svg?style=social\u0026label=Follow)](https://twitter.com/RATOM_Project)\n\nPython library and supporting utilities to parse and process PST and mbox email sources.\n\n## Installation\n\nLibratom requires Python 3.8 or newer, and can be installed from the Python Package Index. Installing with **pip** will automatically install all required dependencies. These dependencies include a version of **libpff** that will be compiled automatically with C tooling during install. A selection of environments we have tested follows:\n\n*   Ubuntu 18.04LTS, 20.04LTS, and 22.04LTS releases require build-essential, python3, python3-pip, and python3-venv packages\n*   RHEL 9 requires python3, python3-pip, python3-devel, and the Development Tools package group.\n*   macOS 10.14 (and newer) releases require Xcode 11 (or newer), Xcode CLI tools, and Python 3 installed using Homebrew (or your preferred method)\n*   Windows 10/11 releases require Visual Studio Code, Build Tools for Visual Studio, and Python 3 installed using Anaconda 3 (or your preferred method)\n\nNeed guidance setting up an environment on your platform? Navigate to one of the linked sections below before continuing.\n\n*   [Windows environment setup](#windows-environment-setup)\n*   [macOS environment setup](#macos-environment-setup)\n*   [Ubuntu environment setup](#ubuntu-environment-setup)\n*   [RHEL environment setup](#rhel-environment-setup)\n\nWe **strongly recommend** you create a Python virtual environment prior to installing libratom. If you followed one of the guides above, you should already have one activated. With the environment configured and a Python virtual environment created and activated, run the following commands.\n\nMake sure pip is upgraded to the latest version:\n```shell\npip install --upgrade pip\n```\n\nInstall libratom:\n```shell\npip install libratom\n```\n\n## CLI Overview\n\nLibratom provides a command line interface to run several different tasks. To see available commands, type:\n\n```shell\n(venv) user@host:~$ ratom -h\n```\n\nFollow one of the section links below for detailed explanations of how the available commands work:\n\n*   [Entity extraction](#entity-extraction): Entity extraction from individual PST and mbox files, or directories containing PST and mbox files. Optionally write message bodies and headers to the database.\n*   [Model management](#model-management): Management tool for spaCy language models. Use to display available models and install specific model versions.\n*   [Scan and report](#scan-and-report): Quickly scan an email source and generate a report. Optionally write message bodies and headers to the database.\n*   [Message export](#message-export): Export selected messages from PST or mbox files as one .eml file per message.\n\n## Entity extraction\n\nTo see detailed help for the entity extraction command, type:\n\n```shell\n(venv) user@host:~$ ratom entities -h\n```\n\nTo run the extractor with default settings over a PST or mbox file, or a directory containing one or more PST and mbox files, type the following:\n\n```shell\n(venv) user@host:~$ ratom entities -p -m /path/to/PST-or-mbox-file-or-directory\n```\n\nNote that the -m flag has been added to this command to write message bodies (stripped of inline attachments and markup) and message headers to the message table. If the -m flag is omitted, these entries will be null. Progress (enabled using the -p flag) is displayed in a bar at the bottom of the window. To terminate a job early and shut down all workers, type Ctrl-C.\n\nBy default, the tool will install and use the latest spaCy 3 en\\_core\\_web\\_sm model (see the [model management](#model-management) section for how to list and install model versions). It will start as many concurrent jobs as there are virtual cores available. Entities are written to a sqlite3 file automatically named using the existing file or directory name and current datetime stamp, and with the following schema:\n\n![RATOM database schema](https://libratom.github.io/ratom-db-schema-v3.png)\n\nThe schema contains five tables. Four tables are used to represent files, messages, attachments, and entities. A fifth table is used to store configuration and environment details relevant to a specific run.\n\nIn the entity table, text is the entity instance, label\\_ is the entity type, filepath is the PST or mbox file associated with this entity. Full message and file information for each entity are also available through message_id and file_report_id respectively. Note that pff_identifier (a message ID specific to PST files) will not be populated for messages located in mbox files. Examples of how to query these tables can be found in the **Interactive examples** section near the end of this README.\n\n## Advanced uses of the entity extraction command\n\nThe CLI provides additional flags to tune performance, output location, and verbosity of the tool. Flags that do not take a value may be chained. For example, \"-p -v\" is equivalent to \"-pv\" Some example use cases are provided below.\n\nThe CLI is \"quiet\" and produces minimal output by default. A single -v flag enables some basic output about job status. To view more detailed output (for example, if you encounter unexpected failures), you can increase the level of output verbosity with -vv (verbosity level 2):\n\n```shell\n(venv) user@host:~$ ratom entities -p -m -vv /path/to/PST-or-mbox-file-or-directory\n```\n\nAll remaining examples are presented with verbosity level 1 enabled.\n\nTo use the latest version of a different entity model, use the --spacy-model flag. The following example directs the tool to use the multi-language model:\n\n```shell\n(venv) user@host:~$ ratom entities -p -m -v --spacy-model xx_ent_wiki_sm /path/to/PST-or-mbox-file-or-directory\n```\n\nThe tool will optimize the number of jobs that may be run concurrently on your system by default, using all available processor cores. To manually set the number of jobs that may be run concurrently, use the -j flag. The following example sets the number of concurrent jobs to 2:\n\n```shell\n(venv) user@host:~$ ratom entities -p -m -v -j 2 /path/to/PST-or-mbox-file-or-directory\n```\n\nTo change the name or location used for the sqlite3 output file, use the -o flag. Specifying a directory will result in the automatically named file being written to that path. Specifying a path that includes a filename will force the use of that filename. In the following example, the sqlite3 database will be named filename.db:\n\n```shell\n(venv) user@host:~$ ratom entities -p -m -v -o /path/to/directory/filename.db /path/to/PST-or-mbox-file-or-directory\n```\n\nAs noted earlier, some of these options can be chained for convenience. The following example specifies that progress will be shown, the second level of verbosity will be used, message headers and bodies (stripped of inline attachments and markup) will be written to the database, 4 jobs will be run concurrently and the sqlite3 database will be named filename.db:\n\n```shell\n(venv) user@host:~$ ratom entities -pvvm -j 4 -o /path/to/directory/filename.db /path/to/PST-or-mbox-file-or-directory\n```\n\n## Model management\n\nNew spaCy releases are generally accompanied by newly trained models. Using different versions of models over the same collection may produce different results. Depending on your workflow and needs, you may wish to install earlier versions of models, upgrade models that were previously installed, or install multiple models. The model command assists with these tasks.\n\nTo see detailed help for the model command, type:\n\n```shell\n(venv) user@host:~$ ratom model -h\n```\n\nTo see a list of available models, type:\n\n```shell\n(venv) user@host:~$ ratom model -l\n```\n\nTo install a specific version of an available model, use the -i and --version flags. For example, to install the 3.4.1 version of en\\_core\\_web\\_sm, type:\n\n```shell\n(venv) user@host:~$ ratom model -i en_core_web_sm --version 3.4.1\n```\n\nNote that a request to install a specific version will replace any existing version of that model, even if the existing version is newer.\n\n## Scan and report\n\nTo generate a report (file metadata, message count, and attachment metadata) from an email source (file or directory) in the same sqlite3 databse format as the entity extraction command without actually extracting any entities, use the report command.\n\nTo see detailed help for the report command, type:\n\n```shell\n(venv) user@host:~$ ratom report -h\n```\n\nAs an example, the following command generates a report (showing progress while running) from a single PST or MBOX file, or directory of files:\n\n```shell\n(venv) user@host:~$ ratom report -p /path/to/PST-or-mbox-file-or-directory\n```\n\nSimilar to the entities command, adding the -m flag will cause message bodies (stripped of inline attachments and markup) and headers to be written to the message table:\n\n```shell\n(venv) user@host:~$ ratom report -p -m /path/to/PST-or-mbox-file-or-directory\n```\n\n## Message export\n\nThe emldump command generates .eml files from selected messages in one or more PST files. The use case for this command is having to extract messages from multiple PST files fetched from cloud storage onto a local filesystem.\nTo see its detailed help type:\n\n```shell\n(venv) user@host:~$ ratom emldump -h\n```\n\nThe required input for emldump is the path to a JSON file listing the source PST files and for each PST file the identifiers of messages to be exported.\n\nA sample JSON input is given below:\n\n```shell\n(venv) user@host:~$ cat files.json\n[\n  {\n    \"filename\" : \"andy_zipper_000_1_1.pst\",\n    \"sha256\": \"7223f79c894f6911ca0fe9a105fa47da0edde31670590d45e2ddfdec37469c4d\",\n    \"id_list\": [\"2203588\", \"2203620\", \"2203652\"]\n  },\n  {\n   \"filename\" : \"andy_zipper_001_1_1.pst\",\n   \"sha256\": \"c10d0bb5ac6ee4386c7c990b22f0d337f8f31cfcb685bb1739dd2d8293043134\",\n   \"id_list\": [\"2133380\", \"2133412\", \"2133444\"]\n  }\n]\n```\n\nNote that the JSON file contains filenames only. When invoking the command you can specify where those files are via the `-l` or `--location` option. By default the tool will look for them in the current working directory.\n\nYou may also specify a base destination folder for the output .eml files via the `-o` or `--out` option. By default the tool will create them under the current working directory, grouped in directories named after each `filename`.\n\nFor example, the above input file can be used as follows:\n\n```shell\n(venv) user@host:~$ ratom emldump -l /tmp/libratom/test_data/RevisedEDRMv1_Complete/andy_zipper/ -o /tmp/eml/ files.json\n```\n\n\n## Interactive examples\n\nWe have prepared a selection of Jupyter notebooks to demonstrate core functionality and additional scripts built using libratom. The repository linked below includes instructions for deploying these notebooks in an executable environment using MyBinder:\n\n[https://github.com/libratom/ratom-notebooks](https://github.com/libratom/ratom-notebooks)\n\n## Environment Setups for Windows, macOS, and Ubuntu\n\n### Windows environment setup\n\nFirst, install Visual Studio Code. Visit \u003chttps://code.visualstudio.com/download\u003e to download and run the 64-bit User Installer for Windows 10. Follow the prompts, accepting all default selections.\n\nDownload and run the Build Tools for Visual Studio 2022 installer from \u003chttps://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022\u003e. Follow the prompts until you see a window with a Workloads tab in the top left hand corner.\n\nIn the Workloads tab, check the box for \"C++ build tools\". Click the Install button at the bottom right of the window. Once you see \"Installation Succeeded!\", close the window.\n\nVisit \u003chttps://www.anaconda.com/products/individual\u003e to download and install the 64-bit Python 3.8 or Python 3.9 Anaconda distribution. Find and double-click the downloaded executable and follow the prompts, accepting all default selections.\n\nOpen the Windows Start Menu, select Anaconda3 (64-bit) and click \"Anaconda Prompt (Anaconda3)\".\n\nIn the terminal that appears, create a new virtual environment in which to install libratom. Replace \"username\" in the prompt with your username. Do not replace the word \"name\" in the \"--name\" flag:\n\n```shell\n(base) C:\\Users\\username\u003econda create --name ratomenv\n```\n\nType y to confirm any prompts and proceed. Activate the environment:\n\n```shell\n(base) C:\\Users\\username\u003econda activate ratomenv\n```\n\nType y to confirm and proceed. At the next prompt, install pip:\n\n```shell\n(ratomenv) C:\\Users\\username\u003econda install pip\n```\n\nType y to confirm and proceed. At the next prompt, install libratom:\n\n```shell\n(ratomenv) C:\\Users\\username\u003epip install libratom\n```\n\nLibratom and the ratom CLI tool should now be ready to use.\n\nPython virtual environments can be deactivated and reactivated as needed. To deactivate the environment, type:\n\n```shell\n(ratomenv) C:\\Users\\username\u003econda deactivate\n```\n\nTo remove the environment completely, type:\n\n```shell\n(base) C:\\Users\\username\u003econda env remove -n ratomenv\n```\n\nNeed additional help getting started with conda? You can find a 20-minute starter guide at \u003chttps://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html\u003e.\n\n### macOS environment setup\n\nInstall the latest version of Xcode from the App Store. Once Xcode is installed, open a terminal (you can find the terminal app by clicking the Spotlight magnifying glass and typing term).\n\nRun the following to install/update the Xcode command line tools:\n\n```shell\nuser-macbook:~ user$ xcode-select --install\n```\n\nYou may need to run the following to agree to the Xcode/iOS licence (requires admin privileges):\n\n```shell\nuser-macbook:~ user$ sudo xcodebuild -license\n```\n\nFollow the instructions at the link below to check your system and install Python 3 if needed:\n\n\u003chttps://installpython3.com/mac/\u003e\n\nNext, create a new Python 3 virtual environment. Use the instructions in the previous link or create and activate one in your home directory with the following commands:\n\n```shell\nuser-macbook:~ user$ python3 -m venv venv\nuser-macbook:~ user$ source venv/bin/activate\n```\n\nFollow the remaining instructions in the Installation section at the top of this README to upgrade pip and install libratom.\n\n### Ubuntu environment setup\n\nTo install and test this software in a new Python virtual environment in Ubuntu 18.04LTS or newer:\n\nMake sure Python 3.8 or newer, python3-pip, python3-venv, and build-essential are installed. Open a terminal and type the following command:\n\n```shell\nsudo apt install build-essential python3 python3-pip python3-venv\n```\n\nCreate and activate a Python virtual environment:\n\n```shell\npython3 -m venv venv\nsource venv/bin/activate\n```\n\nFollow the remaining instructions in the Installation section at the top of this README to upgrade pip and install libratom.\n\n### RHEL environment setup\n\nThese instructions use the *DNF* package manager, and have been tested in RHEL 9. In RHEL 9, you will need to install the python3-pip and python3 devel packages. In a terminal, run the following commands:\n\n```shell\nsudo dnf install python3-pip\nsudo dnf install python3-devel\n```\n\nReply \"y\" if prompted during the installations. Next, you'll need to install some development tools, particularly a C compiler. There are several ways to do this, but the simplest is to install the Development Tools group:\n\n```shell\nsudo dnf group install \"Development Tools\"\n```\n\nCreate and activate a Python virtual environment:\n\n```shell\npython3 -m venv venv\nsource venv/bin/activate\n```\n\nOnce you have completed these installs, follow the remaining instructions at the top of this README to upgrade pip and install libratom.\n\n## License(s)\n\nLogos, documentation, and other non-software products of the RATOM team are distributed under the terms of Creative Commons 4.0 Attribution. Software items in RATOM repositories are distributed under the terms of the MIT License. See the LICENSE file for additional details.\n\n\u0026copy; 2022, The University of North Carolina at Chapel Hill.\n\n## Development Team and Support\n\nDeveloped by the RATOM team at the University of North Carolina at Chapel Hill.\n\nSee [https://ratom.web.unc.edu](https://ratom.web.unc.edu/) for additional project details, staff bios, and news.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibratom%2Flibratom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flibratom%2Flibratom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibratom%2Flibratom/lists"}