{"id":18733398,"url":"https://github.com/icij/datashare-tarentula","last_synced_at":"2025-04-12T18:31:37.272Z","repository":{"id":41880372,"uuid":"203596621","full_name":"ICIJ/datashare-tarentula","owner":"ICIJ","description":"Cli toolbelt for Datashare.","archived":false,"fork":false,"pushed_at":"2024-11-28T17:39:29.000Z","size":958,"stargazers_count":6,"open_issues_count":9,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-09T09:45:02.897Z","etag":null,"topics":["cli","csv","datashare","elasticsearch","python"],"latest_commit_sha":null,"homepage":"https://datashare.icij.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ICIJ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-21T14:02:54.000Z","updated_at":"2024-06-18T09:11:58.000Z","dependencies_parsed_at":"2024-02-20T19:25:19.493Z","dependency_job_id":"4587b456-9cb5-4c8d-acdc-ab82ffa1b842","html_url":"https://github.com/ICIJ/datashare-tarentula","commit_stats":{"total_commits":379,"total_committers":6,"mean_commits":"63.166666666666664","dds":0.4854881266490765,"last_synced_commit":"e2f0cb9c5efa4047cea2e936ecb6a3397b411265"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ICIJ%2Fdatashare-tarentula","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ICIJ%2Fdatashare-tarentula/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ICIJ%2Fdatashare-tarentula/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ICIJ%2Fdatashare-tarentula/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ICIJ","download_url":"https://codeload.github.com/ICIJ/datashare-tarentula/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248613484,"owners_count":21133522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","datashare","elasticsearch","python"],"created_at":"2024-11-07T15:09:35.861Z","updated_at":"2025-04-12T18:31:36.809Z","avatar_url":"https://github.com/ICIJ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Datashare Tarentula [![CircleCI](https://circleci.com/gh/ICIJ/datashare-tarentula.svg?style=svg)](https://circleci.com/gh/ICIJ/datashare-tarentula)\n\nCli toolbelt for [Datashare](https://datashare.icij.org).\n\n```\n     /      \\\n  \\  \\  ,,  /  /\n   '-.`\\()/`.-'\n  .--_'(  )'_--.\n / /` /`\"\"`\\ `\\ \\\n  |  |  \u003e\u003c  |  |\n  \\  \\      /  /\n      '.__.'\n\nUsage: tarentula [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --syslog-address      TEXT    localhost   Syslog address\n  --syslog-port         INTEGER 514         Syslog port\n  --syslog-facility     TEXT    local7      Syslog facility\n  --stdout-loglevel     TEXT    ERROR       Change the default log level for stdout error handler\n  --help                                    Show this message and exit\n  --version                                 Show the installed version of Tarentula\n\nCommands:\n  aggregate\n  count\n  clean-tags-by-query\n  download\n  export-by-query\n  list-metadata\n  tagging\n  tagging-by-query\n```\n\n---\n\u003c!-- TOC depthFrom:2 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 --\u003e\n\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Cookbook 👩‍🍳](#cookbook-)\n  - [Count](#count)\n  - [Clean Tags by Query](#clean-tags-by-query)\n  - [Download](#download)\n  - [Export by Query](#export-by-query)\n  - [Tagging](#tagging)\n    - [CSV formats](#csv-formats)\n  - [Tagging by Query](#tagging-by-query)\n  - [Aggregate](#aggregate)\n  - [Following your changes](#following-your-changes)\n- [Configuration File](#configuration-file)\n- [Testing](#testing)\n- [Releasing](#releasing)\n  - [1. Create a new release](#1-create-a-new-release)\n  - [2. Upload distributions on pypi](#2-upload-distributions-on-pypi)\n  - [3. Build and publish the Docker image](#3-build-and-publish-the-docker-image)\n  - [4. Push your changes on Github](#4-push-your-changes-on-github)\n\n\u003c!-- /TOC --\u003e\n---\n\n## Installation\n\nYou can insatll Datashare Tarentula with your favorite package manager:\n\n```\npip3 install --user tarentula\n```\n\nOr alternativly with Docker:\n\n```\ndocker run icij/datashare-tarentula\n```\n\n## Usage\n\nDatashare Tarentula comes with basic commands to interact with a Datashare instance (running locally or on a remote server). Primarily focus on bulk actions, it provides you with both a cli interface and a python API.\n\n### Cookbook 👩‍🍳\n\nTo learn more about how to use Datashare Tarentula with a list of examples, please refer to \u003ca href=\"./COOKBOOK.md\"\u003ethe Cookbook\u003c/a\u003e.\n\n### Count\n\nA command to just count the number of files matching a query.\n\n```\nUsage: tarentula count [OPTIONS]\n\nOptions:\n  --datashare-url           TEXT        Datashare URL\n  --datashare-project       TEXT        Datashare project\n  --elasticsearch-url       TEXT        You can additionally pass the Elasticsearch\n                                          URL in order to use scrollingcapabilities of\n                                          Elasticsearch (useful when dealing with a\n                                          lot of results)\n  --query                   TEXT        The query string to filter documents\n  --cookies                 TEXT        Key/value pair to add a cookie to each\n                                          request to the API. You can\n                                          separatesemicolons: key1=val1;key2=val2;...\n  --apikey                  TEXT        Datashare authentication apikey\n                                          in the downloaded document from the index\n  --traceback / --no-traceback          Display a traceback in case of error\n  --type [Document|NamedEntity]         Type of indexed documents to download\n  --help                                Show this message and exit\n```\n\n### Clean Tags by Query\n\nA command that uses Elasticsearch `update-by-query` feature to batch untag documents directly in the index.\n\n```\nUsage: tarentula clean-tags-by-query [OPTIONS]\n\nOptions:\n  --datashare-project       TEXT        Datashare project\n  --elasticsearch-url       TEXT        Elasticsearch URL which is used to perform\n                                          update by query\n  --cookies                 TEXT        Key/value pair to add a cookie to each\n                                          request to the API. You can\n                                          separatesemicolons: key1=val1;key2=val2;...\n  --apikey                  TEXT        Datashare authentication apikey\n  --traceback / --no-traceback          Display a traceback in case of error\n  --wait-for-completion / --no-wait-for-completion\n                                        Create a Elasticsearch task to perform the\n                                          updateasynchronously\n  --query                   TEXT        Give a JSON query to filter documents that\n                                          will have their tags cleaned. It can be\n                                          afile with @path/to/file. Default to all.\n  --help                                Show this message and exit\n```\n\n### Download\n\nA command to download all files matching a query.\n\n```\nUsage: tarentula download [OPTIONS]\n\nOptions:\n  --apikey TEXT                   Datashare authentication apikey\n  --datashare-url TEXT            Datashare URL\n  --datashare-project TEXT        Datashare project\n  --elasticsearch-url TEXT        You can additionally pass the Elasticsearch\n                                  URL in order to use scrollingcapabilities of\n                                  Elasticsearch (useful when dealing with a\n                                  lot of results)\n\n  --query TEXT                    The query string to filter documents\n  --destination-directory TEXT    Directory documents will be downloaded\n  --throttle INTEGER              Request throttling (in ms)\n  --cookies TEXT                  Key/value pair to add a cookie to each\n                                  request to the API. You can\n                                  separatesemicolons: key1=val1;key2=val2;...\n\n  --path-format TEXT              Downloaded document path template\n  --scroll TEXT                   Scroll duration\n  --source TEXT                   A comma-separated list of field to include\n                                  in the downloaded document from the index\n\n  -f, --from INTEGER              Passed to the search it will bypass the\n                                  first n documents\n  -l, --limit INTEGER             Limit the total results to return\n  --sort-by TEXT                  Field to use to sort results\n  --order-by [asc|desc]           Order to use to sort results\n  --once / --not-once             Download file only once\n  --traceback / --no-traceback    Display a traceback in case of error\n  --progressbar / --no-progressbar\n                                  Display a progressbar\n  --raw-file / --no-raw-file      Download raw file from Datashare\n  --type [Document|NamedEntity]   Type of indexed documents to download\n  --help                          Show this message and exit.\n```\n\n\n### Export by Query\n\nA command to export all files matching a query.\n\n```\nUsage: tarentula export-by-query [OPTIONS]\n\nOptions:\n  --apikey TEXT                   Datashare authentication apikey\n  --datashare-url TEXT            Datashare URL\n  --datashare-project TEXT        Datashare project\n  --elasticsearch-url TEXT        You can additionally pass the Elasticsearch\n                                  URL in order to use scrollingcapabilities of\n                                  Elasticsearch (useful when dealing with a\n                                  lot of results)\n\n  --query TEXT                    The query string to filter documents\n  --output-file TEXT              Path to the CSV file\n  --throttle INTEGER              Request throttling (in ms)\n  --cookies TEXT                  Key/value pair to add a cookie to each\n                                  request to the API. You can\n                                  separatesemicolons: key1=val1;key2=val2;...\n\n  --scroll TEXT                   Scroll duration\n  --source TEXT                   A comma-separated list of field to include\n                                  in the export\n\n  --sort-by TEXT                  Field to use to sort results\n  --order-by [asc|desc]           Order to use to sort results\n  --traceback / --no-traceback    Display a traceback in case of error\n  --progressbar / --no-progressbar\n                                  Display a progressbar\n  --type [Document|NamedEntity]   Type of indexed documents to download\n  -f, --from INTEGER              Passed to the search it will bypass the\n                                  first n documents\n  -l, --limit INTEGER             Limit the total results to return\n  --size INTEGER                  Size of the scroll request that powers the\n                                  operation.\n\n  --query-field / --no-query-field\n                                  Add the query to the export CSV\n  --help                          Show this message and exit.\n```\n\n\n### Tagging\n\nA command to batch tag documents with a CSV file.\n\n```\nUsage: tarentula tagging [OPTIONS] CSV_PATH\n\nOptions:\n  --datashare-url       TEXT        http://localhost:8080   Datashare URL\n  --datashare-project   TEXT        local-datashare         Datashare project\n  --throttle            INTEGER     0                       Request throttling (in ms)\n  --cookies             TEXT        _Empty string_          Key/value pair to add a cookie to each request to the API. You can separate semicolons: key1=val1;key2=val2;...\n  --apikey              TEXT        None                    Datashare authentication apikey\n  --traceback / --no-traceback                              Display a traceback in case of error\n  --progressbar / --no-progressbar                          Display a progressbar\n  --help                                                    Show this message and exit\n```\n\n#### CSV formats\n\nTagging with a `documentId` and `routing`:\n\n```csv\ntag,documentId,routing\nActinopodidae,l7VnZZEzg2fr960NWWEG,l7VnZZEzg2fr960NWWEG\nAntrodiaetidae,DWLOskax28jPQ2CjFrCo\nAtracidae,6VE7cVlWszkUd94XeuSd,vZJQpKQYhcI577gJR0aN\nAtypidae,DbhveTJEwQfJL5Gn3Zgi,DbhveTJEwQfJL5Gn3Zgi\nBarychelidae,DbhveTJEwQfJL5Gn3Zgi,DbhveTJEwQfJL5Gn3Zgi\n```\n\nTagging with a `documentUrl`:\n\n```csv\ntag,documentUrl\nMecicobothriidae,http://localhost:8080/#/d/local-datashare/DbhveTJEwQfJL5Gn3Zgi/DbhveTJEwQfJL5Gn3Zgi\nMicrostigmatidae,http://localhost:8080/#/d/local-datashare/iuL6GUBpO7nKyfSSFaS0/iuL6GUBpO7nKyfSSFaS0\nMigidae,http://localhost:8080/#/d/local-datashare/BmovvXBisWtyyx6o9cuG/BmovvXBisWtyyx6o9cuG\nNemesiidae,http://localhost:8080/#/d/local-datashare/vZJQpKQYhcI577gJR0aN/vZJQpKQYhcI577gJR0aN\nParatropididae,http://localhost:8080/#/d/local-datashare/vYl1C4bsWphUKvXEBDhM/vYl1C4bsWphUKvXEBDhM\nPorrhothelidae,http://localhost:8080/#/d/local-datashare/fgCt6JLfHSl160fnsjRp/fgCt6JLfHSl160fnsjRp\nTheraphosidae,http://localhost:8080/#/d/local-datashare/WvwVvNjEDQJXkwHISQIu/WvwVvNjEDQJXkwHISQIu\n```\n\n### Tagging by Query\n\nA command that uses Elasticsearch `update-by-query` feature to batch tag documents directly in the index.\n\nTo see an example of input file, refer to [this JSON](tests/fixtures/tags-by-content-type.json).\n\n```\nUsage: tarentula tagging-by-query [OPTIONS] JSON_PATH\n\nOptions:\n  --datashare-project       TEXT        Datashare project\n  --elasticsearch-url       TEXT        Elasticsearch URL which is used to perform\n                                          update by query\n  --throttle                INTEGER     Request throttling (in ms)\n  --cookies                 TEXT        Key/value pair to add a cookie to each\n                                          request to the API. You can\n                                          separatesemicolons: key1=val1;key2=val2;...\n  --apikey                  TEXT        Datashare authentication apikey\n  --traceback / --no-traceback          Display a traceback in case of error\n  --progressbar / --no-progressbar      Display a progressbar\n  --wait-for-completion / --no-wait-for-completion\n                                        Create a Elasticsearch task to perform the\n                                          updateasynchronously\n  --help                                Show this message and exit\n```\n\n\n### List Metadata\n\nYou can list the metadata from the mapping, optionally counting the number of occurrences of each field in the index, with the `--count` parameter. Counting the fields is disabled by default.\n\nIt includes a `--filter_by` parameter to narrow retrieving metadata properties of specific sets of documents. For instance it can be used to get just emails related properties with: `--filter_by \"contentType=message/rfc822\"`\n\n```\n$ tarentula list-metadata --help\nUsage: tarentula list-metadata [OPTIONS]\n\nOptions:\n  --datashare-project TEXT       Datashare project\n  --elasticsearch-url TEXT       You can additionally pass the Elasticsearch\n                                 URL in order to use scrollingcapabilities of\n                                 Elasticsearch (useful when dealing with a lot\n                                 of results)\n  --type [Document|NamedEntity]  Type of indexed documents to get metadata\n  --filter_by TEXT               Filter documents by pairs concatenated by\n                                 coma of field names and values separated by\n                                 =.Example \"contentType=message/rfc822,content\n                                 Type=message/rfc822\"\n  --count / --no-count           Count or not the number of docs for each\n                                 property found\n\n  --help                         Show this message and exit.\n\n```\n\n### Aggregate\n\nYou can run aggregations on the data, the ElasticSearch aggregations API is partially enabled with this command.\nThe possibilities are:\n\n- count: grouping by a given field different values, and count the num of docs.\n- nunique: returns the number of unique values of a given field.\n- date_histogram: returns counting of monthly or yearly grouped values for a given date field.\n- sum: returns the sum of values of number type fields.\n- min: returns the min of values of number type fields.\n- max: returns the max of values of number type fields.\n- avg: returns the average of values of number type fields.\n- stats: returns a bunch of statistics for a given number type fields.\n- string_stats: returns a bunch of string statistics for a given string type fields.\n\n\n\n```\n$ tarentula aggregate --help\nUsage: tarentula aggregate [OPTIONS]\n\nOptions:\n  --apikey TEXT                   Datashare authentication apikey\n  --datashare-url TEXT            Datashare URL\n  --datashare-project TEXT        Datashare project\n  --elasticsearch-url TEXT        You can additionally pass the Elasticsearch\n                                  URL in order to use scrollingcapabilities of\n                                  Elasticsearch (useful when dealing with a\n                                  lot of results)\n  --query TEXT                    The query string to filter documents\n  --cookies TEXT                  Key/value pair to add a cookie to each\n                                  request to the API. You can\n                                  separatesemicolons: key1=val1;key2=val2;...\n  --traceback / --no-traceback    Display a traceback in case of error\n  --type [Document|NamedEntity]   Type of indexed documents to download\n  --group_by TEXT                 Field to use to aggregate results\n  --operation_field TEXT          Field to run the operation on\n  --run [count|nunique|date_histogram|sum|stats|string_stats|min|max|avg]\n                                  Operation to run\n  --calendar_interval [year|month]\n                                  Calendar interval for date histogram\n                                  aggregation\n  --help                          Show this message and exit.\n```\n\n### Following your changes\n\nWhen running Elasticsearch changes on big datasets, it could take a very long time. As we were curling ES to see if the task was still running well, we added a small utility to follow the changes. It makes a live graph of a provided ES indicator with a specified filter.\n\nIt uses [mathplotlib](https://matplotlib.org/) and python3-tk.\n\nIf you see the following message :\n\n```\n$ graph_es\ngraph_realtime.py:32: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure\n```\n\nThen you have to install [tkinter](https://docs.python.org/3/library/tkinter.html), i.e. python3-tk for Debian/Ubuntu.\n\nThe command has the options below:\n\n```\n$ graph_es --help\nUsage: graph_es [OPTIONS]\n\nOptions:\n  --query               TEXT        Give a JSON query to filter documents. It can be\n                                      a file with @path/to/file. Default to all.\n  --index               TEXT        Elasticsearch index (default local-datashare)\n  --refresh-interval    INTEGER     Graph refresh interval in seconds (default 5s)\n  --field               TEXT        Field value to display over time (default \"hits.total\")\n  --elasticsearch-url   TEXT        Elasticsearch URL which is used to perform\n                                      update by query (default http://elasticsearch:9200)\n```\n\n## Configuration File\n\nTarentula supports several sources for configuring its behavior, including an ini files and command-line options.\n\nConfiguration file will be searched for in the following order (use the first file found, all others are ignored):\n\n  * `TARENTULA_CONFIG` (environment variable if set)\n  * `tarentula.ini` (in the current directory)\n  * `~/.tarentula.ini` (in the home directory)\n  * `/etc/tarentula/tarentula.ini`\n\nIt should follow the following format (all values bellow are optional):\n\n```\n[DEFAULT]\napikey = SECRETHALONOPROCTIDAE\ndatashare_url = http://here:8080\ndatashare_project = local-datashare\n\n[logger]\nsyslog_address = 127.0.0.0\nsyslog_port = 514\nsyslog_facility = local7\nstdout_loglevel = INFO\n```\n\n## Testing\n\nTo test this tool, you must have Datashare and Elasticsearch running on your development machine.\n\nAfter you [installed Datashare](https://datashare.icij.org/), just run it with a test project/user:\n\n```\ndatashare -p test-datashare -u test\n```\n\nIn a separate terminal, install the development dependencies:\n\n```\nmake install\n```\n\nFinally, run the test\n\n```\nmake test\n```\n\n\n## Releasing\n\nThe releasing process uses [bumpversion](https://pypi.org/project/bumpversion/) to manage versions of this package, [pypi](https://pypi.org/project/tarentula/) to publish the Python package and [Docker Hub](https://hub.docker.com/) for the Docker image.\n\n### 1. Create a new release\n\n```\nmake [patch|minor|major]\n```\n\n### 2. Upload distributions on pypi\n\n_To be able to do this, you will need to be a maintainer of the [pypi](https://pypi.org/project/tarentula/) project._\n\n```\nmake distribute\n```\n\n### 3. Build and publish the Docker image\n\nTo build and upload a new image on the [docker repository](https://hub.docker.com/repository/docker/icij/datashare-tarentula) :\n\n_To be able to do this, you will need to be part of the ICIJ organization on docker_\n\n```\nmake docker-publish\n```\n\n**Note**: Datashare Tarentula is a multi-platform build. You might need to setup your environment for \nmulti-platform using the `make docker-setup-multiarch` command. Read more \n[on Docker documentation](https://docs.docker.com/build/building/multi-platform/). \n\n### 4. Push your changes on Github\n\nGit push release and tag :\n\n```\ngit push origin master --tags\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficij%2Fdatashare-tarentula","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ficij%2Fdatashare-tarentula","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficij%2Fdatashare-tarentula/lists"}