{"id":29583796,"url":"https://github.com/humanloop/humanloop-export","last_synced_at":"2025-10-27T14:17:52.812Z","repository":{"id":300774999,"uuid":"997396648","full_name":"humanloop/humanloop-export","owner":"humanloop","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-23T14:05:25.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-23T15:24:16.135Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/humanloop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-06T13:08:06.000Z","updated_at":"2025-06-23T14:05:28.000Z","dependencies_parsed_at":"2025-06-23T15:36:26.153Z","dependency_job_id":null,"html_url":"https://github.com/humanloop/humanloop-export","commit_stats":null,"previous_names":["humanloop/humanloop-export"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/humanloop/humanloop-export","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Fhumanloop-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Fhumanloop-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Fhumanloop-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Fhumanloop-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/humanloop","download_url":"https://codeload.github.com/humanloop/humanloop-export/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Fhumanloop-export/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266042415,"owners_count":23867962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-19T23:38:57.634Z","updated_at":"2025-10-27T14:17:52.723Z","avatar_url":"https://github.com/humanloop.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Humanloop Export Tool\n\nA Python script to export data from Humanloop, including Files, Versions, Logs, and Evaluations.\n\n## Prerequisites\n\n- [uv](https://docs.astral.sh/uv/)\n- Humanloop API key\n\nThe script can also be run without `uv` by installing the dependencies manually.\nThe dependencies required can be found in the comments at the top of the script.\nIf doing so, ensure you're using Python 3.10 or higher.\n\nIf you don't have a Humanloop API key, you can create one in the [Humanloop UI](https://app.humanloop.com/account/api-keys).\n\n## Setup\n\nUpdate your `.env` file with the following:\n\n### `HUMANLOOP_API_KEY`\n\nSet `HUMANLOOP_API_KEY` to your Humanloop API key (more details in previous section).\n\n### `HUMANLOOP_DIRECTORY_ID` (optional)\n\nThe `HUMANLOOP_DIRECTORY_ID` variable is optional. Specify this (e.g. `dir_...`) to only\nexport Files within that directory (and all subdirectories). If not specified, all Files\nwill be exported.\n\n### `EXPORT_LOGS` (optional)\n\nIf you have fewer than 1 million Logs, you can set the `EXPORT_LOGS` environment variable to `true` to export Logs.\nBy default, the script will not export Logs.\n\nIf you have more than 1 million Logs and wish to export them, contact us and we can arrange a custom export of your Logs via S3.\n\nE.g.\n\n```env\nHUMANLOOP_API_KEY=hl_sk_4ea7b2d77f532c13af94420c0329c2c012b0cfad6dd9b8ac\nHUMANLOOP_DIRECTORY_ID=dir_xvNi1CKM872aETtBU6SBF\nEXPORT_LOGS=true\n```\n\n## Usage\n\nRun the export script:\n\n```bash\nuv run export.py\n```\n\nBy default, the script will create a new export, exporting all Files.\nYou should see a progress bar indicating the number of Files processed.\nIf you have Files with more than 100k Logs, the script may take a while to complete.\n\n## Output Structure\n\nThe script creates an `exports` directory with timestamped subdirectories.\nEvery time the script is run, it will create a new timestamped directory in the `exports` directory.\n\nEach export contains:\n\n```\nexports/\n└── \u003ctimestamp\u003e/  # YYYYMMDD-HHMMSS format\n    └── export.log  # Logging for the export script\n    └── files/\n        └── \u003cfile_id\u003e/  # E.g. pr_...\n            ├── file.json  # File metadata\n            ├── deployments.json  # List of Versions deployed to Environments\n            ├── versions/\n            │   └── \u003cversion_id\u003e  # E.g. prv_... Version definition (JSON)\n            ├── logs/  # (Only if EXPORT_LOGS is set to true)\n            │   └── 1.jsonl, 2.jsonl, ...  # Each file contains up to 1000 Logs; each line contains a single Log\n            ├── evaluations/\n            │   └── \u003cevaluation_id\u003e/  # E.g. evr_...\n            │       ├── evaluation.json  # Evaluation metadata - E.g. Name, Evaluators\n            │       ├── stats.json  # Evaluation statistics\n            │       ├── runs/\n            │       │   └── \u003crun_id\u003e  # E.g. run_... Run definition, e.g. the dataset and version (JSON)\n            │       └── logs/  # (Only if EXPORT_LOGS is set to true)\n            │           └── 1.jsonl, 2.jsonl, ...  # Each line corresponds to a Log, associated with a Run ID and optional Datapoint, along with the Evaluator Logs containing judgments.\n            └── datapoints/  # (Only for Datasets)\n                └── \u003cversion_id\u003e/  # E.g. dsv_...\n                    └── 1.jsonl, 2.jsonl, ...  # Each file contains up to 1000 Datapoints; each line corresponds to a Datapoint\n```\n\n## Notes\n\n### Request Errors and Retries\n\nThe script retries requests to the Humanloop API if they fail, up to 5 times.\nThese retries, along with errors, are logged to the `export.log` file.\n\n### Resuming an Export\n\nTo resume an existing export that had been previously interrupted, you can specify the export directory\nto resume from within `export.py`. E.g.\n\n```python\n... existing code ...\n\nif __name__ == \"__main__\":\n    ... existing code ...\n    main(\n        directory_id=directory_id,\n        export_dir=Path(\"exports/20250606-120000\"),  # Pass in a directory here to continue a previous export.\n    )\n```\n\nThis will skip any Files that have already been exported.\n\n## Importing data\n\nWe also provide a script to import data, given a prior export.\n\n```bash\nuv run import.py\n```\n\nThis will import the data into Humanloop.\n\nThe script will create a new import directory in the `imports` directory, named with a timestamp.\nThis new import directory will contain logs of the import process (in `import.log`).\n\nHumanloop will be populated with the Files and Versions from the export.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanloop%2Fhumanloop-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumanloop%2Fhumanloop-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanloop%2Fhumanloop-export/lists"}