{"id":23122807,"url":"https://github.com/folio-org/mod-data-export","last_synced_at":"2026-04-24T17:06:12.178Z","repository":{"id":37088494,"uuid":"233570035","full_name":"folio-org/mod-data-export","owner":"folio-org","description":"Exporting inventory records","archived":false,"fork":false,"pushed_at":"2026-04-07T20:19:05.000Z","size":2962,"stargazers_count":2,"open_issues_count":5,"forks_count":5,"subscribers_count":17,"default_branch":"master","last_synced_at":"2026-04-07T20:23:48.992Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/folio-org.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-01-13T10:30:51.000Z","updated_at":"2026-04-07T19:34:22.000Z","dependencies_parsed_at":"2023-10-02T08:48:01.294Z","dependency_job_id":"e89905a7-c09f-4f1e-988d-b39159e02ef1","html_url":"https://github.com/folio-org/mod-data-export","commit_stats":null,"previous_names":[],"tags_count":67,"template":false,"template_full_name":null,"purl":"pkg:github/folio-org/mod-data-export","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-data-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-data-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-data-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-data-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/folio-org","download_url":"https://codeload.github.com/folio-org/mod-data-export/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-data-export/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32232715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-17T07:30:35.243Z","updated_at":"2026-04-24T17:06:12.173Z","avatar_url":"https://github.com/folio-org.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mod-data-export\n\nCopyright (C) 2018-2023 The Open Library Foundation\n\nThis software is distributed under the terms of the Apache License,\nVersion 2.0. See the file \"[LICENSE](LICENSE)\" for more information.\n\n* [Introduction](#introduction)\n* [Compiling](#compiling)\n* [Docker](#docker)\n* [System requirements](#system-requirements)\n\n## Introduction\n\nFOLIO data export module.\n\n#### Important notes\n\n## LIMITATIONS OF THE MODULE\nFor current releases, S3-compatible file storage (using MinIO client) is supported\nto store the exported MARC files.\n\n## OTHER\nIn a multi-tenant cluster, each tenant data is stored in it's own folder(prefix) under one bucket. For example, if the tenants are tenant001, tenant002, and bucket name \"exportMarc\" below is the pattern used stored in the bucket\n/{tenantId}/{jobExecutionId}/{fileName}.mrc\n- exportMarc/tenant001/448ae575-daec-49c1-8041-d64c8ed8e5b1/testFile-20200116100503.mrc\n- exportMarc/tenant002/295e28b4-aea2-4458-9073-385a31e1da05/uc-20200116100503.mrc\n\n\nNote: There are plans to have hosting agnostic implementation available in 2020 Q2.\n\n## Compiling\n\n```\n   mvn install\n```\n\nSee that it says \"BUILD SUCCESS\" near the end.\n\n## Docker\n\nBuild the docker container with:\n\n```\n   docker build -t mod-data-export .\n```\n\nTest that it runs with:\n\n```\n   docker run -d -t -i -p 8081:8081 mod-data-export\n```\n\n## Installing the module\n\nFollow the guide of\n[Deploying Modules](https://github.com/folio-org/okapi/blob/master/doc/guide.md#example-1-deploying-and-using-a-simple-module)\nsections of the Okapi Guide and Reference, which describe the process in detail.\n\n## Deploying the module\n\nNext we need to deploy the module. There is a deployment descriptor in\n`target/DeploymentDescriptor.json`. It tells Okapi to start the module on 'localhost'.\n\nDeploy it via Okapi discovery:\n\n```\ncurl -w '\\n' -D - -s \\\n  -X POST \\\n  -H \"Content-type: application/json\" \\\n  -d @target/DeploymentDescriptor.json  \\\n  http://localhost:9130/_/discovery/modules\n```\n\nThen we need to enable the module for the tenant:\n\n```\ncurl -w '\\n' -X POST -D -   \\\n    -H \"Content-type: application/json\"   \\\n    -d @target/TenantModuleDescriptor.json \\\n    http://localhost:9130/_/proxy/tenants/\u003ctenant_name\u003e/modules\n```\n\n\nThis module requires default mapping and job profiles for the work. To load them, the `loadRefernce=true` tenant initialization parameter should be passed when installing the module :\n\n```\ncurl -w '\\n' -X POST -d '[ { \"id\": \"mod-data-export-\u003cmodule_version\u003e\", \"action\": \"enable\" } ]' http://localhost:9130/_/proxy/tenants/\u003ctenant_name\u003e/install?tenantParameters=loadReference%3Dtrue\n```\n\n## Storage configuration\nMinIO remote storage or Amazon S3 can be used as storage for generated files MARC files.\nThe storage is selected by specifying the url of S3-compatible storage by using ENV variable `AWS_URL`.\n`AWS_SDK` is used to specify client to communicate with storage.\nIt requires `true` in case if S3 usage or `false` in case if MinIO usage. By default it equals to `false`.\nIn addition,the following ENV variables can be specified: `AWS_REGION`, `AWS_BUCKET`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`.\n\n## Issue tracker\n\nSee project [MDEXP](https://issues.folio.org/browse/MDEXP)\nat the [FOLIO issue tracker](https://dev.folio.org/guidelines/issue-tracker/).\n\n## Additional information\n\n### Configure your own mapping rules\n\nTo configure your own rules, you need to add it to the [mod-configuration module](https://github.com/folio-org/mod-configuration).\n\"Code\" value to use in the request - \"RULES_OVERRIDE\".\n\"Value\" field represents the rules, create your rules in the proper format(the example of default data-export rules - https://github.com/folio-org/mod-data-export/blob/master/src/main/resources/rules/rulesDefault.json).\nConvert the rules to the String format (you can use online [converter](https://tools.knowledgewalls.com/jsontostring)). Put the result string to the \"value\" field in the request body.\nThe description of how to create a configuration in mod-configuration module - https://github.com/folio-org/mod-configuration/blob/master/README.md.\n`If there are rules in mod-configuration, and they are enabled, they always will be used for the mapping process for a given tenant.`\n\n## System requirements\n\nTesting was done with a mod-data-export docker image memory limit = 512 MiB.\n\n### Updated system requirements\nmemory: 844\n\nmemoryReservation: 768\n\ncpu: 1024\n\nMetaspaceSize=88m\n\nMaxMetaspaceSize=88m\n\nXmx=840m\n\n#### Default mapping profile\n\nFor default mapping profile, the max usage of memory is 270 MiB during the process of export 1 million records.\nThe average memory usage is ~200 MiB.\n\n#### Custom mapping profile with holdings and items\nFor Custom mapping profile, the max usage of memory is 250 MiB during the process of export 1 million records.\nThe average memory usage is ~190 MiB.\n\n#### MINIO storage memory settings for /data-export/export-all\nDepending on the total amount of records to be exported, the size of minio storage required for the successful export can be various.\nFor example, 1 output mrc file with 100k records occupies 115Mb of minio storage and for 8180068 instances\nit requires 82 files, so the total size of files will be 115 * 82 = 9.43Gb for only 1 export execution.\nThese results should be taken into account when setting up configuration for minio storage, especially for\nlarger data sets and parallel export.\n\n\n\nThe [raml-module-builder](https://github.com/folio-org/raml-module-builder) framework.\n\nOther [modules](https://dev.folio.org/source-code/#server-side).\n\nOther FOLIO Developer documentation is at [dev.folio.org](https://dev.folio.org/)\n\n### Environment variables\nThis module uses S3 storage for files. AWS S3 and Minio Server are supported for files storage.\nIt is also necessary to specify variable S3_IS_AWS to determine if AWS S3 is used as files storage. By default,\nthis variable is `false` and means that MinIO server is used as storage.\nThis value should be `true` if AWS S3 is used.\n\n| Name                                           | Default value              | Description                                |\n|:-----------------------------------------------|:---------------------------|:-------------------------------------------|\n| S3_URL                                         | http://127.0.0.1:9000/     | S3 url                                     |\n| S3_REGION                                      | -                          | S3 region                                  |\n| S3_BUCKET                                      | -                          | S3 bucket                                  |\n| S3_SUB_PATH                                    | -                          | S3 object key prefix (subfolder)           |\n| S3_ACCESS_KEY_ID                               | -                          | S3 access key                              |\n| S3_SECRET_ACCESS_KEY                           | -                          | S3 secret key                              |\n| S3_IS_AWS                                      | false                      | Specify if AWS S3 is used as files storage |\n| EXPORT_TMP_STORAGE                             | -                          | Volume to store exports files              |\n| PLATFORM                                       | okapi                      | Specifies if okapi or eureka platform      |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffolio-org%2Fmod-data-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffolio-org%2Fmod-data-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffolio-org%2Fmod-data-export/lists"}