{"id":19294596,"url":"https://github.com/web3-storage/fauna-dump","last_synced_at":"2025-04-22T08:30:34.581Z","repository":{"id":66220440,"uuid":"418514724","full_name":"web3-storage/fauna-dump","owner":"web3-storage","description":"Export data from fauna","archived":true,"fork":false,"pushed_at":"2021-10-18T15:36:55.000Z","size":69,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-24T00:28:08.330Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/web3-storage.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-18T13:30:34.000Z","updated_at":"2024-04-18T16:24:53.000Z","dependencies_parsed_at":"2023-03-17T12:00:55.454Z","dependency_job_id":null,"html_url":"https://github.com/web3-storage/fauna-dump","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Ffauna-dump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Ffauna-dump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Ffauna-dump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web3-storage%2Ffauna-dump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/web3-storage","download_url":"https://codeload.github.com/web3-storage/fauna-dump/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250205976,"owners_count":21392158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:38:51.009Z","updated_at":"2025-04-22T08:30:34.574Z","avatar_url":"https://github.com/web3-storage.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fauna-dump\n\nFetch all the docs from all the collections from the FaunaDB\n\n## Getting started\n\n- Install the deps with `npm i`\n- Create an `.env` file with your fauna key in `FAUNA_KEY=xxx`\n- Run it with `node index.js`\n\n**.env**\n```ini\nFAUNA_KEY=\u003cyour key here\u003e\n# FAUNA_POINT_IN_TIME=\"2021-10-18T15:10:10.575Z\"\n# FAUNA_PAGE_SIZE=1000\n```\n\nThe script will log progress to the console and write the data as new-line delimited JSON to the `/dump` dir, with one file per Collection.\n\n**example run**\n```console\n$ node index.js\n🗂 Found 15 collections\n✔ Upload 565 [565]\n✔ Deal 2254 [1000,1000,254]\n✔ Content 547 [547]\n✔ PinLocation 6 [6]\n✔ Pin 2558 [1000,1000,558]\n✔ AuthToken 68 [68]\n✔ User 44 [44]\n✔ content_deals 0 [0]\n✔ Metrics 0 [0]\n✔ AggregateEntry 1212 [1000,212]\n✔ Aggregate 382 [382]\n✔ PinRequest 88 [88]\n✔ Metric 9 [9]\n✔ PinSyncRequest 0 [0]\n✔ Backup 59 [59]\n⏱: 6.014s\n```\n\n### Alternate adventure\n\nWe explore using `fdm` _(deprecated)_ to extract the data too.\n\nsee: https://docs.fauna.com/fauna/v4/integrations/fdm/install\n\n```console\n# build the image\n$ docker build -t fdm .\n\n# run the imaage and mount a local folder to export things to.\n$ docker run -it --rm -v $(pwd)/dump:/dump fdm /bin/bash\n\n# run fdm!\n\u003e ./fdm -source key=*** -dest path=/dump\n2021-10-18.13.20.26\n===================\n\nFDM   1.14\n2020-04-10 16:30\nLoaded property file path: /usr/local/bin/fdm-1.14/fdm.props\n\nField Formatting\n================\n\u003c\u003cNone\u003e\u003e\n\nConfig File\n===========\n   load:                      data.json\n   host:                      cloud\n   policy.data:               COPY\n\nConnecting to: CLOUD\nConnection was successful.\nERROR insufficient privilages to read all roles.\nOutput to directory [/dump].\nStarting Load at 2021-10-18 13:20:29.\nReading data at 2021-10-18 13:20:29  Fauna Timestamp: 1634563229194000\nCollection Aggregate -\u003e Aggregate documents 382 size 0 KB  execution time:    0.226 sec  0.000 MB/sec  1690.3 docs/sec\nCollection AggregateEntry -\u003e AggregateEntry documents 1212 size 0 KB  execution time:    0.650 sec  0.000 MB/sec  1864.6 docs/sec\nCollection AuthToken -\u003e AuthToken documents 68 size 0 KB  execution time:    0.101 sec  0.000 MB/sec  673.3 docs/sec\nCollection Backup -\u003e Backup documents 59 size 0 KB  execution time:    0.066 sec  0.000 MB/sec  893.9 docs/sec\nCollection Content -\u003e Content documents 547 size 0 KB  execution time:    0.191 sec  0.000 MB/sec  2863.9 docs/sec\nCollection content_deals -\u003e content_deals documents 0 size 0 KB  execution time:    0.050 sec  0.000 MB/sec    0.0 docs/sec\nCollection Deal -\u003e Deal documents 2254 size 0 KB  execution time:    1.168 sec  0.000 MB/sec  1929.8 docs/sec\nCollection Metric -\u003e Metric documents 9 size 0 KB  execution time:    0.292 sec  0.000 MB/sec   30.8 docs/sec\nCollection Metrics -\u003e Metrics documents 0 size 0 KB  execution time:    0.047 sec  0.000 MB/sec    0.0 docs/sec\nCollection Pin -\u003e Pin documents 2558 size 0 KB  execution time:    1.177 sec  0.000 MB/sec  2173.3 docs/sec\nCollection PinLocation -\u003e PinLocation documents 6 size 0 KB  execution time:    0.127 sec  0.000 MB/sec   47.2 docs/sec\nCollection PinRequest -\u003e PinRequest documents 88 size 0 KB  execution time:    0.110 sec  0.000 MB/sec  800.0 docs/sec\nCollection PinSyncRequest -\u003e PinSyncRequest documents 0 size 0 KB  execution time:    0.040 sec  0.000 MB/sec    0.0 docs/sec\nCollection Upload -\u003e Upload documents 565 size 0 KB  execution time:    0.176 sec  0.000 MB/sec  3210.2 docs/sec\nCollection User -\u003e User documents 44 size 0 KB  execution time:    0.068 sec  0.000 MB/sec  647.1 docs/sec\nFDM finished at 2021-10-18 13:20:33.\n```\n\n\n## Which documents do we update\n\nThe set of Collections and fields that we update\n\n| Collections      | Source\n|-----------------|-----------------------\n| Content.dagSize | [addAggregateEntries.js](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/addAggregateEntries.js/#L71)\n| Deal            | [createOrUpdateDeal.js](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createOrUpdateDeal.js)\n| Metric          | [createOrUpdateMetric.js](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createOrUpdateMetric.js/#L43)\n| Pin.status      | [createOrUpdatePin.js#L79](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createOrUpdatePin.js#L79)\n| User.*          | [createOrUpdateUser.js/#L37](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createOrUpdateUser.js/#L37)\n| User.usedStorage | [createUpload.js/#L110](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createUpload.js/#L110)\n| Pin             | [createUpload.js/#L123](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/createUpload.js/#L123)\n| User.usedStorage, Upload.deleted | [deleteUserUpload.js/#L58](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/deleteUserUpload.js/#L58)\n| PinRequest.attempts | [incrementPinRequestAttempts.js/#L29](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/incrementPinRequestAttempts.js/#L29)\n| User.usedStorage | [incrementUserUsedStorage.js/#L29](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/incrementUserUsedStorage.js/#L29)\n| Upload.name | [renameUserUpload.js/#L48](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/renameUserUpload.js/#L48)\n| Content.dagSize | [updateContentDagSize.js/#L28](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/updateContentDagSize.js/#L28)\n| Pin.status | [updatePins.js/#L27](https://github.com/web3-storage/web3.storage/blob/main/packages/db/fauna/resources/Function/updatePins.js/#L27)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Ffauna-dump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweb3-storage%2Ffauna-dump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb3-storage%2Ffauna-dump/lists"}