{"id":24698633,"url":"https://github.com/faradayio/pachyderm_large_file_test","last_synced_at":"2026-04-18T11:02:18.033Z","repository":{"id":136464833,"uuid":"79643779","full_name":"faradayio/pachyderm_large_file_test","owner":"faradayio","description":"Code to test Pachyderm's large file support","archived":false,"fork":false,"pushed_at":"2017-05-09T20:14:24.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-13T18:49:04.511Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/faradayio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-01-21T12:16:37.000Z","updated_at":"2018-09-24T17:22:00.000Z","dependencies_parsed_at":"2023-04-26T18:49:22.389Z","dependency_job_id":null,"html_url":"https://github.com/faradayio/pachyderm_large_file_test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/faradayio/pachyderm_large_file_test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faradayio%2Fpachyderm_large_file_test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faradayio%2Fpachyderm_large_file_test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faradayio%2Fpachyderm_large_file_test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faradayio%2Fpachyderm_large_file_test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/faradayio","download_url":"https://codeload.github.com/faradayio/pachyderm_large_file_test/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faradayio%2Fpachyderm_large_file_test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31966217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-27T04:29:33.889Z","updated_at":"2026-04-18T11:02:18.027Z","avatar_url":"https://github.com/faradayio.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Testing Pachyderm with very large files in `FILE` mode\n\nThis is a test case for [Pachyderm][] focusing on large files and commits,\nand avoiding the use of `BLOCK` mode.  It uses data from the [OPUS][]\nproject, which is a collection of multi-lingual texts that have been\n\"aligned\" at the sentence level.  Total data size roughly 60 GB at moment,\nincluding a single 33 GB input file.\n\nFor more information on where the data comes from and how to transfer it to\nS3, see [data/README.md](./data/README.md).  For the moment, we provide a\nversion of the data in `us-east-1`.\n\nOPUS is a great data set for all sorts of linguistics and translation\ntasks, though you usually need to massage into a more useful format for\nyour specific application first.\n\n[Pachyderm]: https://www.pachyderm.io/\n[OPUS]: http://opus.lingfil.uu.se/index.php\n\n## Cluster configuration\n\nKubernetes master and two minions, each with the following specs:\n\n- Docker: 1.12.3\n- OS: RancherOS v0.7.1 (4.4.24)\n- CPU: 2x2.49 GHz\n- RAM: 7.3 GiB\n- Disk: 469 GiB\n\nThis cluster was created using Rancher 1.3.2, with the fix for\n[rancher/rancher#7370][] applied.  The servers were created using the Rancher\nREST API using the following options:\n\n[rancher/rancher#7370]: https://github.com/rancher/rancher/issues/7370\n\n```typescript\n  const config = {\n    amazonec2Config: {\n      accessKey: process.env['RANCHER_AWS_ACCESS_KEY_ID'],\n      ami: 'ami-dfdff3c8',\n      deviceName: '/dev/sda1',\n      iamInstanceProfile: 'kubernetes',\n      instanceType: 'm3.large',\n      // Allocate a public address so the servers can easily\n      // access outside resources (like Docker Hub).\n      privateAddressOnly: false,\n      region: 'us-east-1',\n      retries: '5',\n      rootSize: '500',\n      secretKey: process.env['RANCHER_AWS_SECRET_ACCESS_KEY'],\n      securityGroup: ['rancher-machine'],\n      spotPrice: '0.50',\n      sshUser: 'rancher',\n      subnetId: '...',\n      // Never use public addresses to communicate with servers,\n      // because the security group will block most of it.\n      usePrivateAddress: true,\n      volumeType: 'gp2',\n      vpcId: '...',\n      zone: 'a'\n    },\n```\n\nHosts can also be added manually using the UI and these options.  For\ntesting purposes, you can set up the instance profile `kubernetes` with EBS\nattach/detach permissions:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ec2:AttachVolume\",\n        \"ec2:DetachVolume\"\n      ],\n      \"Resource\": \"arn:aws:ec2:us-east-1:YOUR-AWS-ID-HERE:instance/*\"\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ec2:AttachVolume\",\n        \"ec2:DetachVolume\"\n      ],\n      \"Resource\": \"arn:aws:ec2:us-east-1:YOUR-AWS-ID-HERE:volume/*\"\n    }\n  ]\n}\n```\n\n...and you can allow Rancher to create the `rancher-machine` security group\nwhen creating a server through the UI.\n\n## Running all tests automatically\n\nMake sure `pachctl` can see your cluster and `~/pfs` is mounted.  Then run:\n\n```sh\n./test.sh\n```\n\n## Test 1: Commiting a large file via S3 URL\n\nThis is a 33GB tarball stored on S3.\n\n```sh\npachctl create-repo eubookshop_s3\npachctl put-file eubookshop_s3 master EUbookshop0.2.tar.gz -c \\\n    -f s3://fdy-pachyderm-public-test-data/opus/EUbookshop0.2.tar.gz\n```\n\n**Result in local test:** The ingestion hung for a while and returned:\n\n```\nread tcp 10.42.131.61:55858-\u003e52.216.226.112:443: read: connection reset by peer\n```\n\nIt looks like `pachd` and `rethinkdb` may have broken:\n\n```\n$ kubectl get all\nNAME               READY     STATUS              RESTARTS   AGE\npo/etcd-4h30v      1/1       Running             0          1d\npo/pachd-dn1b6     0/1       Error               4          2m\npo/pachd-k8v2f     1/1       Unknown             7          1d\npo/rethink-11szv   0/1       ContainerCreating   0          2m\npo/rethink-mfjvn   1/1       Unknown             0          1d\n\nNAME         DESIRED   CURRENT   READY     AGE\nrc/etcd      1         1         1         1d\nrc/pachd     1         1         0         1d\nrc/rethink   1         1         0         1d\n\nNAME             CLUSTER-IP     EXTERNAL-IP   PORT(S)                                          AGE\nsvc/etcd         10.43.8.144    \u003cnone\u003e        2379/TCP,2380/TCP                                1d\nsvc/kubernetes   10.43.0.1      \u003cnone\u003e        443/TCP                                          1d\nsvc/pachd        10.43.37.253   \u003cnodes\u003e       650:30650/TCP,651:30651/TCP                      1d\nsvc/rethink      10.43.23.240   \u003cnodes\u003e       8080:32080/TCP,28015:32081/TCP,29015:30438/TCP   1d\n\nNAME              DESIRED   SUCCESSFUL   AGE\njobs/pachd-init   1         1            1d\n```\n\nNeither of the minions appears to be particularly low on disk space:\n\n```\nroot@5a7c7b3ebed6:/# df -h\nFilesystem      Size  Used Avail Use% Mounted on\noverlay         469G   65G  384G  15% /\ntmpfs           3.7G     0  3.7G   0% /dev\ntmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup\n/dev/xvda1      469G   65G  384G  15% /.r\nshm              64M     0   64M   0% /dev/shm\n```\n\n```\nroot@b8363e1a9fc6:/# df -h\nFilesystem      Size  Used Avail Use% Mounted on\noverlay         469G  4.9G  444G   2% /\ntmpfs           3.7G     0  3.7G   0% /dev\ntmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup\n/dev/xvda1      469G  4.9G  444G   2% /.r\nshm              64M     0   64M   0% /dev/shm\n```\n\n## Test 2: Commiting a large file via HTTP URL\n\nThis is similar to the above, except we use a explicit URL.  In practice,\nthis URL might be signed using `aws s3 presign`.\n\n## Test 3: Adding all files\n\nThis is intended to approximate a test of adding 50GB to 200GB of CSV data\nfrom a single source, spread out across varying numbers of multi-GB files.\nWe'll just try it with 60 GB for now.\n\n```sh\npachctl create-repo opus_tars\npachctl put-file opus_tars master -c -i URLS.txt\n```\n\n**Result in local test:** Hung for about 5 minutes, then `pachd version`\nstarted erroring, suggesting that the backend fell over again.\n\n## Test 4: Copy-through of all files\n\n(Tested successfully with a smaller input.)\n\nWe create a pipeline that copies everyting in `/pfs/opus_tars` to\n`/pfs/out` unchanged, using `FILE` mode and multiple worker containers.\nThis has been tested using a single, smaller `tar.gz` input file.\n\n## Test 5: Unpack tarballs, sort by language, repack\n\n(Tested successfully with a smaller input.)\n\nWe use two pipelines:\n\n1. `opus_unpack`: This unpacks all the tarballs in `opus_tars` and\n   organizes the raw data by language in `/pfs/out/en/`, `/pfs/out/es/`, etc.\n2. `opus_repack`: This takes `/pfs/opus_unpack/$LANG/` and repacks it as\n   `/pfs/out/$LANG.tar`.  We set `\"constant\": 2` but we only want to\n   produce a single, valid `en.tar` file!  This is a test of how `FILE`\n   reduction works with multiple parallel workers and files in\n   subdirectories, and it assumes the semantics I would natually imagine,\n   which may not be correct.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaradayio%2Fpachyderm_large_file_test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffaradayio%2Fpachyderm_large_file_test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaradayio%2Fpachyderm_large_file_test/lists"}