{"id":36876979,"url":"https://github.com/arthurheitmann/arctic_shift","last_synced_at":"2026-02-07T20:03:33.021Z","repository":{"id":185850375,"uuid":"674139399","full_name":"ArthurHeitmann/arctic_shift","owner":"ArthurHeitmann","description":"Making Reddit data accessible to researchers, moderators and everyone else.  Interact with the data through large dumps, an API or web interface.","archived":false,"fork":false,"pushed_at":"2026-01-27T12:42:58.000Z","size":2688,"stargazers_count":721,"open_issues_count":7,"forks_count":49,"subscribers_count":15,"default_branch":"master","last_synced_at":"2026-01-28T00:29:49.896Z","etag":null,"topics":["archived","pushshift","reddit"],"latest_commit_sha":null,"homepage":"https://arctic-shift.photon-reddit.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArthurHeitmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-03T08:25:15.000Z","updated_at":"2026-01-27T14:59:32.000Z","dependencies_parsed_at":"2024-01-19T21:27:14.257Z","dependency_job_id":"092607f8-6629-4320-b503-41bdef912707","html_url":"https://github.com/ArthurHeitmann/arctic_shift","commit_stats":null,"previous_names":["arthurheitmann/arctic_shift"],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/ArthurHeitmann/arctic_shift","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurHeitmann%2Farctic_shift","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurHeitmann%2Farctic_shift/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurHeitmann%2Farctic_shift/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurHeitmann%2Farctic_shift/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArthurHeitmann","download_url":"https://codeload.github.com/ArthurHeitmann/arctic_shift/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurHeitmann%2Farctic_shift/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29206843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T17:44:10.191Z","status":"ssl_error","status_checked_at":"2026-02-07T17:44:07.936Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archived","pushshift","reddit"],"created_at":"2026-01-12T15:12:23.257Z","updated_at":"2026-02-07T20:03:33.016Z","avatar_url":"https://github.com/ArthurHeitmann.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project Arctic Shift\n\nMaking Reddit data accessible to researchers, moderators and everyone else.\n\nInteract with the data through large dumps, an API or web interface.\n\n## Downloads\n\nAll download links are organized [here](./download_links.md). Once a new dump is available, it will\nalso be added on the [releases page](https://github.com/ArthurHeitmann/arctic_shift/releases).\n\nAlternatively for downloading data of users or smaller subreddits, you can use [this tool](https://arctic-shift.photon-reddit.com/download-tool).\n\nFor information on how the data was collected and modified, see [here](./file_content_explanations.md).\n\n## API\n\nDepending on your use case, you can try my (limited) [API](./api). For manual queries, you can use [this tool](https://arctic-shift.photon-reddit.com/search).\n\n## Usage\n\nFirst download one more dumps from the above links.\n\nGenerally I'd recommend to work with the compressed files instead of unpacking them. Unless of\ncourse you have seemingly infinite disk space.\n\nWith the helper scripts in this repository you can quickly get started.\n\n(You need at least Python 3.10)\n\n1. Clone this repository and its submodules\n\n```bash\ngit clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git\ncd arctic_shift\n```\n\n2. Install the `zstandard` library\n\n```bash\npip install zstandard\n```\n\n3. Open [scripts/processFiles.py](scripts/processFiles.py) in your editor. That script can process .zst, .zst_blocks, .jsonl/ndjson and .json files.\n\n4. Enter the path to a file or folder in `fileOrFolderPath` (since it is a raw string, you don't have to escape\n   backslashes). If you enter a folder, all files in that folder will be processed.\n\n5. Add your code to the `processFile` function.\n\n6. Run the file and be (very) patient.\n\n## Contact \u0026 Removal requests\n\nRemoval requests and generic support requests can be submitted [here](https://docs.google.com/forms/d/e/1FAIpQLSfzkmE8Bg6K_xii7aRm66ljzvo2tR59lTsdJ99acW4WX786Vw/viewform?usp=sf_link).\nTo check if your data is in the dataset, search for your username [here](https://arctic-shift.photon-reddit.com/search).\n\nRemoval forms of other archives: [Pushshift](https://docs.google.com/forms/d/1JSYY0HbudmYYjnZaAMgf2y_GDFgHzZTolK6Yqaz6_kQ) | [PullPush](https://removals.pullpush.io) | potentially [archive.org](https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org).\n\nIf you have questions, you can DM me on [discord](https://discord.com/users/282513022734565377) (*raiderbv* if the link doesn't work) or [email](mailto:arctic.shift.contact@gmail.com\n).\nAlternatively open an issue or pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurheitmann%2Farctic_shift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthurheitmann%2Farctic_shift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurheitmann%2Farctic_shift/lists"}