{"id":13557384,"url":"https://github.com/DocNow/twarc","last_synced_at":"2025-04-03T11:31:55.894Z","repository":{"id":6368384,"uuid":"7605723","full_name":"DocNow/twarc","owner":"DocNow","description":"A command line tool (and Python library) for archiving Twitter JSON","archived":false,"fork":false,"pushed_at":"2023-10-31T23:24:06.000Z","size":2807,"stargazers_count":1377,"open_issues_count":49,"forks_count":255,"subscribers_count":33,"default_branch":"main","last_synced_at":"2025-03-27T20:08:42.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://twarc-project.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DocNow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2013-01-14T14:35:40.000Z","updated_at":"2025-03-21T19:37:05.000Z","dependencies_parsed_at":"2024-01-09T19:01:21.302Z","dependency_job_id":"faf9d966-d119-4528-bc32-4863efb3cf20","html_url":"https://github.com/DocNow/twarc","commit_stats":{"total_commits":1486,"total_committers":77,"mean_commits":19.2987012987013,"dds":"0.47644683714670255","last_synced_commit":"7420be3c185e1e37182bd8e6dc04daa2008e32f7"},"previous_names":[],"tags_count":162,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DocNow%2Ftwarc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DocNow%2Ftwarc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DocNow%2Ftwarc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DocNow%2Ftwarc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DocNow","download_url":"https://codeload.github.com/DocNow/twarc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246993113,"owners_count":20865939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:04:19.019Z","updated_at":"2025-04-03T11:31:50.876Z","avatar_url":"https://github.com/DocNow.png","language":"Python","funding_links":[],"categories":["Tools \u0026 Software","Python","others","常用 Python 工具","Third-Party Tools","Tools","Social Networks"],"sub_categories":["Acquisition","Twitter"],"readme":"# twarc\n\n[![DOI](https://zenodo.org/badge/7605723.svg)](https://zenodo.org/badge/latestdoi/7605723) [![Build Status](https://github.com/docnow/twarc/workflows/tests/badge.svg)](https://github.com/DocNow/twarc/actions/workflows/main.yml) [![Standard](https://img.shields.io/endpoint?url=https%3A%2F%2Ftwbadges.glitch.me%2Fbadges%2Fstandard)](https://developer.twitter.com/en/docs/twitter-api) [![Premium](https://img.shields.io/endpoint?url=https%3A%2F%2Ftwbadges.glitch.me%2Fbadges%2Fpremium)](https://developer.twitter.com/) [![v2](https://img.shields.io/endpoint?url=https%3A%2F%2Ftwbadges.glitch.me%2Fbadges%2Fv2)](https://developer.twitter.com/en/docs/twitter-api)\n\ntwarc is a command line tool and Python library for collecting and archiving Twitter JSON\ndata via the Twitter API. It has separate commands (twarc and twarc2) for working with the older\nv1.1 API and the newer v2 API and Academic Access (respectively).\n\n* Read the [documentation](https://twarc-project.readthedocs.io)\n* Ask questions here in [GitHub](https://github.com/DocNow/twarc/discussions), in [Slack](https://bit.ly/docnow-slack) or [Matrix](https://matrix.to/#/#docnow:matrix.org?via=matrix.org\u0026via=petrichor.me\u0026via=converser.eu)\n\ntwarc has been developed with generous support from the [Mellon Foundation](https://mellon.org/).\n\n## Contributing \n\nNew features are welcome and encouraged for twarc. However, to keep the core twarc library and command line tool sustainable we will look at new functionality with the following principles in mind:\n\n1. Purpose: twarc is for *collection* and *archiving* of Twitter data via the Twitter API.\n2. Sustainability: keeping the surface area of twarc and it's dependencies small enough to ensure high quality.\n3. Utility: what is exposed by twarc should be applicable to different people, projects and domains, and not specific use cases.\n4. API consistency: as much as sensible we aim to make twarc consistent with the Twitter API, and also aim to make twarc consistent with itself - so commands in core twarc should work similarly to each other, and twarc functionality should align towards the Twitter API.\n\nFor features and approaches that fall outside of this, twarc enables external packages to hook into the twarc2 command line tool via [click-plugins](https://github.com/click-contrib/click-plugins). This means that if you want to propose new functionality, you can create your own package without coordinating with core twarc.\n\n### Documentation\n\nThe documentation is managed at ReadTheDocs. If you would like to improve the documentation you can edit the Markdown files in `docs` or add new ones. Then send a pull request and we can add it.\n\nTo view your documentation locally you should be able to:\n\n    pip install -r requirements-mkdocs.txt\n    pip install -e .\n    mkdocs serve\n    open http://127.0.0.1:8000/\n\nIf you prefer you can create a page on the [wiki](https://github.com/docnow/twarc/wiki/) to workshop the documentation, and then when/if you think it's ready to be merged with the documentation create an [issue](https://github.com/docnow/twarc/issues). Please feel free to create whatever documentation is useful in the wiki area.\n\n### Code\n\nIf you are interested in adding functionality to twarc or fixing something that's broken here are the steps to setting up your development environment:\n\n    git clone https://github.com/docnow/twarc\n    cd twarc\n    pip install -r requirements.txt\n\nCreate a .env file that included Twitter App keys to use during testing:\n\n    BEARER_TOKEN=CHANGEME\n    CONSUMER_KEY=CHANGEME\n    CONSUMER_SECRET=CHANGEME\n    ACCESS_TOKEN=CHANGEME\n    ACCESS_TOKEN_SECRET=CHANGEME\n\nNow run the tests:\n\n    python setup.py test\n\nAdd your code and some new tests, and send a pull request!\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDocNow%2Ftwarc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDocNow%2Ftwarc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDocNow%2Ftwarc/lists"}