{"id":15359245,"url":"https://github.com/hugovk/pypi-tools","last_synced_at":"2025-04-15T07:22:36.565Z","repository":{"id":33683284,"uuid":"138175341","full_name":"hugovk/pypi-tools","owner":"hugovk","description":"Command-line Python scripts to do things with PyPI","archived":false,"fork":false,"pushed_at":"2025-04-07T20:28:58.000Z","size":443467,"stargazers_count":24,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-07T21:36:43.054Z","etag":null,"topics":["chart","csv","graph","json","pypi","python","python-script","tools","trends"],"latest_commit_sha":null,"homepage":"https://hugovk.github.io/pypi-tools","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hugovk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-21T13:40:30.000Z","updated_at":"2025-04-07T20:29:01.000Z","dependencies_parsed_at":"2025-04-07T21:37:29.457Z","dependency_job_id":null,"html_url":"https://github.com/hugovk/pypi-tools","commit_stats":{"total_commits":536,"total_committers":2,"mean_commits":268.0,"dds":0.04104477611940294,"last_synced_commit":"4b2847bb5e45eb3b53e2f6e5d3b4aa4536a00bd6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugovk%2Fpypi-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugovk%2Fpypi-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugovk%2Fpypi-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugovk%2Fpypi-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hugovk","download_url":"https://codeload.github.com/hugovk/pypi-tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249024213,"owners_count":21200062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chart","csv","graph","json","pypi","python","python-script","tools","trends"],"created_at":"2024-10-01T12:44:11.970Z","updated_at":"2025-04-15T07:22:36.552Z","avatar_url":"https://github.com/hugovk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pypi-tools\n\n[![Test](https://github.com/hugovk/pypi-tools/actions/workflows/test.yml/badge.svg)](https://github.com/hugovk/pypi-tools/actions/workflows/test.yml)\n[![Python: 3.9+](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://www.python.org/downloads/)\n[![Code style: Black](https://img.shields.io/badge/code%20style-Black-000000.svg)](https://github.com/psf/black)\n\nCommand-line Python scripts to do things with the\n[Python Package Index (PyPI)](https://pypi.org/).\n\n## pypi-trends.py\n\n`pypi-trends.py` is a wrapper around [pypinfo](https://github.com/ofek/pypinfo)\nand [pypistats](https://github.com/hugovk/pypistats) to fetch all monthly downloads from\nthe PyPI database on Google BigQuery and save them as JSON files.\n\nFor the examples below, data was downloaded over many days as getting all months uses up\na lot of free BigQuery quota.\n\n## jsons2img.py\n\n`jsons2img.py` converts the JSON files into a single CSV file for chart-wrangling\nin a spreadsheet, and can generate a chart using Matplotlib. Once an image exists, it\ncan be re-generated with `make -j4`.\n\n## Examples\n\nHere the pip installs for all packages from the Python Package Index (PyPI),\nfrom January 2016 onwards:\n\n![](images/all.png)\n\n[start_generated]: # (start_generated)\n### [urllib3](https://github.com/urllib3/urllib3)\n\nHTTP client\n\n![](images/urllib3.png)\n\n### [Requests](https://github.com/psf/requests)\n\nHTTP library\n\n![](images/requests.png)\n\n[end_generated]: # (end_generated)\n\n**[See more charts here.](charts.md)**\n\n### See also\n\n* [Python version share over time](https://medium.com/@hugovk/python-version-share-over-time-cf4498822650)\n\n## source-finder.py\n\nGiven a PyPI package, `source_finder.py` looks for the source repository in its metadata.\n\n```console\n$ python source_finder.py six\nhttps://github.com/benjaminp/six\n$ python source_finder.py urllib3\nNone\n```\n\nIt caches the JSON metadata downloaded from PyPI in a temporary directory, use the\n`--verbose` option to see where. The cache files will be deleted the next month.\n\n```console\n$ python source_finder.py s3transfer --verbose\nAPI URL: https://pypi.org/pypi/s3transfer/json\nCache file: /Users/hugo/Library/Caches/source-finder/2019-10-https-pypi-org-pypi-s3transfer-json.json\nCache file exists\nproject_urls    Homepage        https://github.com/boto/s3transfer\nSuccess!\nproject_urls    Homepage        https://github.com/boto/s3transfer\nSuccess!\nhttps://github.com/boto/s3transfer\n```\n\n## top_repos.py\n\nThis will look for the source repo for the most-downloaded packages, using a JSON file\nfrom [Top PyPI Packages](https://hugovk.github.io/top-pypi-packages/), and save them to\n[`data/top-repos.json`](https://hugovk.github.io/pypi-tools/data/top-repos.json).\n\nFirst, fetch fresh copy of the top packages:\n\n```console\n$ wget https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json -O  data/top-pypi-packages.json\n\n--2019-10-14 18:12:45--  https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json\nResolving hugovk.github.io (hugovk.github.io)... 185.199.110.153, 185.199.108.153, 185.199.111.153, ...\nConnecting to hugovk.github.io (hugovk.github.io)|185.199.110.153|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 250885 (245K) [application/json]\nSaving to: ‘data/top-pypi-packages.json’\n\ndata/top-pypi-packages.json      100%[========================================================\u003e] 245.00K  --.-KB/s    in 0.02s\n\n2019-10-14 18:12:45 (14.7 MB/s) - ‘data/top-pypi-packages.json’ saved [250885/250885]\n```\n\nCheck the first 10 packages:\n\n```console\n$ python top_repos.py -n 10\nLoad data/top-repos.json...\nLoad top-pypi-packages.json...\nAlready done: 0\nFind new repos...\n1 urllib3\n2 six       https://github.com/benjaminp/six\n3 requests\n4 botocore  https://github.com/boto/botocore\n5 python-dateutil\n6 certifi\n7 s3transfer        https://github.com/boto/s3transfer\n8 pip\n9 idna      https://github.com/kjd/idna\n10 docutils\nOld repos: 0\nNew repos: 4\nNot found: 6\nSave data/top-repos.json...\n```\n\nWhen running again:\n* if a package already has a repo, it's not checked again in case it's changed\n* if a package doesn't have a repo, it will be checked\n\nCurrently, it finds 3,951 repos for the top 5,000 packages.\n\nI'm not planning on automating this, but can run it from time to time to update it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhugovk%2Fpypi-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhugovk%2Fpypi-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhugovk%2Fpypi-tools/lists"}