{"id":15576423,"url":"https://github.com/njsmith/codetrawl","last_synced_at":"2026-04-11T23:36:59.376Z","repository":{"id":145901691,"uuid":"42155065","full_name":"njsmith/codetrawl","owner":"njsmith","description":"Better software through corpus analysis","archived":false,"fork":false,"pushed_at":"2015-09-20T22:16:44.000Z","size":160,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-30T07:30:44.747Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njsmith.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-09T04:02:05.000Z","updated_at":"2017-05-30T11:42:07.000Z","dependencies_parsed_at":"2023-04-07T23:12:06.705Z","dependency_job_id":null,"html_url":"https://github.com/njsmith/codetrawl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/njsmith/codetrawl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcodetrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcodetrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcodetrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcodetrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njsmith","download_url":"https://codeload.github.com/njsmith/codetrawl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcodetrawl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31699438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T21:17:31.016Z","status":"ssl_error","status_checked_at":"2026-04-11T21:17:24.556Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T18:49:02.004Z","updated_at":"2026-04-11T23:36:59.360Z","avatar_url":"https://github.com/njsmith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"codetrawl - cast a wide net\n===========================\n\nThis is a little idiosyncratic and mostly undocumented tool for when\nyou want to get a list of everyone in the world who is using some API\n(e.g. so you know who to talk to if you want to break it).\n\nWell, not everyone in the world. But it knows how to search both\n`searchcode.com \u003csearchcode.com\u003e`_ which covers basically all of\nbitbucket, Fedora, and others, and GitHub which covers all\nof... GitHub. (The latter is done via screen scraping, because the\n`search API is useless for this, woohoo\n\u003chttps://developer.github.com/changes/2013-10-18-new-code-search-requirements/\u003e`_\n-- GitHub support helped me work out this strategy, the tool clearly\nidentifies itself in the User-Agent header, and it gracefully accepts\nthrottling, so while it's awkward we're not trying to break any rules\nhere.)\n\n\nLicense\n-------\n\nGPLv3+\n\n\nRequirements\n------------\n\n``pip install requests docopt jinja2 lxml``\n\nOptional requirement: ``browser_cookie`` if you want to speed up\nGithub searches slightly by performing them while logged in.\n\n\nInstallation\n------------\n\nI haven't bothered making this work -- I just stick the source\ncheckout on my ``PYTHONPATH``.\n\n\nUsage\n-----\n\nBasic idea is that first you performs some searches, e.g.::\n\n  python -m codetrawl github \"PyUFuncObject language:C\" | gzip \u003e results/C.gz\n\nThis gives you a file with one-JSON-blob-per-line, and each JSON blob\ncontains some metadata plus the full contents of the matching file.\n\nThen you write a report script to post-process the results, e.g.::\n\n  import glob\n  from codetrawl.report import make_report, group, filter\n  make_report(r\"PyUFuncObject\",\n              glob.glob(\"results/*.gz\"),\n              \"my-report.html\",\n              [group(\"NumPy forks / precursors / etc.\",\n                     [filter(\"repo\", \"bohrium/numpy\", \"numpy fork\"),\n                      filter(\"repo\", \"wolfgarnet/numpycbe\", \"numpy fork\"),\n                      filter(\"path\", \"libndarray/\"),\n                      ]),\n               group(\"Irrelevant\",\n                     [filter(\"path\", \"/site-packages/\",\n                             \"someone checked in their venv/conda tree\"),\n                     ]),\n              ])\n\nThis finds all the lines in your search files that match the given\nregexp pattern, groups them into bins according to the criteria you\nspecify, and writes a report to ``my-report.html``. Filters are all\nregexes, and can be on ``\"repo\"``, ``\"path\"``, ``\"line\"``,\n``\"content\"`` (catches all lines that appear in a file that contains\nany matches to the given regex), or ``\"raw_url\"`` (the URL where we\ndownloaded the file from). Uncompressed or gzip-compressed results\nfiles are both supported.\n\nAlso handy: ``python -m codetrawl.dump``\n\n\nGithub notes\n------------\n\nGitHub code search `documentation\n\u003chttps://help.github.com/articles/searching-code/\u003e`_.\n\nThe query language is extremely quirky -- I don't understand the\ntokenization rules at all, for example. Do quotes in the search string\ndo something? Maybe? Anyway, `try out your search manually first\n\u003chttps://github.com/search?type=Code\u003e`_.\n\nIMPORTANT: you can never see beyond 1000 results for a single\nsearch. If your search has more than this, then you need to figure out\nhow to break it into multiple sub-searches. E.g., you can search\nseparately by each language (using ``language:C``, ``language:C++``,\netc.), or if searching for ``a`` gives \u003e1000 results, then try instead\ntaking the union of ``a AND b`` and ``a NOT b`` (each of which\nhopefully have \u003c=1000 results). Of course the trick is to figure out\nwhat to use for ``b`` -- unfortunately neither I nor Github support\nhave ideas for how to do this robustly or automatically. (An\napparently clever idea was to use the ``size:`` specified to slice the\ndata into files of size 0-1000 bytes, 1000-2000 bytes, etc., but in\npractice such searches seem to time-out on the server side and return\npartial results, which defeats the purpose.)\n\nHere are some valid language codes: ``C``, ``C++`` *or* ``cpp``,\n``objective-c``, ``python``, ``cython``. Swig apparently gets detected\nas being \"C\" (`example\n\u003chttps://github.com/search?l=c\u0026q=PyArray_Dtype+in%3Afile%2Cpath+NOT+numpy%2Fcore+NOT+extras%2Fnumpy_include+NOT+ndarrayobject\u0026ref=searchresults\u0026type=Code\u0026utf8=%E2%9C%93\u003e`_).\n\nSome challenges that codetrawl.py works hard to overcome:\n\n* Sometimes a search that normally works will randomly have a\n  server-side timeout. We only give up on a query if we see three\n  server-side timeouts in a row.\n* Github has throttling controls to prevent abuse, which appear to be\n  signalled with a ``429 Too Many Requests`` HTTP error. If we see\n  this then we retry with exponential backoff until we succeed.\n* The order of search results is not stable, so if you have 5 pages of\n  search results, just requesting each of these pages may or may not\n  give you all of the results. A particular hit might appear on page 3\n  while you're fetching page 2, and then on page 2 when you're\n  fetching page 3, so you miss it. (And of course you can also end up\n  seeing the same hit twice via a similar mechanism.) So if github\n  says there are 123 results, we keep refreshing the search results\n  until we see 123 distinct results.\n\n\nSearchcode notes\n----------------\n\nSearchcode is less tricky -- you can basically just do a string\nsearch, and get up to 4900 results. There's no throttling or\nanything.\n\nI haven't done any exhaustive checking on whether paging through the\nresults actually gives you all the results, but it did at least pass\nsome spot-checks that github search fails.\n\nAccording to its author (personal communication), it's believed to cover:\n\n* Part of Github (~3 million repositories, which turns out to only be\n  a fraction of Github -- though it's biased towards higher profile\n  projects.)\n* All of Bitbucket\n* All of Fedora (though possibly out of date)\n* All of CodePlex\n* Substantial chunks of: Google Code, Sourceforge, Tizen, Android,\n  Minix3, GNU, ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fcodetrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjsmith%2Fcodetrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fcodetrawl/lists"}