{"id":14155568,"url":"https://github.com/commoncrawl/cc-notebooks","last_synced_at":"2025-06-12T08:41:11.086Z","repository":{"id":47653838,"uuid":"197762383","full_name":"commoncrawl/cc-notebooks","owner":"commoncrawl","description":"Various Jupyter notebooks about Common Crawl data","archived":false,"fork":false,"pushed_at":"2025-04-01T16:54:24.000Z","size":3154,"stargazers_count":51,"open_issues_count":0,"forks_count":9,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-04-01T17:51:34.590Z","etag":null,"topics":["aws-athena","common-crawl","commoncrawl","jupyter-notebook","webarchiving","webgraph-framework"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/commoncrawl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-19T11:38:10.000Z","updated_at":"2025-04-01T16:54:28.000Z","dependencies_parsed_at":"2025-04-01T17:41:20.476Z","dependency_job_id":"218d32cd-0bf7-4786-a18e-407efcffc175","html_url":"https://github.com/commoncrawl/cc-notebooks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/commoncrawl/cc-notebooks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-notebooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-notebooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-notebooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-notebooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/commoncrawl","download_url":"https://codeload.github.com/commoncrawl/cc-notebooks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-notebooks/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259432321,"owners_count":22856726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-athena","common-crawl","commoncrawl","jupyter-notebook","webarchiving","webgraph-framework"],"created_at":"2024-08-17T08:03:57.710Z","updated_at":"2025-06-12T08:41:11.081Z","avatar_url":"https://github.com/commoncrawl.png","language":"Jupyter Notebook","funding_links":[],"categories":["Tools \u0026 Software","jupyter-notebook"],"sub_categories":["Analysis"],"readme":"# Jupyter Notebooks to Analyze Common Crawl Data\n\n\n* analyzing data using the [columnar index](https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/)\n  - blocking of internet connections from and to the Islamic Republic of Iran during the November 2019 crawl: [net-blocking-iran-cc-main-2019-47.ipynb](./cc-index-table/net-blocking-iran-cc-main-2019-47.ipynb)\n  - total number of captures 2013 – 2019, domain coverage and approximation of unique URLs for the `.edu` top-level domain: [cc-main-2013-2019-metrics.ipynb](./cc-index-table/cc-main-2013-2019-metrics.ipynb)\n  - correlations between character sets and lanuages: [correlation-language-charset.ipynb](./cc-index-table/correlation-language-charset.ipynb)\n* analyze the Common Crawl webgraph data sets and interactively explore the graphs: [cc-webgraph-statistics](./cc-webgraph-statistics/)\n* how to explore WARC files [running a notebook on AWS EMR](./cc-emr-notebook/cluster_setup.md)\n* [truncated record payloads in WARC Files](./warc-truncation/):\n  - verify that all truncated payloads are annotated by the [WARC-Truncated header](https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#warc-truncated)\n  - which MIME types are mostly affected by truncation? Aggregations using the columnar index.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommoncrawl%2Fcc-notebooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcommoncrawl%2Fcc-notebooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommoncrawl%2Fcc-notebooks/lists"}