{"id":20337851,"url":"https://github.com/jigsaw-code/censoredplanet-analysis","last_synced_at":"2025-04-11T23:11:33.231Z","repository":{"id":52950813,"uuid":"276225969","full_name":"Jigsaw-Code/censoredplanet-analysis","owner":"Jigsaw-Code","description":"Pipeline for Analysing CensoredPlanet Data.","archived":false,"fork":false,"pushed_at":"2021-04-12T16:27:14.000Z","size":13838,"stargazers_count":6,"open_issues_count":0,"forks_count":5,"subscribers_count":11,"default_branch":"master","last_synced_at":"2023-03-01T06:45:45.912Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jigsaw-Code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-30T22:56:29.000Z","updated_at":"2021-11-24T03:24:53.000Z","dependencies_parsed_at":"2022-08-28T09:01:47.398Z","dependency_job_id":null,"html_url":"https://github.com/Jigsaw-Code/censoredplanet-analysis","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jigsaw-Code%2Fcensoredplanet-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jigsaw-Code%2Fcensoredplanet-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jigsaw-Code%2Fcensoredplanet-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jigsaw-Code%2Fcensoredplanet-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jigsaw-Code","download_url":"https://codeload.github.com/Jigsaw-Code/censoredplanet-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224691667,"owners_count":17353686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T21:10:35.803Z","updated_at":"2024-11-14T21:10:36.517Z","avatar_url":"https://github.com/Jigsaw-Code.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":":warning: This repository has been deprecated in favor of [censoredplanet/censoredplanet-analysis](https://github.com/censoredplanet/censoredplanet-analysis). Please check there for ongoing work.\n\n# Censored Planet Data Analysis Pipeline\n\nThis pipeline takes raw data from the\n[Censored Planet Observatory](https://censoredplanet.org/data/raw) and runs it\nthrough a pipeline to create bigquery tables for easier data analysis.\n\nBecause of the size of the data involved (many TB) this project requires a\ndevoted Google Cloud project to run in. It is not recommended to run yourself,\nbut the code is made available for anyone who wants to understand how the data\npipeline works.\n\n![Master Build Status](https://github.com/Jigsaw-Code/censoredplanet-analysis/workflows/build/badge.svg?branch=master)\n\n## System Diagram\n\n![System Diagram](system-diagram.svg)\n\n## Running as an Automated Pipeline\n\nThere are two main top-level pieces\n\n`python -m mirror.data_transfer`\n\nThis sets up a daily data transfer job to copy scan files from the Censored\nPlanet cloud bucket to an internal bucket.\n\n`python -m schedule_pipeline`\n\nThis does some additional daily data processing and schedules an daily\nincremental Apache Beam pipeline over the data. It expects to be run via a\nDocker container on a GCE machine.\n\n`./deploy.sh prod`\n\nWill deploy the main pipeline loop to a GCE machine\n\n## Running Manually\n\nIndividual pieces of the pipeline can be run manually.\n\n### Mirroring Data\n\nThese scripts pull in a large amount of data from external datasources and\nmirror it in the correct locations in google cloud buckets.\n\n`python -m mirror.untar_files.sync_files`\n\nDecompresses any Censored Planet scan files which have been transfered into the\nproject but are still compressed. This can also be used as a backfill tool to\ndecompress missing files.\n\n`python -m mirror.routeviews.sync_routeviews`\n\nTransfers in the latest missing CAIDA routeview files. (Can only mirror in data\nfrom the last 30 days of data.)\n\n`python -m mirror.routeviews.bulk_download`\n\nTransfers in all CAIDA routeview files from a certain date. This is used for\nbackfilling data from older than 30 days.\n\nIn all cases to fix missing or incorrect data simple delete the incorrect data\nfrom the google cloud bucket and re-run the appropriate script to re-mirror it.\n\n### Processing Data\n\n`python -m pipeline.run_beam_tables --env=prod --full`\n\nRuns the full Apache Beam pipeline. This will re-process all data and rebuild\nexisting base tables.\n\n`python -m table.run_queries`\n\nRuns queries to recreate any tables derived from the base tables.\n\n## Testing\n\nAll tests and lint checks will be run on pull requests using\n[github actions](https://github.com/Jigsaw-Code/censoredplanet-analysis/actions)\n\nTo run all tests run\n\n`python -m unittest`\n\nThere are a few end-to-end tests which aren't run by the unittest framework\nbecause they require cloud resource access. To run these tests manually use the\ncommand\n\n`python -m unittest pipeline.manual_e2e_test.PipelineManualE2eTest`\n\nTo typecheck all files install `mypy` and run\n\n`mypy **/*.py --namespace-packages --explicit-package-bases`\n\nTo format all files install `yapf` and run\n\n`yapf --in-place --recursive .`\n\nTo get all lint errors install `pylint` and run\n\n`python -m pylint **/*.py --rcfile=setup.cfg`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjigsaw-code%2Fcensoredplanet-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjigsaw-code%2Fcensoredplanet-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjigsaw-code%2Fcensoredplanet-analysis/lists"}