{"id":20450088,"url":"https://github.com/ooni/backend","last_synced_at":"2025-04-13T02:11:46.386Z","repository":{"id":7380588,"uuid":"8707973","full_name":"ooni/backend","owner":"ooni","description":"Everything related to OONI backend infrastructure: ooni/api, ooni/pipeline, ooni/sysadmin, collector, bouncers and test-helpers","archived":false,"fork":false,"pushed_at":"2024-04-09T13:31:34.000Z","size":7914,"stargazers_count":45,"open_issues_count":217,"forks_count":28,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-04-14T06:49:45.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ooni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2013-03-11T16:37:48.000Z","updated_at":"2024-04-15T23:21:39.369Z","dependencies_parsed_at":"2023-09-23T09:59:29.241Z","dependency_job_id":"f8209d0b-2e76-4fc0-a3c3-3d01e371f3c3","html_url":"https://github.com/ooni/backend","commit_stats":{"total_commits":2828,"total_committers":42,"mean_commits":67.33333333333333,"dds":0.7118104667609618,"last_synced_commit":"57de6256627bfa776d0c73113be80f4dfd31fa6f"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fbackend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fbackend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fbackend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2Fbackend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ooni","download_url":"https://codeload.github.com/ooni/backend/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248654089,"owners_count":21140236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T10:50:17.046Z","updated_at":"2025-04-13T02:11:46.382Z","avatar_url":"https://github.com/ooni.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OONI Backend\n\n**NOTE** this repository contains both the monolith API backend code (inside of `api/`) and the new port to some updated patterns based on fastapi (see: `ooniapi/`) \n\nThe backend infrastructure performs multiple functions:\n\n- Provide APIs for data consumers\n\n- Instruct probes on what measurements to perform\n\n- Receive measurements from probes, process them and store them in the database\n\n- Upload new measurements to a bucket on [S3 data bucket](#s3-data-bucket)\n\n- Fetch data from external sources e.g. fingerprints from a GitHub repository\n\n## Main data flows\n\nOONI Probes will run generally once every hour or every day, depending on the platform.\nAs part of these runs the sequence diagram of a probe run looks like the following:\n\n```mermaid\nsequenceDiagram\n  participant OONIProbe as OONI Probe\n  participant ProbeServices as OONI Backend\n  participant Internet\n  OONIProbe -\u003e\u003e+ Internet: lookupProbeMeta()\n  Internet -\u003e\u003e- OONIProbe: ProbeMeta\n  OONIProbe -\u003e\u003e+ ProbeServices: checkIn(ProbeMeta)\n  ProbeServices --\u003e\u003e- OONIProbe: []Targets\n  loop Every target\n    OONIProbe -\u003e\u003e+ Internet: runExperiment(target)\n    opt Control\n        OONIProbe -\u003e\u003e+ ProbeServices: runControl(target)\n        ProbeServices -\u003e\u003e- OONIProbe: CtrlMeasurement\n    end\n    Internet -\u003e\u003e- OONIProbe: Measurement\n    OONIProbe -\u003e\u003e ProbeServices: upload(Measurement)\n  end\n```\n\nThe following diagram on the other hand, represents the main flow of measurement data.\n\nThe dark rectangles represent processes. The cilinders represent data at rest:\nas files on disk, files on S3 or records in database tables.\n\n```mermaid\nflowchart LR\n    A((\"Measurement\")):::measurement --\u003e B[\"Measurement is uploaded\"]\n    B --\u003e C[\"Fastpath (realtime)\"]:::gray8Node \u0026 D[\"Disk Queue\"]\n    C --\u003e E[\"Fastpath Table\"]:::gray3Node@{ shape: cyl}\n    D --\u003e F[\"S3 Uploader (every hour)\"]:::gray8Node\n    F --\u003e G[\"s3://ooni-data-eu-fra bucket\"]@{shape: cyl}\n    E --\u003e H[\"OONI API\"]:::gray8Node\n    D --\u003e decision{\"`is older than 1h?`\"}\n    G --\u003e decision\n    decision --\u003e H\n\n    G --\u003e PipelineV5[\"OONI Pipeline v5 (every day)\"]:::gray8Node\n    PipelineV5 --\u003e O[\"Observation Tables\"]:::gray3Node@{ shape: cyl}\n    O --\u003e H\n\n    classDef measurement fill:#0588cb,color:#fff\n    classDef gray2Node fill:#e9ecef,color:#000000\n    classDef gray3Node fill:#ced4da,color:#000000\n    classDef gray8Node fill:#343a40,color:#fff\n```\n\nProbes submit measurements to the API with a POST at the following path:\n\u003chttps://api.ooni.io/apidocs/#/default/post_report__report_id_\u003e The\nmeasurement is optionally decompressed if zstd compression is detected.\nIt is then parsed and added with a unique ID and saved to disk. Very\nlittle validation is done at this time in order to ensure that all\nincoming measurements are accepted.\n\nMeasurements are enqueued on disk using one file per measurement. On\nhourly intervals they are batched together, compressed and uploaded to\nS3 by the [Measurement uploader](#measurement-uploader)\u0026thinsp;⚙. The batching is\nperformed to allow efficient compression. See the\n[dedicated subchapter](#measurement-uploader)\u0026thinsp;⚙ for details.\n\nThe measurement is also sent to the [Fastpath](#fastpath)\u0026thinsp;⚙. The\nFastpath runs as a dedicated daemon with a pool of workers. It\ncalculates scoring for the measurement and writes a record in the\nfastpath table. Each measurement is processed individually in real time.\nSee the [dedicated subchapter](#fastpath)\u0026thinsp;⚙ below.\n\nThe disk queue is also used by the API to access recent measurements\nthat have not been uploaded to S3 yet. See the\n[measurement API](#getting-measurement-bodies)\u0026thinsp;🐝 for details.\n\n## Reproducibility\n\nThe measurement processing pipeline is meant to generate outputs that\ncan be equally generated by 3rd parties like external researchers and\nother organizations.\n\nThis is meant to keep OONI accountable and as a proof that we do not\narbitrarily delete or alter measurements and that we score them as\naccessible/anomaly/confirmed/failure in a predictable and transparent\nway.\n\n\u003e **important**\n\u003e The only exceptions were due to privacy breaches that required removal\n\u003e of the affected measurements from the [S3 data bucket](#s3-data-bucket)\u0026thinsp;💡\n\u003e bucket.\n\nAs such, the backend infrastructure is\n[FOSS](https://en.wikipedia.org/wiki/Free_and_open-source_software) and\ncan be deployed by 3rd parties. We encourage researchers to replicate\nour findings.\n\nIncoming measurements are minimally altered by the\n[Measurement uploader](#measurement-uploader)\u0026thinsp;⚙ and uploaded to S3.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2Fbackend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fooni%2Fbackend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2Fbackend/lists"}