{"id":20450039,"url":"https://github.com/ooni/2022-04-websteps-illustrated","last_synced_at":"2025-07-11T05:05:08.635Z","repository":{"id":40368464,"uuid":"465480952","full_name":"ooni/2022-04-websteps-illustrated","owner":"ooni","description":"websteps: winter 2022 edition","archived":false,"fork":false,"pushed_at":"2023-05-30T22:56:35.000Z","size":1212,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-06-22T08:50:07.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://ooni.org","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ooni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-03-02T21:53:26.000Z","updated_at":"2023-05-30T22:54:34.000Z","dependencies_parsed_at":"2023-09-24T03:47:54.788Z","dependency_job_id":null,"html_url":"https://github.com/ooni/2022-04-websteps-illustrated","commit_stats":{"total_commits":414,"total_committers":2,"mean_commits":207.0,"dds":"0.0024154589371980784","last_synced_commit":"41ba115289da0cca3fb633b55066f35622d56091"},"previous_names":["bassosimone/websteps-illustrated"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ooni/2022-04-websteps-illustrated","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2F2022-04-websteps-illustrated","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2F2022-04-websteps-illustrated/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2F2022-04-websteps-illustrated/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2F2022-04-websteps-illustrated/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ooni","download_url":"https://codeload.github.com/ooni/2022-04-websteps-illustrated/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ooni%2F2022-04-websteps-illustrated/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262196946,"owners_count":23273531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T10:50:02.515Z","updated_at":"2025-07-11T05:05:08.618Z","avatar_url":"https://github.com/ooni.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Websteps illustrated\n\nThis repository contains the third prototype of\n[websteps](https://github.com/ooni/probe/issues/1714) (codename\n\"winter 2022\"). This prototype follows after the\n[\"summer 2021\"](https://github.com/ooni/probe-cli/tree/v3.14.1/internal/engine/experiment/websteps)\nand the\n[\"fall 2021\"](https://github.com/ooni/probe-cli/tree/v3.14.1/internal/engine/experiment/webstepsx)\nprotypes.\n\n## Content of this repository\n\nThe [internal](internal) directory contains code derived from\n[ooni/probe-cli](https://github.com/ooni/probe-cli/) v3.14.1 as\nwell as new code. As a rule of thumb, most directories could\nbe easily merged back, except `measurex` and `websteps`, which\nhave been significantly rewritten and would require either\nmore careful merging or a yolo-rewrite-everything approach.\n\nThe [cmd](cmd) directory contains commands using code in the\n[internal](internal) library. The most important commands are:\n\n- [cmd/websteps](cmd/websteps): websteps client;\n\n- [cmd/thd](cmd/thd): test helper.\n\nThe [spec](spec) directory contains the current draft\nspecification of websteps, which still needs to be discussed\nwith my colleagues and other friends of OONI.\n\nThe [python](python) directory contains:\n\n- [python/websteps.py](python/websteps.py): minimal implementation\nof the websteps concept written in Python, not meant to become production ready,\nbut rather useful to see the algorithms in a smaller context\nand show that websteps implementations not written in Go could\ninter-operated with the test helper written in Go;\n\n- [python/analysis/dbsteps](python/analysis/dbsteps): Python script\nto analyze websteps measurements and view them in the browser;\n\n- [python/testcase/create](python/testcase/create): script to create\nintegration test cases for websteps while running measurements;\n\n- [python/testcase/shell](python/testcase/shell): script to manage\nthe integration tests for websteps;\n\n- [python/ooni](python/ooni): comprehensive library to import\nand process websteps measurements using Python.\n\nThe [testdata/testcase](testdata/testcase) directory contains a few\ntest cases collected using the [create](python/testcase/create)\ncommand and managed using the [shell](python/testcase/shell) command.\n\nThe [html](html) directory contains support file for browsing\nwebsteps measurements and test cases using HTML.\n\n**NOTE**: while I spent some time to make this code polished, this is\nstill experimental code, with little unit testing and, for sure, a bunch\nof inconsistencies betwenn the spec and the implementation. This is\nnormal, given that for now websteps is still a bit of a moving target.\n\n## Building the websteps client\n\nYou need to use go1.17.13 to build this repository.\n\n```bash\ngo build -v ./cmd/websteps\n```\n\n## Building the TH\n\n```bash\ngo build -v ./cmd/thd\n```\n\n## Changes since websteps fall 2021\n\nThese are the main changes since the fall 2021 edition (collection?! 😅):\n\n1. added support for `PTR` and `NS` queries as well as for\nopportunistially extract the `CNAME` from replies;\n\n2. implemented a parallel DNSResolver using custom DNSTransport;\n\n3. reworked the system resolver to fake a DNSTransport and\nproduce more easily the OONI DNS data format;\n\n4. several reliability and correctness fixes in DNS code;\n\n5. significantly reworked the conceptual model of `measurex` to\nmore easily accommodate for implementing websteps;\n\n6. around one month of experience running websteps code in\nseveral countries (including China, Italy, and Iran), which\ndramatically helped to improve the robustness of the\nimplementation as well as to develop \"scoring\" algorithms;\n\n7. developed a set of algorithms to assign blocking flags\nto websteps measurements as well as heuristics to spot common\nclasses of false positives and flag them correctly;\n\n8. implemented and integrated a `dnsping` extension for\nwebsteps that allows to confirm with more confidence cases\nof DNS blocking as well as to retreat DNS timeout claims\nwhen there are transient timeouts;\n\n9. integration testing framework based on caching the\nTH and the probe's measurements that is based on replaying\nmeasurements collected on the field (thus being more\ntrue to real world censorship than simulated censorship\nusing `jafar` or similar tools);\n\n10. robust caching mechanism for the TH;\n\n11. started experimenting with using `TLSH` to classify\nwebpages in addition to using the traditional Web Connectivity\nalgorithm (but this effort is so far a bit inconclusive);\n\n12. TH protocol using WebSocket in addition to web APIs to\nincrease robustness when middleboxes close connections that\nstay silent for a number of seconds;\n\n13. figure out ways in which the original, optimistic\nlet's-measure-every-endpoint model breaks when coupled with\nthe typical OONI constraints of timing and single-URL-at-a-time\nand add to the algorithm reasonable settings to strike a\nbalance between depth and breadth;\n\n14. learn that my effort estimate is usually off by a 5x factor 😬.\n\n15. the design incorporates future improvements in the check-in\nAPI that will allow us to customize how we measure URLs\ndepending on the context (so, we will be able to say for\neach URL in a given country and ASN, the amount of body bytes\nto download, whether to follow redirects, etc).\n\nThis work addresses in part of completely:\n\n| issue | level of completion |\n| -- | -- |\n| [probe#2034](https://github.com/ooni/probe/issues/2034) | complete |\n| [probe#1190](https://github.com/ooni/probe/issues/1990) | complete |\n| [probe#1806](https://github.com/ooni/probe/issues/1806) | complete |\n| [probe#1803](https://github.com/ooni/probe/issues/1803) | now unnecessary |\n| [probe#1516](https://github.com/ooni/probe/issues/1516) | mostly(?) complete |\n| [probe#1718](https://github.com/ooni/probe/issues/1718) | complete |\n\n## What happens now\n\n- [ ] continuing to discuss the spec with OONI friends;\n\n- [ ] prepare short presentation for pitching websteps since\nthe spec is long and it may be beneficial to also provide\npeople with short introductions;\n\n- [ ] continue extensive data analysis and start preparing\nreports/blog posts based on this work;\n\n- [ ] write spec for extensions (including `dnsping`, already\nimplemented, and `sniblocking`, which we need);\n\n- [ ] collect more test cases and add support for automatically\nchecking that we're still passing these test cases;\n\n- [ ] figure out ways to auto-generate parts of the codebase\nif possible (especially python data structs that depend on\nGo data structs: that would be nice);\n\n- [ ] perform again a performance comparison with Web Connectivity\nand also a comparison in terms of accuracy;\n\n- [ ] double check that our level of parallelism is adequate\nfor testing in low bandwidth scenarios;\n\n- [ ] start merging back into `probe-cli` the easy parts and\ngenerally aim to reduce the diff between this fork and the\noriginal codebase;\n\n- [ ] sync up the OONI issue tracker with the work I have\nbeem doing here basically in `sti` mode;\n\n- [ ] extend the underlying library to add support for as many\nraw errors as possible;\n\n- [ ] decide how to adapt tutorials to changes in here.\n\n\n## Nginx setup\n\nIf `thd` is running locally (and please rememeber to\nforce it to drop `root` privileges), you can integrate\nit with an existing `nginx` setup by adding:\n\n```\n  location /websteps/v1/websocket {\n      proxy_read_timeout 900;\n      proxy_pass http://127.0.0.1:9876;\n      proxy_set_header X-Forwarded-Proto $scheme;\n      proxy_set_header Upgrade $http_upgrade;\n      proxy_set_header Connection \"Upgrade\";\n      proxy_set_header Host $host;\n  }\n  location /websteps/v1/http {\n      proxy_set_header X-Forwarded-Proto $scheme;\n      proxy_read_timeout 900;\n      proxy_pass http://127.0.0.1:9876;\n  }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2F2022-04-websteps-illustrated","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fooni%2F2022-04-websteps-illustrated","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooni%2F2022-04-websteps-illustrated/lists"}