{"id":40943281,"url":"https://github.com/stratosphereips/computationalpropagandadetection","last_synced_at":"2026-01-22T04:37:29.020Z","repository":{"id":40249038,"uuid":"273964287","full_name":"stratosphereips/computationalPropagandaDetection","owner":"stratosphereips","description":"Werge: Web Reversed Graph Extractor for computational propaganda detection on the web","archived":false,"fork":false,"pushed_at":"2022-07-05T20:40:14.000Z","size":27334,"stargazers_count":1,"open_issues_count":15,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-12-13T12:57:26.173Z","etag":null,"topics":["computational-propaganda","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stratosphereips.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-06-21T18:46:37.000Z","updated_at":"2024-03-19T21:15:20.000Z","dependencies_parsed_at":"2022-08-24T14:33:40.220Z","dependency_job_id":null,"html_url":"https://github.com/stratosphereips/computationalPropagandaDetection","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/stratosphereips/computationalPropagandaDetection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FcomputationalPropagandaDetection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FcomputationalPropagandaDetection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FcomputationalPropagandaDetection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FcomputationalPropagandaDetection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stratosphereips","download_url":"https://codeload.github.com/stratosphereips/computationalPropagandaDetection/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FcomputationalPropagandaDetection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28654885,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-propaganda","machine-learning"],"created_at":"2026-01-22T04:37:28.970Z","updated_at":"2026-01-22T04:37:29.013Z","avatar_url":"https://github.com/stratosphereips.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Werge: Web Reversed Graph Extractor\n\nA set of tools to work on computational propaganda detection\n\n## Running script\n\n    python web-network-mapper.py -l LINK -n NUMBER_OF_ITERATIONS -p (include if it is propaganda)\n\nExample of propaganda\n\n    python web-network-mapper.py -l https://euvsdisinfo.eu/report/despite-promises-to-the-contrary-nato-has-been-expanding-further-eastwards/ -p -n 30\n\n\nBuild a graph\n\n    python graph.py -l LINK\n\nQuery the DB for the links to a particular URL\n\n    DB/query_db.py -l LINK\n\n## How we get the info\n\n- We use the serapi api.\n- We download the results from one seed url\n- For each result\n\t- We filter that is not in our blacklist\n\t- We download its content with curl\n\t- We store its datetime of search from the engine\n\t- We store its datetime of our search\n\t- We ask the links to this page and we loop.\n\n\n## Blacklist\nWe apply some blacklisting of webpages based on some criteria in order to know which pages should be considered good links.\n\n# Packages you need\n- See requirements.txt\n\n## Useful Info\n- https://towardsdatascience.com/current-google-search-packages-using-python-3-7-a-simple-tutorial-3606e459e0d4\n\n## Database structure\n\n### Table LINKS\nTable links has the following fields: \n\n- link_id:\n- parent_id:\n- child_id:\n- date:\n- source: Twitter, Facebook, Webpage (this is any generic webpage), VK, Reddit, etc.\n- linktype:\n\n## Extraction of Features\n\nThe features that we are extracting from each graph are:\n\n1. Time histograms\n    Generate the features based on time\n    - For each level:\n        - Calculate a histogram of how many links there are by:\n            - In the next 48hs after the publication date of the main URL (2 days). \n                - Compute the histogram of urls published by minute\n            - After more than 48hs of the publication, for the next 120hs (5 days), that is from \u003e 2nd day to \u003c= 7th day.\n                - Compute the histogram of urls published by hour\n            - After more than 168hs of the publication, for the next 23 days, that is from \u003e 7th day to \u003c= 30th\n                - Compute the histogram of urls published by day\n\n              ```Publication of    Up to                        More than 48hs\n              main URL          48hs hs                      and up to 120hs\n                    | By minute |         By hour             |                    By day\n                    \\/          \\/                            \\/\n            Days: |--*--|-----|--*--|-----|-----|-----|-----|--*--|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n                     1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16   17...```\n\n2. The number of urls published before the source\n\n3. The total number of urls in each level \n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fcomputationalpropagandadetection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstratosphereips%2Fcomputationalpropagandadetection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fcomputationalpropagandadetection/lists"}