{"id":23158975,"url":"https://github.com/jakesteam/stumbleupon-extract","last_synced_at":"2025-04-04T18:41:54.877Z","repository":{"id":258610183,"uuid":"874264097","full_name":"JakeSteam/StumbleUpon-extract","owner":"JakeSteam","description":"Extracted \u0026 parsed StumbleUpon data","archived":false,"fork":false,"pushed_at":"2024-10-27T19:12:50.000Z","size":6264,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-10T03:47:10.191Z","etag":null,"topics":["csv","html","python","stumbleupon","wayback-machine"],"latest_commit_sha":null,"homepage":"https://blog.jakelee.co.uk/bulk-downloading-website-history-and-parsing/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JakeSteam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-17T14:21:17.000Z","updated_at":"2024-10-27T19:12:54.000Z","dependencies_parsed_at":"2024-10-27T20:05:59.917Z","dependency_job_id":"790f2806-56f9-41d5-ae50-aa04a20c6206","html_url":"https://github.com/JakeSteam/StumbleUpon-extract","commit_stats":null,"previous_names":["jakesteam/stumbleupon-extract"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JakeSteam%2FStumbleUpon-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JakeSteam%2FStumbleUpon-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JakeSteam%2FStumbleUpon-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JakeSteam%2FStumbleUpon-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JakeSteam","download_url":"https://codeload.github.com/JakeSteam/StumbleUpon-extract/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234844,"owners_count":20905852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","html","python","stumbleupon","wayback-machine"],"created_at":"2024-12-17T22:28:29.203Z","updated_at":"2025-04-04T18:41:54.850Z","avatar_url":"https://github.com/JakeSteam.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StumbleUpon extract tools \u0026 data\n\nFully processed StumbleUpon data extracted from the Wayback Machine, for [an article](https://blog.jakelee.co.uk/bulk-downloading-website-history-and-parsing/).\n\n## What's in this repo?\n\n- `/data-parsed/`\n  - `parsed-cleaned.csv`: Final deduplicated extracted data.\n  - `parsed.csv`: Data before deduplication.\n- `/data-raw/`: Output of `waybackpack`, organised by timestamp and URL.\n- `/samples/`: Examples of the downloaded HTML, an individual StumbleUpon link, and the resulting CSV data.\n- `/url-analysis/`: The raw URLs from `parsed-cleaned.csv`, plus their status codes using `vl`.\n- `clean_stumbleupon_metadata.py`: Tool to deduplicate a CSV by `id` field (convert `parsed.csv` into `parsed-cleaned.csv`).\n- `extract_stumbleupon_metadata.py`: Tool to extract contents of downloaded StumbleUpon pages (convert `data-raw` contents into `parsed.csv`).\n- `analyse_stumbleupon_metadata.py`: Misc code to analyse the parsed data. This changes as required, full scripts available in original article.\n\n## How to recreate results?\n\nTo recreate the final output ([`parsed-cleaned.csv`](/data-parsed/parsed-cleaned.csv)):\n\n1. Install Python dependencies (`pip install beautifulsoup4 lxml pandas`)\n2. Run Wayback Machine download script (`waybackpack http://www.stumbleupon.com/discover/toprated/ -d \"/Projects/StumbleUpon-extract/data-raw\"`)\n3. Run parsing script (`python extract_stumbleupon_metadata.py`)\n4. Run deduping script (`python clean_stumbleupon_metadata.py`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakesteam%2Fstumbleupon-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjakesteam%2Fstumbleupon-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakesteam%2Fstumbleupon-extract/lists"}