{"id":20364884,"url":"https://github.com/joeri-abbo/python-credly-scraper","last_synced_at":"2025-09-23T17:31:41.723Z","repository":{"id":157221643,"uuid":"622022029","full_name":"Joeri-Abbo/python-credly-scraper","owner":"Joeri-Abbo","description":"This project is a set of Python scripts designed to crawl and extract data from the Credly platform, focusing on skills, organizations, and badges. The scripts allow users to perform searches using command-line arguments, predefined search terms, or skills listed in a JSON file. The collected data is then saved to JSON files for further analysis an","archived":false,"fork":false,"pushed_at":"2025-07-26T20:39:28.000Z","size":66760,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-26T22:08:00.886Z","etag":null,"topics":["badges","crawler","credly","data-extraction","json","organizations","python","python3","requests-library","skills","web-crawling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Joeri-Abbo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-31T23:02:54.000Z","updated_at":"2025-07-26T17:50:22.000Z","dependencies_parsed_at":"2023-10-04T13:55:58.238Z","dependency_job_id":"7167e175-9240-44b0-bcce-6c2708ad2a67","html_url":"https://github.com/Joeri-Abbo/python-credly-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Joeri-Abbo/python-credly-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joeri-Abbo%2Fpython-credly-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joeri-Abbo%2Fpython-credly-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joeri-Abbo%2Fpython-credly-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joeri-Abbo%2Fpython-credly-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Joeri-Abbo","download_url":"https://codeload.github.com/Joeri-Abbo/python-credly-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joeri-Abbo%2Fpython-credly-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276616636,"owners_count":25674130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-23T02:00:09.130Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["badges","crawler","credly","data-extraction","json","organizations","python","python3","requests-library","skills","web-crawling"],"created_at":"2024-11-15T00:14:07.105Z","updated_at":"2025-09-23T17:31:36.709Z","avatar_url":"https://github.com/Joeri-Abbo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project: Credly Crawler (WIP)\n\nThis project consists of Python scripts designed to crawl and extract data from the Credly platform. The main components\nof the project are:\n\n1. crawl-by-arg.py\n2. crawl-by-search-terms.py\n3. crawl-by-skills.py\n4. get-badges.py\n5. helper.py\n\n## Requirements\n\nPython 3.x\nrequests library\nInstall the requirements using the following command:\n\n```bash\npip install requests \n```\n\n## Usage\n\n### 1. crawl-by-arg.py\n\nThis script crawls the Credly platform using a single search term passed as a command-line argument.\n\nUsage:\n\n```bash\npython crawl-by-arg.py \u003csearch_term\u003e\n```\n\n### 2. crawl-by-search-terms.py\n\nThis script crawls the Credly platform using a list of search terms specified in the `data/search-terms.json` file.\n\n#### Usage:\n\n```bash\npython crawl-by-search-terms.py\n```\n\n### 3. crawl-by-skills.py\n\nThis script crawls the Credly platform using a list of skills that are retrieved from the data/skills.json file.\n\n#### Usage:\n\n```bash\npython crawl-by-skills.py\n```\n\n### 4. get-badges.py\n\nThis script retrieves all badges for each organization specified in the data/organizations.json file. The badges are\nthen saved to the data/badges.json file.\n\n#### Usage:\n\n```bash\npython get-badges.py\n```\n\n### 5. helper.py\n\nThis script contains helper functions used by the other scripts in this project. Functions include:\n\n- get_skills_file()\n- get_organizations_file()\n- get_badges_file()\n- get_search_terms_file()\n- get_items_by_search_term(search_term)\n- search_terms()\n- get_items_from_file(file_name)\n- set_items_from_file(file_name, items)\n- crawl_search_terms(terms)\n\n## Notes\n\nBefore running the scripts, make sure to create the necessary data files in the data directory:\n\n- skills.json\n- organizations.json\n- badges.json\n- search-terms.json\n\nEach of these files should contain an empty JSON object {} if there is no initial data.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeri-abbo%2Fpython-credly-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoeri-abbo%2Fpython-credly-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeri-abbo%2Fpython-credly-scraper/lists"}