{"id":13678507,"url":"https://github.com/Stormix/sourcerer-io-scrapper","last_synced_at":"2025-04-29T15:31:33.906Z","repository":{"id":111597340,"uuid":"273107485","full_name":"Stormix/sourcerer-io-scrapper","owner":"Stormix","description":"Sourcerer.io scrapper + API - https://sourcerer.io/stormiix","archived":false,"fork":false,"pushed_at":"2023-05-01T21:24:45.000Z","size":13,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-11T21:37:20.429Z","etag":null,"topics":["api","caprover","docker","flask","python","selenium","sourcerer"],"latest_commit_sha":null,"homepage":"https://sourcerer.stormix.co/fetch/stormiix","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Stormix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-18T00:44:51.000Z","updated_at":"2022-12-29T18:28:21.000Z","dependencies_parsed_at":"2024-01-14T15:23:24.462Z","dependency_job_id":"311c29d1-0d02-4cbc-9648-4a3d13ec2aba","html_url":"https://github.com/Stormix/sourcerer-io-scrapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stormix%2Fsourcerer-io-scrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stormix%2Fsourcerer-io-scrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stormix%2Fsourcerer-io-scrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stormix%2Fsourcerer-io-scrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Stormix","download_url":"https://codeload.github.com/Stormix/sourcerer-io-scrapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251529550,"owners_count":21603975,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","caprover","docker","flask","python","selenium","sourcerer"],"created_at":"2024-08-02T13:00:54.367Z","updated_at":"2025-04-29T15:31:33.630Z","avatar_url":"https://github.com/Stormix.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# Sourcerer.io scrapper + API\n\n[![GitHub issues](https://img.shields.io/github/issues/Stormix/sourcerer-io-scrapper.svg?style=flat-square)](https://github.com/Stormix/sourcerer-io-scrapper/issues)\n[![GitHub forks](https://img.shields.io/github/forks/Stormix/sourcerer-io-scrapper.svg?style=flat-square)](https://github.com/Stormix/sourcerer-io-scrapper/network)\n[![GitHub stars](https://img.shields.io/github/stars/Stormix/sourcerer-io-scrapper.svg?style=flat-square)](https://github.com/Stormix/sourcerer-io-scrapper/stargazers)\n \nBecause they don't have an API. https://sourcerer.io/Stormix\n\n## Requirements\n\n- Python 3.x \n- Selenium\n- BeautifulSoup4\n- Flask\n\n## How to use\n\nInstall requirements:\n\n```\npip install -r requirements.txt\n```\n\nRun the flask app:\n```\npython main.py\n```\n\n## API\n### GET Request `/fetch/username`\nJust send a `GET` request to: https://sourcerer.stormix.co/fetch/\u003cusername\u003e. Give it a second, it spins up a chrome instance to load the site and fetch your profile info. Due to the time it takes, I highly suggest you cache the API response for later use and schedule API calls to update later.\n### `200` Response\n\nThe response contains:\n- updated_at timestamp\n- the overview chart info with datapoints containing the specific date range and the commits/language information\n- name, username, job, website\n- overview technologies\n- detailed technologies + libraries\n- fun_facts\n  \nand has the following format:\n```\n{\n  \"chart\": {\n    \"dataset\": [\n      { \n        \"data\": [\n          {\n            \"commits\": 15, \n            \"language\": \"Python\", \n            \"locs\": 570\n          }, \n          {\n            \"commits\": 6, \n            \"language\": \"PHP\", \n            \"locs\": 20\n          }\n        ], \n        \"date\": [\n          \"2017-01-01T00:00:00\", \n          \"2017-04-02T00:00:00\"\n        ]\n      }, \n      ...\n      }\n    ], \n    \"updated_at\": \"2020-06-18T00:06:00\"\n  }, \n  \"commits\": \"994\", \n  \"fun_facts\": [\n    {\n      \"fact\": [\n        \"I'm most productive during daytime\", \n        \"49% of users\", \n        \"d\"\n      ], \n      \"stat\": [\n        \"\"\n      ]\n    }, \n    ...\n  ], \n  \"job\": \"Software Engineer\", \n  \"loc\": \"5617325\", \n  \"location\": \"Nantes, France\", \n  \"name\": \"Anas Mazouni\", \n  \"overview_technologies\": [\n    \"Python\", \n    \"PHP\", \n    \"Slugify\", \n    \"JavaScript\", \n    \"Travis CI\", \n    \"HTML\", \n    \"Laravel\", \n    \"CSS\", \n    \"vue\", \n    \"express\", \n    \"async\", \n    \"vuex\"\n  ], \n  \"site\": \"stormix.co\", \n  \"technologies\": [\n    {\n      \"commits\": 994, \n      \"libraries\": [\n        {\n          \"loc\": 98503, \n          \"name\": \"vue\", \n          \"type\": \"ui-framework\"\n        }, \n        ...\n      ], \n      \"title\": \"JavaScript Web\"\n    },\n    ...\n  ], \n  \"updated_at\": \"2020-06-18T00:06:00\", \n  \"username\": \"Stormix\"\n}\n```\n### `500` Response\n`Something went wrong ¯\\_(ツ)_/¯`\n\n### `404` Response\n`\n{\n  \"error\": \"Not found\"\n}\n`\n## Deployment \n\n### Docker\n\nThe repo includes a docker file to build an image of the project, you'll need to expose port `3000` to expose the flask application. \n\nIts built around the `python:3.8` image, it installs google-chrome and downloads the chrome driver to use with selenium.\n\n### [CapRover](http://caprover.com/)\n\nIf you have a [CapRover](http://caprover.com/) server somewhere, you can run the `caprover deploy` command to deploy the project.\n\nDon't forget to set the Container HTTP Port to `3000`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStormix%2Fsourcerer-io-scrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FStormix%2Fsourcerer-io-scrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStormix%2Fsourcerer-io-scrapper/lists"}