{"id":31532650,"url":"https://github.com/dearopen/django-easy-scraper","last_synced_at":"2026-05-14T12:35:05.805Z","repository":{"id":57419874,"uuid":"313551477","full_name":"dearopen/django-easy-scraper","owner":"dearopen","description":"Django apps to scrape data from web page easily","archived":false,"fork":false,"pushed_at":"2020-11-23T19:02:01.000Z","size":16,"stargazers_count":2,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-07T15:28:39.789Z","etag":null,"topics":["automation","django","django-rest-framework","python","python3","webcrawler","webcrawling","webscraper","webscraping"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/django-easy-scraper/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dearopen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-17T08:16:45.000Z","updated_at":"2022-01-28T11:53:24.000Z","dependencies_parsed_at":"2022-09-01T04:53:06.923Z","dependency_job_id":null,"html_url":"https://github.com/dearopen/django-easy-scraper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dearopen/django-easy-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dearopen%2Fdjango-easy-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dearopen%2Fdjango-easy-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dearopen%2Fdjango-easy-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dearopen%2Fdjango-easy-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dearopen","download_url":"https://codeload.github.com/dearopen/django-easy-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dearopen%2Fdjango-easy-scraper/sbom","scorecard":{"id":330912,"data":{"date":"2025-08-11","repo":{"name":"github.com/dearopen/django-easy-scraper","commit":"eeddf2c6da220ac7c19ed0ea0b5173000c9e4227"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/14 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-18T03:35:41.210Z","repository_id":57419874,"created_at":"2025-08-18T03:35:41.210Z","updated_at":"2025-08-18T03:35:41.210Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33025217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","django","django-rest-framework","python","python3","webcrawler","webcrawling","webscraper","webscraping"],"created_at":"2025-10-04T03:57:15.836Z","updated_at":"2026-05-14T12:35:05.778Z","avatar_url":"https://github.com/dearopen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Django Easy Scraper\n\nAn standalone django app that can be used/intigrated with both django and no-django application easily. the scraping mechanism is  on `Regular Expression` and `xpath` which is mean you can use what you are familiar with very easily.\n\nIt requires to install python `requests` modules\n\n\n## Install\n\n`pip install django-easy-scraper`\n\n## Basic Uses\n\nif you use regex:\n```\nfrom django_easy_scraper import scraper\n\nclass ScrapeExampleDotCom(scraper.Scraper):\n\n    regex_fields = {\n        'price': \"Write Your Regex pattern for price here\",\n        'title': \"Write your regex pattern for title here\",\n        # Like above way you can add as much fields/keys as you want\n    }\n\n```\n\nif you use xpath:\n\n```\nfrom django_easy_scraper import scraper\n\nclass ScrapeExampleDotCom(scraper.Scraper):\n\n    xpath_fields = {\n        'price': \"Write Your xpath pattern for price here\",\n        'title': \"Write your xpath pattern for title here\",\n        # Like above way you can add as much fields/keys as you want\n    }\n\n```\n\n\n### Scrape now\n```\nurl = 'www.example.com/bla-bla-details-page/\ndata = ScrapeExampleDotCom.regex_url_scraper(url)\n\nprint(data)\n```\nand the response should look like this if your regex pattern are correct:\n```\n{\n    'price': 4,\n    'title': 'an scraped title',\n}\n```\n\n`regex_url_scraper` method always gives you json response,\n\nSo if you add many regex pattern in `regex_fields`, it will give you response that number of dictionary key with result that you added in that dictionary.\n\n\n# Multiple Sites Scraping together\n\nYou don't need to call different method for different site all the time !! Just call once and scrape all, something fun, right?\n\n### Like you are gonna scrape three sites:\n\u003e www.example.com\n\n\u003e www.exampletwo.com\n\n\u003e www.examplethree.com\n\n\n\nBut how will those site product scrape automatically, it scares you ? \n\n- Wirte Regex pattern for all above site with the fields that you want to scrape:\n\n\n```\nfrom django_easy_scraper import scraper\n\nclass ScrapeExampleDotCom(scraper.Scraper):\n    regex_fields = {\n        'price': \"Write Your Regex pattern for price here\",\n        'title': \"Write your regex pattern for title here\",\n        # Like above way you can add as much fields/keys as you want\n    }\n\nclass ScrapeExampleTwo(scraper.Scraper):\n    regex_fields = {\n        'price': \"Write Your Regex pattern for price here\",\n        'title': \"Write your regex pattern for title here\",\n        # Like above way you can add as much fields/keys as you want\n    }\n\nclass ScrapeExampleThree(scraper.Scraper):\n    regex_fields = {\n        'price': \"Write Your Regex pattern for price here\",\n        'title': \"Write your regex pattern for title here\",\n        # Like above way you can add as much fields/keys as you want\n    }\n\n\n```\n\n\nYou have written regular expression for you all the site you are gonna scrape,\n\nNow it's time use our `Switch` class that will route your script/class based on the site you are gonna scrape? Cool, right ? !!\n\nIt's where the magic really begins:\n\nJust place all your class in the dictionary `switcher`.\n\n\n\u003e Important Note:\n\n`key` name should be domain name, pure domain name, no www or http or slash, dont add anything prefix/suffix\n`value` should be the class of that domain you have written for and place it's method `regex_url_scraper\n\n```\nfrom django_easy_scraper import switch\n\nclass Switch(switch.BaseSwitch):\n    switcher = {\n        'example.com': ScrapeExampleDotCom.regex_url_scraper,\n        'exampletwo.com': ScrapeExampleTwo.regex_url_scraper,\n        'examplethree.com': ScrapeExampleThree.regex_url_scraper,\n    }\n\n```\n\n\u003e If you use xpath, you have pass `xpath_scraper` instead of `regex_url_scraper`\n\nSo you have done routing your script/class based on the url it gets.\n\n### Get response data as python dictionary like above site:\n\n```\nurl = 'Any of site you have written class for the site and added in switch class'\n\nresponse = Switch.get_data(url=url, raise_exception=False)\n\nprint(response) # Will give you an object of data that you trying to scrape\n\n```\n\nSwitch class is giving you facilities to route your scraping class automacally based whatever site link pass to it's `get_data` method.\n\n`get_data` method's `raise_exception` is it handle if you want to raise excepiton when your expected fields not found\n\n\n### Got an issue ?\nPlease open an issue on our github repo: https://github.com/dearopen/django-easy-scraper\n\nDon't forget to star to this project if you like this.\n\nHappy Scraping !!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdearopen%2Fdjango-easy-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdearopen%2Fdjango-easy-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdearopen%2Fdjango-easy-scraper/lists"}