{"id":16208916,"url":"https://github.com/stephanebruckert/gocrawl","last_synced_at":"2026-07-30T10:31:12.096Z","repository":{"id":71168205,"uuid":"73859191","full_name":"stephanebruckert/GoCrawl","owner":"stephanebruckert","description":"Crawl every pages and assets of a web domain","archived":false,"fork":false,"pushed_at":"2016-11-21T19:17:03.000Z","size":38,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-13T22:18:06.915Z","etag":null,"topics":["crawler","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stephanebruckert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-15T22:02:42.000Z","updated_at":"2018-07-23T18:46:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"ebd32b44-cd49-44ba-88a3-b9d142a818e6","html_url":"https://github.com/stephanebruckert/GoCrawl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephanebruckert%2FGoCrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephanebruckert%2FGoCrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephanebruckert%2FGoCrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephanebruckert%2FGoCrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stephanebruckert","download_url":"https://codeload.github.com/stephanebruckert/GoCrawl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247726465,"owners_count":20985907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","python"],"created_at":"2024-10-10T10:26:24.328Z","updated_at":"2025-10-16T07:39:17.493Z","avatar_url":"https://github.com/stephanebruckert.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoCrawl\n\n## How to\n\nAll these commands can be ran from the current folder.\n\n### Prerequisites\n\n1. Make sure you run Python2.7\n2. `easy_install pip`\n3. `pip install --upgrade pip`\n\n### Setting up a virtual environment (optional)\n\n1. `pip install virtualenv`\n2. `virtualenv env` to create a project-owned environment\n3. `source env/bin/activate` to activate it\n\n### Install required modules\n\n`make` or `pip install -r ./requirements.txt`\n\n### Run with options\n\n    $ python main.py -h\n    usage: main.py [-h] -L LINK [--silent] [-W WAIT]\n\n    GoCrawl\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      -L LINK, --link LINK  Entry point URL\n      --silent              Silent mode\n      -W WAIT, --wait WAIT  Minimum wait time in seconds between each request\n\n#### Examples:\n\n- `python main.py -h`\n- `python main.py -L http://google.fr`\n- `python main.py -L http://google.fr --silent` to hide the progress outputs\n- `python main.py -L http://google.fr --wait 5` to wait between each requests\n\n### Tasks\n\n- `make test` to run unit tests\n- `make lint` to run linter\n\n### Search rules\n\nSearch rules follow the format:\n\n    'data_category':\n      {\n        'data_type':\n                [\n                  ['tag', 'condition_key', 'condition_value', 'source_attr'],\n                  ['tag', 'condition_key', 'condition_value', 'source_attr']\n                ]\n      }\n    },\n\nCurrent rules to retrieve links, images, javascript and stylesheets are defined in the `Parser` class:\n\n    {\n        'next': {\n            'url':    [['a', 'href', True, 'href']]\n        },\n        'assets': {\n            'images': [['img', 'src', True, 'src']],\n            'css':    [\n                        ['link', 'rel', 'stylesheet', 'href'],\n                        ['link', 'type', 'text/css', 'href'],\n                        ['link', 'rel', 'stylesheet/less', 'href'],\n                        ['link', 'rel', 'stylesheet/css', 'href']\n                      ],\n            'js':     [['script', 'src', True, 'src']]\n        }\n    }\n\n### Sample output\n\n    $ python main.py -L http://hackcss.com/\n    Crawling http://hackcss.com/...\n    http://hackcss.com/                                          Visited: 1        Remaining: 4       \n    http://hackcss.com/standard.html                             Visited: 2        Remaining: 3       \n    http://hackcss.com/dark.html                                 Visited: 3        Remaining: 2       \n    http://hackcss.com/dark-grey.html                            Visited: 4        Remaining: 1       \n    http://hackcss.com/solarized-dark.html                       Visited: 5        Remaining: 0       \n\n    Done:\n    {\n      \"failures\": {\n        \"total\": 0,\n        \"data\": []\n      },\n      \"success\": {\n        \"total\": 5,\n        \"data\": [\n          {\n            \"url\": \"http://hackcss.com/\",\n            \"assets\": {\n              \"images\": [],\n              \"css\": [\n                \"http://hackcss.com/prism.css\",\n                \"http://hackcss.com/hack.css?t=1473587248285\",\n                \"http://hackcss.com/site.css?t=1473587248285\"\n              ],\n              \"js\": [\n                \"http://hackcss.com/app.js\",\n                \"http://hackcss.com/prism.js\"\n              ]\n            }\n          },\n          {\n            \"url\": \"http://hackcss.com/standard.html\",\n            \"assets\": {\n              \"images\": [],\n              \"css\": [\n                \"http://hackcss.com/site.css?t=1473587248285\",\n                \"http://hackcss.com/prism.css\",\n                \"http://hackcss.com/hack.css?t=1473587248285\",\n                \"http://hackcss.com/standard.css?t=1473587248285\"\n              ],\n              \"js\": [\n                \"http://hackcss.com/app.js\",\n                \"http://hackcss.com/prism.js\"\n              ]\n            }\n          },\n          {\n            \"url\": \"http://hackcss.com/dark.html\",\n            \"assets\": {\n              \"images\": [],\n              \"css\": [\n                \"http://hackcss.com/site-dark.css?t=1473587248285\",\n                \"http://hackcss.com/prism.css\",\n                \"http://hackcss.com/hack.css?t=1473587248285\",\n                \"http://hackcss.com/dark.css?t=1473587248285\",\n                \"http://hackcss.com/site.css?t=1473587248285\"\n              ],\n              \"js\": [\n                \"http://hackcss.com/app.js\",\n                \"http://hackcss.com/prism.js\"\n              ]\n            }\n          },\n          {\n            \"url\": \"http://hackcss.com/dark-grey.html\",\n            \"assets\": {\n              \"images\": [],\n              \"css\": [\n                \"http://hackcss.com/site-dark.css?t=1473587248285\",\n                \"http://hackcss.com/site.css?t=1473587248285\",\n                \"http://hackcss.com/prism.css\",\n                \"http://hackcss.com/hack.css?t=1473587248285\",\n                \"http://hackcss.com/dark-grey.css?t=1473587248285\"\n              ],\n              \"js\": [\n                \"http://hackcss.com/app.js\",\n                \"http://hackcss.com/prism.js\"\n              ]\n            }\n          },\n          {\n            \"url\": \"http://hackcss.com/solarized-dark.html\",\n            \"assets\": {\n              \"images\": [],\n              \"css\": [\n                \"http://hackcss.com/solarized-dark.css?t=1473587248285\",\n                \"http://hackcss.com/site-dark.css?t=1473587248285\",\n                \"http://hackcss.com/prism.css\",\n                \"http://hackcss.com/hack.css?t=1473587248285\",\n                \"http://hackcss.com/site.css?t=1473587248285\"\n              ],\n              \"js\": [\n                \"http://hackcss.com/app.js\",\n                \"http://hackcss.com/prism.js\"\n              ]\n            }\n          }\n        ]\n      }\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanebruckert%2Fgocrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephanebruckert%2Fgocrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanebruckert%2Fgocrawl/lists"}