{"id":15044305,"url":"https://github.com/elastic/crawler","last_synced_at":"2025-04-04T19:07:07.728Z","repository":{"id":243030832,"uuid":"780912711","full_name":"elastic/crawler","owner":"elastic","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-01T10:56:52.000Z","size":158562,"stargazers_count":161,"open_issues_count":36,"forks_count":16,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-01T11:34:53.779Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"docs/SECURITY.md","support":"docs/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-02T11:57:06.000Z","updated_at":"2025-04-01T10:56:54.000Z","dependencies_parsed_at":"2024-09-17T17:58:19.909Z","dependency_job_id":"5481bf37-54b3-45a4-858f-135cf9b48011","html_url":"https://github.com/elastic/crawler","commit_stats":{"total_commits":105,"total_committers":8,"mean_commits":13.125,"dds":"0.20952380952380956","last_synced_commit":"4a124a1fd8b484398bea849a53fec1643e4c2d75"},"previous_names":["elastic/crawler"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234921,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:50:25.205Z","updated_at":"2025-04-04T19:07:07.692Z","avatar_url":"https://github.com/elastic.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elastic Open Web Crawler\n\nThis repository contains code for the Elastic Open Web Crawler.\nOpen Crawler enables users to easily ingest web content into Elasticsearch.\n\n\u003e [!IMPORTANT]\n\u003e _The Open Crawler is currently in **beta**_.\nBeta features are subject to change and are not covered by the support SLA of generally available (GA) features.\nElastic plans to promote this feature to GA in a future release.\n\n### ES Version Compatibility\n\n| Elasticsearch | Open Crawler       |\n|---------------|--------------------|\n| `8.x`         | `v0.2.x`           |\n| `9.x`         | `v0.2.1` and above |\n\n### Quickstart\n\nThe docker image can be found at https://www.docker.elastic.co/r/integrations/crawler.\n\nThe following will run the latest Crawler in a Docker container using a docker-compose file.\nIt mounts the `config` directory as a shared volume, so any changes made to files there are automatically accessible to Crawler.\n\n1. Run the init script, which will output the current Crawler version when complete:\n    ```bash\n    git clone git@github.com:elastic/crawler.git \u0026\u0026 \\\n    cd crawler \u0026\u0026 \\\n    docker-compose up -d \u0026\u0026 \\\n    cp config/examples/simple.yml config/my-crawler.yml \u0026\u0026 \\\n    docker exec -it crawler bin/crawler version\n    ```\n2. Update the new config file `config/my-crawler.yml` if necessary\n3. Run a crawl:\n    ```bash\n    docker exec -it crawler bin/crawler crawl config/my-crawler.yml\n    ```\n\n### User workflow\n\nIndexing web content with the Open Crawler requires:\n\n1. Running an instance of Elasticsearch (on-prem, cloud, or serverless)\n2. Running the official Docker image (see [Setup](#setup))\n3. Configuring a crawler config file (see [Configuring crawlers](#configuring-crawlers))\n4. Using the CLI to begin a crawl job (see [CLI commands](#cli-commands))\n\n### Execution logic\n\nCrawler runs crawl jobs on command, based on config files in the `config` directory.\nEach URL endpoint found during the crawl will result in one document to be indexed into Elasticsearch.\nCrawler performs crawl jobs in a multithreaded environment, where one thread will be used to visit one URL endpoint.\n\nCrawls are performed in two stages:\n\n#### 1. Primary crawl\n\nBeginning with URLs included as `seed_urls`, the Crawler begins crawling web content.\nWhile crawling, each link it encounters will be added to the crawl queue, unless the link should be ignored due to [crawl rules](./docs/features/CRAWL_RULES.md) or [crawler directives](./docs/features/CRAWLER_DIRECTIVES.md).\n\nThe crawl results from visiting these webpages are added to a pool of results.\nThese are indexed into Elasticsearch using the `_bulk` API once the pool reaches the configured threshold.\n\n#### 2. Purge crawl\n\nAfter a primary crawl is completed, Crawler will then fetch every doc from the associated index that was not encountered during the primary crawl.\nIt does this through comparing the `last_crawled_at` date on the doc to the primary crawl's start time.\nIf `last_crawled_at` is earlier than the start time, that means the webpage was not updated during the primary crawl and should be added to the purge crawl.\n\nCrawler then re-crawls all of these webpages.\nIf a webpage is still accessible, Crawler will update its Elasticsearch doc.\nA webpage can be inaccessible due to any of the following reasons:\n\n- Updated [crawl rules](./docs/features/CRAWL_RULES.md) in the configuration file that now exclude the URL\n- Updated [crawler directives](./docs/features/CRAWLER_DIRECTIVES.md) on the server or webpage that now exclude the URL\n- Non-`200` response from the webserver\n\nAt the end of the purge crawl, all docs in the index that were not updated during either the primary crawl or the purge crawl are deleted.\n\n### Setup\n\n#### Prerequisites\n\nA running instance of Elasticsearch is required to index documents into.\nIf you don't have this set up yet, you can sign up for an [Elastic Cloud free trial](https://www.elastic.co/cloud/cloud-trial-overview) or check out the [quickstart guide for Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/master/quickstart.html).\n\n#### Connecting to Elasticsearch\n\nOpen Crawler will attempt to use the `_bulk` API to index crawl results into Elasticsearch.\nTo facilitate this connection, Open Crawler needs to have either an API key or a username/password configured to access the Elasticsearch instance.\nIf using an API key, ensure that the API key has read and write permissions to access the index configured in `output_index`.\n\n- [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html) for managing API keys for more details\n- [elasticsearch.yml.example](config/elasticsearch.yml.example) file for all of the available Elasticsearch configurations for Crawler\n\n\u003cdetails\u003e\n  \u003csummary\u003eCreating an API key\u003c/summary\u003e\n  Here is an example of creating an API key with minimal permissions for Open Crawler.\n  This will return a JSON with an `encoded` key.\n  The value of `encoded` is what Open Crawler can use in its configuration.\n\n  ```bash\n  POST /_security/api_key\n  {\n    \"name\": \"my-api-key\",\n    \"role_descriptors\": { \n      \"my-crawler-role\": {\n        \"cluster\": [\"all\"],\n        \"indices\": [\n          {\n            \"names\": [\"my-crawler-index-name\"],\n            \"privileges\": [\"monitor\"]\n          }\n        ]\n      }\n    },\n    \"metadata\": {\n      \"application\": \"my-crawler\"\n    }\n  }\n  ```\n\u003c/details\u003e\n\n#### Running Open Crawler with Docker\n\n\u003e [!IMPORTANT]\n\u003e **Do not trigger multiple crawl jobs that reference the same index simultaneously.**\nA single crawl execution can be thought of as a single crawler.\nEven if two crawl executions share a configuration file, the two crawl processes will not communicate with each other.\nTwo crawlers simultaneously interacting with a single index can lead to data loss.\n\n1. Run the official Docker image through the docker-compose file `docker-compose up -d`\n    - `-d` allows the container to run \"detached\" so you don't have to dedicate a terminal window to it\n2. Confirm that CLI commands are working `docker exec -it crawler bin/crawler version` \n3. Create a config file for your crawler\n4. See [Configuring crawlers](#configuring-crawlers) for next steps.\n\n#### Running Open Crawler from source\n\n\u003e [!TIP]\n\u003e We recommend running from source only if you are actively developing Open Crawler.\n\n\u003cdetails\u003e\n  \u003csummary\u003eInstructions for running from source\u003c/summary\u003e\n  ℹ️ Open Crawler uses both JRuby and Java.\n  We recommend using version managers for both.\n  When developing Open Crawler we use \u003cb\u003erbenv\u003c/b\u003e and \u003cb\u003ejenv\u003c/b\u003e.\n  There are instructions for setting up these env managers here:\n\n  - [Official documentation for installing jenv](https://www.jenv.be/)\n  - [Official documentation for installing rbenv](https://github.com/rbenv/rbenv?tab=readme-ov-file#installation)\n\n  1. Clone the repository: `git clone https://github.com/elastic/crawler.git`\n  2. Go to the root of the Open Crawler directory and check the expected Java and Ruby versions are being used:\n      ```bash\n      # should output the same version as `.ruby-version`\n      $ ruby --version\n\n      # should output the same version as `.java-version`\n      $ java -version\n      ```\n\n  3. If the versions seem correct, you can install dependencies:\n      ```bash\n      $ make install\n      ```\n\n     You can also use the env variable `CRAWLER_MANAGE_ENV` to have the install script automatically check whether `rbenv` and `jenv` are installed, and that the correct versions are running on both:\n     Doing this requires that you use both `rbenv` and `jenv` in your local setup.\n\n      ```bash\n      $ CRAWLER_MANAGE_ENV=true make install\n      ```\n\u003c/details\u003e\n\n### Configuring Crawlers\n\nIf you ran Crawler using `docker-compose up`, config YAML files should be placed in your _local_ `./config` directory.\nThis is because the Crawler docker image uses a mounted volume to connect your local `./config` to the Crawler container's `/app/config` directory.\nThis means any edits you make to config files in the `./config` directory are immediately accessible to the running Crawler image.\n\nCrawler has template configuration files that contain every configuration available.\n\n- [config/crawler.yml.example](config/crawler.yml.example)\n- [config/elasticsearch.yml.example](config/elasticsearch.yml.example)\n\nTo use these files, make a copy locally without the `.example` suffix.\nThen remove the `#` comment-out characters from the configurations that you need.\n\nCrawler can be configured using two config files, a Crawler configuration and an Elasticsearch configuration.\nThe Elasticsearch configuration file is optional.\nIt exists to allow users with multiple crawlers to only need a single Elasticsearch configuration.\nSee [CONFIG.md](docs/CONFIG.md) for more details on these files.\n\n### Running a Crawl Job\n\nOnce everything is configured, you can run a crawl job using the CLI:\n\n```bash\n$ docker exec -it crawler bin/crawler crawl path/to/my-crawler.yml\n```\n\n### Scheduling Recurring Crawl Jobs\n\nCrawl jobs can also be scheduled to recur.\nScheduled crawl jobs run until terminated by the user.\n\nThese schedules are defined through a cron expression.\nThis expression needs to be included in the Crawler config file.\nYou can use the tool https://crontab.guru to test different cron expressions.\nCrawler supports all standard cron expressions.\n\nSee an example below for a crawl schedule that will execute once every 30 minutes.\n\n```yaml\ndomains:\n  - url: \"https://elastic.co\"\nschedule:\n  pattern: \"*/30 * * * *\" # run every 30th minute\n```\n\nThen, use the CLI to then begin the crawl job schedule:\n\n```bash\ndocker exec -it crawler bin/crawler schedule path/to/my-crawler.yml\n```\n\n**Scheduled crawl jobs from a single execution will not overlap.**\nScheduled jobs will also not wait for existing jobs to complete.\nIf a crawl job is already in progress when another schedule is triggered, the job will be dropped.\nFor example, if you have a schedule that triggers at every hour, but your crawl job takes 1.5 hours to complete, the crawl schedule will effectively trigger on every 2nd hour.\n\n**Executing multiple crawl schedules _can_ cause overlap**.\nBe wary of executing multiple schedules against the same index.\nAs with ad-hoc triggered crawl jobs, two crawlers simultaneously interacting with a single index can lead to data loss.\n\n### Crawler Document Schema and Mappings\n\nSee [DOCUMENT_SCHEMA.md](docs/DOCUMENT_SCHEMA.md) for information regarding the Elasticsearch document schema and mappings.\n\n### CLI Commands\n\nOpen Crawler does not have a graphical user interface.\nAll interactions with Open Crawler take place through the CLI.\nWhen given a command, Open Crawler will run until the process is finished.\nOpenCrawler is not kept alive in any way between commands.\n\nSee [CLI.md](docs/CLI.md) for a full list of CLI commands available for Crawler.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Fcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fcrawler/lists"}