{"id":20898478,"url":"https://github.com/achannarasappa/locust","last_synced_at":"2025-05-13T01:31:02.930Z","repository":{"id":36803875,"uuid":"166658286","full_name":"achannarasappa/locust","owner":"achannarasappa","description":"Distributed web data discovery and collection framework built for serverless","archived":false,"fork":false,"pushed_at":"2022-12-30T20:33:07.000Z","size":786,"stargazers_count":16,"open_issues_count":14,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-13T19:37:20.333Z","etag":null,"topics":["aws-lambda","crawler","locust","scraping","serverless"],"latest_commit_sha":null,"homepage":"https://locust.dev/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/achannarasappa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-20T12:24:23.000Z","updated_at":"2023-12-23T20:31:47.000Z","dependencies_parsed_at":"2023-01-17T04:57:07.458Z","dependency_job_id":null,"html_url":"https://github.com/achannarasappa/locust","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achannarasappa%2Flocust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achannarasappa%2Flocust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achannarasappa%2Flocust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achannarasappa%2Flocust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/achannarasappa","download_url":"https://codeload.github.com/achannarasappa/locust/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225165631,"owners_count":17431341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-lambda","crawler","locust","scraping","serverless"],"created_at":"2024-11-18T11:10:50.578Z","updated_at":"2024-11-18T11:10:51.108Z","avatar_url":"https://github.com/achannarasappa.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/achannarasappa/locust.svg?branch=master)](https://travis-ci.com/achannarasappa/locust) [![Coverage Status](https://coveralls.io/repos/github/achannarasappa/locust/badge.svg?branch=master)](https://coveralls.io/github/achannarasappa/locust?branch=master)\n\n\u003cp align=\"center\"\u003e\n\u003cimg align=\"center\" width=\"100\" src=\"https://locust.dev/img/locust-logo.svg\" /\u003e\n\u003c/p\u003e\n\u003ch1 align=\"center\"\u003eLocust\u003c/h2\u003e\n\u003cp align=\"center\"\u003e\nDistributed web data discovery and collection framework\n\u003c/p\u003e\n\n## Quick Start\n\n```\nnpm install @achannarasappa/locust\n```\n\n## Features\n\n* Configuration driven jobs\n* Distributed execution model to support serverless architectures\n* Handle client-side JavaScript execution\n* Data extraction using CSS selectors\n* Depth-based stop condition along with support for custom stop conditions\n* Robust dev tooling with [locust-cli](https://github.com/achannarasappa/locust-cli) to build and test jobs\n\n## Use Cases\n\n* Web indexing (i.e. web crawling)\n* Web data extraction (i.e. web scraping)\n\n## Reference\n\n* Documentation\n  * [Quick start guide](https://locust.dev/docs/getting_started)\n  * [API](https://locust.dev/docs/api)\n  * [CLI](https://locust.dev/docs/cli)\n* [Examples](https://github.com/achannarasappa/locust-examples)\n* Related\n  * [locust-cli](https://github.com/achannarasappa/locust-cli)\n  * [locust-aws-terraform](https://github.com/achannarasappa/locust-aws-terraform)\n  * [locust-website](https://github.com/achannarasappa/locust-website)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachannarasappa%2Flocust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fachannarasappa%2Flocust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachannarasappa%2Flocust/lists"}