Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/achannarasappa/locust

Distributed web data discovery and collection framework built for serverless
https://github.com/achannarasappa/locust

aws-lambda crawler locust scraping serverless

Last synced: about 1 month ago
JSON representation

Distributed web data discovery and collection framework built for serverless

Host: GitHub
URL: https://github.com/achannarasappa/locust
Owner: achannarasappa
License: mit
Created: 2019-01-20T12:24:23.000Z (almost 6 years ago)
Default Branch: master
Last Pushed: 2022-12-30T20:33:07.000Z (almost 2 years ago)
Last Synced: 2024-11-13T19:37:20.333Z (about 1 month ago)
Topics: aws-lambda, crawler, locust, scraping, serverless
Language: JavaScript
Homepage: https://locust.dev/
Size: 768 KB
Stars: 16
Watchers: 3
Forks: 1
Open Issues: 14
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        [![Build Status](https://travis-ci.com/achannarasappa/locust.svg?branch=master)](https://travis-ci.com/achannarasappa/locust) [![Coverage Status](https://coveralls.io/repos/github/achannarasappa/locust/badge.svg?branch=master)](https://coveralls.io/github/achannarasappa/locust?branch=master)







Locust



Distributed web data discovery and collection framework



## Quick Start

```

npm install @achannarasappa/locust

```

## Features

* Configuration driven jobs

* Distributed execution model to support serverless architectures

* Handle client-side JavaScript execution

* Data extraction using CSS selectors

* Depth-based stop condition along with support for custom stop conditions

* Robust dev tooling with [locust-cli](https://github.com/achannarasappa/locust-cli) to build and test jobs

## Use Cases

* Web indexing (i.e. web crawling)

* Web data extraction (i.e. web scraping)

## Reference

* Documentation

  * [Quick start guide](https://locust.dev/docs/getting_started)

  * [API](https://locust.dev/docs/api)

  * [CLI](https://locust.dev/docs/cli)

* [Examples](https://github.com/achannarasappa/locust-examples)

* Related

  * [locust-cli](https://github.com/achannarasappa/locust-cli)

  * [locust-aws-terraform](https://github.com/achannarasappa/locust-aws-terraform)

  * [locust-website](https://github.com/achannarasappa/locust-website)