Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/karthikuj/sasori

Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.
https://github.com/karthikuj/sasori

automation crawler crawling dast dynamic endpoint-discovery infosec puppeteer scraping security

Last synced: 4 days ago
JSON representation

Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.

Host: GitHub
URL: https://github.com/karthikuj/sasori
Owner: karthikuj
License: mit
Created: 2023-11-11T04:50:07.000Z (12 months ago)
Default Branch: master
Last Pushed: 2024-07-23T11:33:14.000Z (4 months ago)
Last Synced: 2024-10-11T00:46:44.743Z (26 days ago)
Topics: automation, crawler, crawling, dast, dynamic, endpoint-discovery, infosec, puppeteer, scraping, security
Language: JavaScript
Homepage:
Size: 4.89 MB
Stars: 129
Watchers: 4
Forks: 13
Open Issues: 5
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

Sasori: Dynamic Web Crawler

Sasori Demo

## Project Description:

Sasori is a powerful and flexible dynamic web crawler built on Puppeteer. It allows you to automate the crawling of web applications, even those behind authentication, offers seamless integration with security testing tools like Zaproxy or Burp Suite and provides customizable configurations for enhanced flexibility.

## Features

- **Dynamic Crawling:** Sasori excels at crawling dynamic web applications, handling AJAX-loaded content, and interacting with complex user interfaces.

- **Authentication Support:** Easily spider applications behind authentication barriers by passing the puppeteer recording for the login sequence.

- **Proxy Integration:** Sasori provides the option to set up a proxy server, allowing you to route requests through tools like Zaproxy or Burp Suite for security testing.

- **State-Based Navigation:** The project is designed around a state-based system, keeping track of URLs, DOM structures, and interactable elements for efficient crawling.

- **Efficient Endpoint Coverage:** Utilizes efficient algorithms for intelligent crawling, ensuring coverage of more endpoints while maintaining speed.

- **Crawl Customization:** Allows you to customize what elements to interact with to target specific use cases.

## Getting Started:

To get started with Sasori, follow these steps:

### Recommended

1. Install the package globally:

```bash
npm install -g sasori-crawl
```

2. Create Sasori's configuration file:

```bash
sasori init
```

3. Edit the configuration file. Check [Configuration](#configuration)

4. Run Sasori:

```bash
sasori start --config /path/to/config.json
```

### Alternative

1. Clone the repository:

```bash
git clone https://github.com/karthikuj/sasori.git
```

2. Install dependencies:

```bash
cd sasori
npm install
```

3. Configure Sasori by editing the configuration file in the `config` directory. Check [Configuration](#configuration).

4. Run Sasori:

```bash
node . start --config ./config/config.json
```

## Configuration

The Sasori configuration consists of two main sections: `browser` and `crawler`. Each section contains specific settings to customize the behavior of the crawler and the browser used for crawling.

### Browser Configuration

The `browser` section contains settings related to the browser used by the crawler.

- **headless**: (boolean) Specifies whether the browser should run in headless mode. Default: `false`.
- **maximize**: (boolean) Specifies whether the browser window should be maximized. Default: `false`.
- **proxy**: (object) Configuration for proxy settings.
- **enabled**: (boolean) Specifies whether proxy is enabled.
- **host**: (string) Hostname of the proxy server. Required if `enabled` is `true`.
- **port**: (integer) Port of the proxy server. Required if `enabled` is `true`.
- **instances**: (integer): Number of browser instances to use simultaneously for crawling.

Example:

```json
{
"browser": {
"headless": true,
"maximize": false,
"proxy": {
"enabled": true,
"host": "proxy.example.com",
"port": 8080
},
"instances": 4
}
}
```

### Crawler Configuration

The `crawler` section contains settings related to the behavior of the crawler.

- **entryPoint**: (string) URL of the entry point from where the crawling starts. Required.
- **eventTimeout**: (integer) Timeout (in milliseconds) for waiting for events during crawling. Required.
- **navigationTimeout**: (integer) Timeout (in milliseconds) for waiting for navigation to complete during crawling. Required.
- **eventWait**: (integer) Timeout (in milliseconds) for waiting between events during crawling. Required.
- **maxDuration**: (integer) Maximum duration (in milliseconds) for the crawling process. `0` means crawl indefinitely. Required.
- **elements**: (array of css paths) List of HTML css paths to click during crawling. Required.
- **maxChildren**: (integer) Maximum number of child elements to crawl from each parent state. `0` means infinite children. Required.
- **maxDepth**: (integer) Maximum depth of the crawling process. `0` means infinite depth. Required.
- **authentication**: (object) Authentication settings for crawler.
- **basicAuth**: (object) Configuration for HTTP basic authentication.
- **enabled**: (boolean) Specifies whether basic authentication is enabled.
- **username**: (string) Username for basic authentication. Required if `enabled` is `true`.
- **password**: (string) Password for basic authentication. Required if `enabled` is `true`.
- **recorderAuth**: (object) Configuration for recorder based authentication.
- **enabled**: (boolean) Specifies whether recorder based authentication is enabled.
- **pptrRecording**: (string) Path to [Puppeteer recording](#creating-a-puppeteer-recording) for authentication. Required if `enabled` is `true`.

Example:

```json
{
"crawler": {
"entryPoint": "https://example.com",
"eventTimeout": 10000,
"navigationTimeout": 30000,
"eventWait": 1000,
"maxDuration": 600000,
"elements": ["a", "button", "input[type=\"submit\"]"],
"maxChildren": 10,
"maxDepth": 5,
"authentication": {
"basicAuth": {
"enabled": true,
"username": "user",
"password": "password"
},
"recorderAuth": {
"enabled": false,
"pptrRecording": "/path/to/pptr/recording.json"
}
},
"includeRegexes": ["https?://example\\.com(?:/.*|)"],
"excludeRegexes": ["^.*\\.pdf$", "https?://example\\.com/logout"]
}
}
```

### Creating a puppeteer recording
1. Open DevTools in Google Chrome and click on the 3 dots icon in the top-right corner.

DevTools Options

2. Go to `More tools` > `Recorder`.

Recorder

3. Click on `Create a new recording`.

4. Give a name to your recording and then click on `Start recording`.

5. Create the recording and then click on `End recording`.

Create Recording

7. If you want to execute a custom JS script while authenticating, add another step in the auth flow, set the type as `customStep`, set the name as `executeScript` and lastly in the `parameters` field add a JSON object with a `target` key and the value should be the script that you want to run.

Create Recording

6. Lastly export the recording by clicking on the downward arrow and then choosing `JSON` as the type.

Export Recording

## Contributing:

Contributions to Sasori are welcome! If you encounter any bugs, have feature requests, or would like to contribute code improvements, please follow the guidelines in the [CONTRIBUTING.md](CONTRIBUTING.md) file.

## License:

This project is licensed under the [MIT License](LICENSE).