Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/YektaDev/Krawler
A configurable HTML Crawler written in Kotlin (JVM), powered by Coroutines, Kotlin Serialization (JSON), Ktor Client, Exposed, and SQLite.
https://github.com/YektaDev/Krawler
crawl crawler crawlers crawling
Last synced: 5 days ago
JSON representation
A configurable HTML Crawler written in Kotlin (JVM), powered by Coroutines, Kotlin Serialization (JSON), Ktor Client, Exposed, and SQLite.
- Host: GitHub
- URL: https://github.com/YektaDev/Krawler
- Owner: YektaDev
- License: apache-2.0
- Created: 2023-11-12T06:09:13.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2023-11-18T18:15:18.000Z (about 1 year ago)
- Last Synced: 2024-10-08T08:05:02.680Z (4 months ago)
- Topics: crawl, crawler, crawlers, crawling
- Language: Kotlin
- Homepage:
- Size: 3.6 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Krawler: Asynchronous Kotlin Crawler 🚀
## Overview
Krawler is a fully configurable and asynchronous HTML Crawler written in Kotlin (JVM). Powered by **Coroutines**,
**Kotlin Serialization (JSON)**, **Ktor Client**, **Exposed**, **SQLite**, and **SQLite JDBC**, Krawler provides a way
to easily scrape HTML webpages.
## Features
- **Asynchronous Processing**: Utilizing Kotlin's coroutines, Krawler is designed for high-performance, concurrent web
crawling.
- **Configurability**: Krawler is highly customizable through the `krawler_config.json` file, placed at the project
path.
- **Extensive Logging**: Verbose logs can be enabled via the configuration file.
- **Persisting Errors**: Errors during the crawling process are stored in the `CrawlErrors` table (with the necessary
metadata) and printed to the standard output.
## Database Schema
Krawler uses the following tables to persist data:
```
CrawlActivities : IntIdTable() {
varchar("sessionId", 100)
long("atEpochSeconds")
varchar("type", 50)
}
CrawlErrors : IntIdTable() {
varchar("sessionId", 100)
long("atEpochSeconds")
text("url")
text("error")
}
CrawlingStates : IntIdTable() {
varchar("sessionId", 100)
text("url")
integer("depth")
long("priority")
}
Webpages : IntIdTable() {
varchar("sessionId", 100)
long("atEpochSeconds")
text("url")
text("html")
}
```
## Configuration
Krawler is highly customizable through the `krawler_config.json` file, placed at the project path. Below is a sample
configuration containing all settings:
```json
{
"seeds": [
"https://en.wikipedia.org/wiki/NASA"
],
"filter": {
"#": "dev.yekta.krawler.model.CrawlingFilter.Whitelist",
"allowPatterns": [
"https://en\\.wikipedia\\.org/wiki/.*"
]
},
"depth": 8,
"maxPages": 100,
"maxPageSizeKb": null,
"concurrentConnections": 16,
"verbose": true,
"shouldFollowRedirects": true,
"userAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"connectTimeoutMs": 6000,
"readTimeoutMs": 6000,
"retriesOnServerError": 0,
"customHeaders": null
}
```
+ **`seeds`**: Starting URLs for crawling.
+ **`filter`**: Crawling filter configuration, either **Whitelist** or **Blacklist**.
+ **`depth`**: Maximum depth of crawling.
+ **`maxPages`**: Maximum number of pages to crawl.
+ **`maxPageSizeKb`**: Maximum page size in kilobytes.
+ **`concurrentConnections`**: Number of concurrent connections for crawling.
+ **`verbose`**: Enable verbose logging.
+ **`shouldFollowRedirects`**: Specify if redirects should be followed.
+ **`userAgent`**: User agent string for HTTP requests.
+ **`connectTimeoutMs`**: Connection timeout in milliseconds.
+ **`readTimeoutMs`**: Read timeout in milliseconds.
+ **`retriesOnServerError`**: Number of retries on server errors (`5xx`).
+ **`customHeaders`**: Additional custom headers for HTTP requests.
## Good Next Steps
Things that would benefit Krawler the most:
+ Implementing Pause/Resume
+ _Hint:_ The `UrlPool` is the only state that isn't currently being persisted but needs to be, in order to be able
to restore paused sessions.
+ Config: `respectRobotsTxt: Boolean`
+ Config: `consecutiveErrorsToPause: Int?`
## Disclaimer
Krawler was conceived and brought to life over a weekend, starting as a pet project. It was initially planned to be made
as a component of the coursework for the Web & Search Engines course at Yazd University, then growing exponentially due
to a sudden desire to make a "good thing" out of it! It's important to note that, no explicit guarantees are extended
regarding its correctness of functionality, support, or any other aspect. With that in mind, happy Krawling!
## License
Please **refer to [LICENSE](./LICENSE)** to view the project's license.