https://github.com/osamikoyo/geass
web crawler for you, with some api function and configuration
https://github.com/osamikoyo/geass
docker go golang searching url web-crawler web-crawling web-scraping
Last synced: 11 months ago
JSON representation
web crawler for you, with some api function and configuration
- Host: GitHub
- URL: https://github.com/osamikoyo/geass
- Owner: osamikoyo
- License: bsd-2-clause
- Created: 2025-02-04T01:34:04.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-08T12:21:57.000Z (over 1 year ago)
- Last Synced: 2025-03-14T13:18:19.909Z (about 1 year ago)
- Topics: docker, go, golang, searching, url, web-crawler, web-crawling, web-scraping
- Language: Go
- Homepage:
- Size: 32.8 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Geass
web crawler for you
## Usage
# Nead
* Go >= 1.23
* Docker-compose
* task(recomended)
### Instalization manual
```bash
git clone github/osamikoyo/geass
cd geass
task run
```
### Docker
```bash
git clone github/osamikoyo/geass
task run
```
### Handler
```bash
curl "localhost:PORT_IN_CONFIG/get/content?url=EXEMPLE_URL"
```
### output:
```json
{
"url": "https://example.com",
"title": "Example Domain",
"meta_description": "This domain is for use in illustrative examples in documents.",
"content": {
"headings": ["Example Domain", "This domain is for use..."],
"text": "This domain is for use in illustrative examples in documents...",
"images": [
{
"src": "https://example.com/image.png",
"alt": "Example Image"
}
]
},
"links": [
{
"text": "More information...",
"href": "https://www.iana.org/domains/example"
}
],
"technical": {
"status_code": 200,
"content_type": "text/html",
"last_modified": "2023-10-01T12:00:00Z"
},
"metadata": {
"language": "en",
"canonical": "https://example.com",
"robots": "index, follow"
}
}
```
## Handler
```bash
curl "localhost:PORT_IN_CONFIG/get/urls?url=EXEMPLE_URL"
```
### output
```
http://exemplenumber1
http://exemplenumber2
http://exemplenumber3
http://exemplenumber4
```