https://github.com/zalbia/spt-web-crawler

https://github.com/zalbia/spt-web-crawler

Last synced: 7 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/zalbia/spt-web-crawler
• Owner: zalbia
• Created: 2022-08-28T05:02:37.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2022-08-31T05:20:48.000Z (about 2 years ago)
• Last Synced: 2023-10-18T08:43:56.734Z (about 1 year ago)
• Language: Scala
• Size: 25.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.MD

Awesome Lists containing this project

README

        
# Scala Programming Test - Web Crawler

Runs a ZIO HTTP server on port 8080 that exposes an API endpoint, `POST /api/crawl` that takes a JSON object containing

a list of URL's, crawls their text content, and returns the

crawled data or a message detailing a crawling error.

## Running

Run the server with `sbt run`.

Run the test suite with `sbt test`.

## Running on Docker

To build a docker image and publish locally in SBT:

    sbt> docker:publishLocal

To run built image as a container:

    sbt> docker run -d -p 0.0.0.0:8080:8080 --name "spt-web-crawler" spt-web-crawler:0.1.0-SNAPSHOT

Example command to test crawler:

    curl -d '{"urls":["https://google.com","https://github.com"]}' -X POST http://localhost:8080/api/crawl