https://github.com/vinhlh/frontendmasters-crawler

A demo of a serverless crawler built on AWS Lambda (scheduled tasks) and store results in S3
https://github.com/vinhlh/frontendmasters-crawler

aws crawler lambda s3 serverless

Last synced: 8 months ago
JSON representation

A demo of a serverless crawler built on AWS Lambda (scheduled tasks) and store results in S3

• Host: GitHub
• URL: https://github.com/vinhlh/frontendmasters-crawler
• Owner: vinhlh
• License: mit
• Created: 2018-05-09T17:12:29.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2022-06-25T04:19:07.000Z (almost 4 years ago)
• Last Synced: 2025-06-22T03:35:10.720Z (9 months ago)
• Topics: aws, crawler, lambda, s3, serverless
• Language: JavaScript
• Homepage:
• Size: 274 KB
• Stars: 10
• Watchers: 1
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# FrontendMasters Crawler

A serverless crawler built on AWS Lambda (scheduled tasks) and store results in S3.







## Usage

- Setup your infrastructure on AWS by `make apply`.

- Make your own event by copying `event.example.json` to `event.json` in `functions/fm_crawler/` and updating its content to yours.

- Install dependencies by running `yarn` in `functions/fm_crawler/`.

- Sync course list into S3 by `make upload`. This file works as a dead simple storage for storing course list, download state.

- Deploy by `apex deploy`.

- Invoke lambda function by `make run-lambda`.

## Development

In order to execute the crawler in dev environment, running `make run`.

## Player

You can use [fm-player](https://github.com/vinhlh/fm-player) to play the content stored in S3.

## Note

This project is just a demo for using serverless service. We encourage you guys to buy Frontend Masters's subscription instead.