https://github.com/minhhungit/github-action-rss-crawler

Auto crawl RSS feeds using Github Action
https://github.com/minhhungit/github-action-rss-crawler

crawler csharp github-actions litedb netcore rss rss-crawler rss-items

Last synced: about 2 months ago
JSON representation

Auto crawl RSS feeds using Github Action

• Host: GitHub
• URL: https://github.com/minhhungit/github-action-rss-crawler
• Owner: minhhungit
• Created: 2020-08-09T17:24:45.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2024-10-29T08:04:47.000Z (2 months ago)
• Last Synced: 2024-10-29T09:22:55.699Z (2 months ago)
• Topics: crawler, csharp, github-actions, litedb, netcore, rss, rss-crawler, rss-items
• Language: HTML
• Homepage: https://minhhungit.github.io/github-action-rss-crawler/
• Size: 5.91 GB
• Stars: 32
• Watchers: 3
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-rainmana - minhhungit/github-action-rss-crawler - Auto crawl RSS feeds using Github Action (HTML)

README

        
## Rss auto crawling using Github Action

**Github Action does all these steps automatically, it run rss crawler every 4 hours**

Steps:

  - Github will pull repository, build and run crawler code (crawling code is C# (.net core), github will run it directly)

  - Read channel urls from LiteDB

  - Fetch rss feed items

  - Insert feed items into LiteDB after checking blacklist and existing

  - Generate all rss items to static page (index.html - https://minhhungit.github.io/github-action-rss-crawler/ )

  - Commit change (litedb database & index.html page) and push to this repo

  

### Workflow

```yml

on:

  schedule:

    # Runs every 4h

    - cron: '0 */4 * * *'

  workflow_dispatch:

  

jobs:

  update-readme-with-blog:

    name: Crawl rss and generate static page

    runs-on: windows-2019

    steps:

      - uses: actions/checkout@main

        with:

          repository: minhhungit/github-action-rss-crawler

          token: ${{ secrets.GITHUB_TOKEN }}

      - uses: actions/setup-dotnet@v1

        with:

          dotnet-version: 3.1.x

      #- run: dotnet build DemoApp\DemoApp.sln      

      - run: dotnet run --project RssCrawler\RssCrawler.csproj

      - run: git config --local user.email "[email protected]"

      - run: git config --local user.name "Jin"

      - run: git add .

      - run: git commit -m "Add changes"

      - run: git push

```

---

### Demo

> https://minhhungit.github.io/github-action-rss-crawler/



### Donate ^^

**If you like my works and would like to support then you can buy me a coffee ☕️ anytime**

 

**I would appreciate it!!!**