Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/foxmask/yeoboseyo

The bus for your internet services
https://github.com/foxmask/yeoboseyo

automation mastodon mattermost microservice rss self-hosted starlette

Last synced: about 2 months ago
JSON representation

The bus for your internet services

Host: GitHub
URL: https://github.com/foxmask/yeoboseyo
Owner: foxmask
License: wtfpl
Created: 2024-02-12T17:55:02.000Z (7 months ago)
Default Branch: master
Last Pushed: 2024-02-12T18:11:20.000Z (7 months ago)
Last Synced: 2024-04-16T01:48:48.458Z (5 months ago)
Topics: automation, mastodon, mattermost, microservice, rss, self-hosted, starlette
Language: Python
Homepage:
Size: 2.06 MB
Stars: 0
Watchers: 2
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

awesome-joplin - Yeoboseyo - the-bus-for-your-internet-services/2771)]. "The bus for your internet services." A companion for Joplin which allows creating notes from RSS Feeds, plus others possibilities, like publishing the same RSS Feeds on another service, like Mastodon. (Tools / Unmaintained Themes)

README

# 여보세요

![Yeoboseyo home page](https://github.com/foxmask/yeoboseyo/master/doc/Yeoboseyo_list.png)

## Description

From your favorite RSS feeds, spread/share those news to services of your choice

## Services covered

* RSS
* Mastodon
* Mattermost
* Slack
* Discord
* Telegram
* Wallabag
* Local Markdown files
* Any services using a Webhook to push data

### used cases:

* With Yeoboseyo you set an RSS URL and the "local storage" (the folder where to store news in markdown) then you synchronize those files with [syncthing](https://syncthing.net/) on any of your devices and read them with [Espilon Notes](http://epsilonexpert.com/), [Notable](https://notable.app/), [markor](https://gsantner.net/project/markor.html). Any markdown editor/reader can open/import those files.
This use case can be extended to any application that is able to import markdown file, use file manager or has import function, like [TagSpaces](https://www.tagspaces.org/)

* You want to publish news from your favorite news websites to `Mastodon`, Yeoboseyo will do that without any problem

* You need to spread news from your project on `Mattermost`/`Slack`/`Discord`, just set the URL of the `Webhook` of one of those services and the news will be published as expected on the `channel` of your choice
* If you prefer to publish that on `Telegram`, set the `TELEGRAM_CHAT_ID` of the group or channel

Let see how to setup all of that below

## :package: Installation

### pre requisistes

- python 3.8+
- [starlette](https://www.starlette.io/) (the web application)
- [feedparser](https://feedparser.readthedocs.io/en/latest/) (for RSS support)
- [pypandoc](https://pypi.org/project/pypandoc/) (to convert html to markdown)
- [wallabag API](https://guthub.com/foxmask/wallabagapi) for [Wallabag](https://wallabag.org/en) readit later applications

### Installation

create a virtualenv

```bash
python3 -m venv yeoboseyo
cd yeoboseyo
source bin/activate
```

then

```bash
pip install -r requirements.txt
```

```bash
pip install yeoboseyo
```

## :wrench: Settings

copy or rename the config file

```bash
mv env.sample .env
```

set the correct values for your own environment

```ini
DATABASE_URL=sqlite:///db.sqlite3
TIME_ZONE=Europe/Paris
FORMAT_FROM=markdown_github
FORMAT_TO=html
BYPASS_BOZO=False # if you don't want to get the malformed RSS Feeds set it to False
LOG_LEVEL=logging.INFO
MASTODON_USERNAME=your username@
MASTODON_PASSWORD=your pass
MASTODON_INSTANCE=https://
MASTODON_VISIBILITY=unlisted # default is 'public', can be 'unlisted', 'private', 'direct'
TOKEN=''
TELEGRAM_TOKEN=0123456789:AZERTYUIOPQSDFGHJKLMWXCVBN123456789
TELEGRAM_CHAT_ID=-NNNNNNNNN
WALLABAG_URL=http://wallabag/
WALLABAG_CLIENTID=your id
WALLABAG_CLIENTSECRET=your secret
WALLABAG_PASSWORD=wallabag
WALLABAG_USERNAME=wallabag
```

### Mastodon

to create the app on mastodon :

on https://yourmastoinstance/settings/applications/new

* Application name : `Yeoboseyo`
* Scopes : check : `read` / `write` / `push` / `follow`
* then `submit`

then select Yeoboseyo again to retreive the access token, and fill the file `yeoboseyo_clientcred.secret` put on the first line the value of "Your access token" and on the second line the https url of your masto instance eg

```bash
Azdfghy5678hefdsgghjuju09knb
https://framapiaf.org
```

this file will be read each time something will be posted on masto

### Slack/Mattermost/Discord: Webhook

in the 'integrations' page set an "incoming webhooks" (eg from https://mattermost/teamname/integrations) and copy the URL into the field 'webhook' of the Yeoboseyo form

### Telegram

you will need to create a `Telegram Bot` by invoking [@BotFather](https://core.telegram.org/bots).
Once the bot is created, create a group / channel, and invite your bot on that group/channel and give it admin rights.

Now you need to get the ID of the group / channel where the bot can "speak"

for that : access to your bot history url

https://api.telegram.org/botXXXXX:YYYYY/getUpdates

and spot this kind of data

```json
chat:
id: -NNNNNN
title: "my group name"
```

pick up the `-NNNNNN` and put it as the `TELEGRAM_CHAT_ID` in the `.env` config file

### Wallabag

Create a client API like explain here https://doc.wallabag.org/fr/developer/api/oauth.html

this will give you something like this

![Wallabag](https://github.com/foxmask/wallabagapi/blob/master/wallabag_api_key.png)

Then replace the client_id / client_secret / login / pass in the .env file

```ini
WALLABAG_URL=http://wallabag/
WALLABAG_CLIENTID=your id
WALLABAG_CLIENTSECRET=your secret
WALLABAG_PASSWORD=wallabag
WALLABAG_USERNAME=wallabag
```

## :dvd: Database

create the database (to execute only once)

```bash
python models.py
```

## :mega: Running the Web application

start the application

```bash
cd yeoboseyo
python app.py &
여보세요 !
INFO: Started server process [13588]
INFO: Waiting for application startup.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```

### :eyes: Adding some Feeds to track

Go on `http://0.0.0.0:8000` and fill the form to add new Feeds to track

* If you plan to publish RSS Feeds
* into a `local folder`, fill the `local storage` field with the complet path of that folder, if not leave it empty.
* on your `Mastodon` account, tick the checkbox `Publish on Mastodon?`, if not, leave it unchecked
* on your `Telegram` group/channel tick the checkbox `Publish on Telegram?`, if not, leave it unchecked
* on your `Mattermost`/`Slack`/`Discord` group/channel, just set the URL of the `Webhook` of one of those services, if not, leave it empty.

### :dizzy: Running the engine

now that you fill settings, and form, launch the command and see how many feeds are comming

```bash
여보세요 !
usage: python run.py [-h] -a {report,go,switch} [-trigger_id TRIGGER_ID]

Yeoboseyo

optional arguments:
-h, --help show this help message and exit
-a {report,go,switch}
choose -a report or -a go or -a swtch -trigger_id
-trigger_id TRIGGER_ID
trigger id to switch of status

python run.py -a go

여보세요 ! RUN and GO
Trigger FoxMasK blog
Entries created 1 / Read 1

```

### get the list

get the list of your feeds to check which one provided articles or not

```bash
$ python run.py -a report
여보세요 !
Report
┏━━━━┳━━
┃ ID ┃ Name
┡━━━━╇━━
│ 1 │ Mon Blog
│ 2 │ KBS Culture
│ 3 │ KBS
│ 4 │ KBS Show biz
│ 5 │ Jux Video
│ 6 │ PlayStation Blog
│ 7 │ GameKult
│ 8 │ Gameblog
│ 9 │ NoFrag
│ 10 │ Frandroid
│ 11 │ Les Numeriques
│ 12 │ VueJS News
│ 13 │ Cacktus Blog
│ 14 │ Python News
│ 15 │ nedbatchelder
│ 16 │ Django News
│ 17 │ Python Insider
│ 18 │ PyCharm Blog
│ 19 │ Real Python
│ 20 │ VueJS
│ 21 │ Odieux Connard
└────┴── ━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Md Folder ┃ Tags ┃ Status ┃ Triggered ┃ ━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ test │ News │ Ok │ 2021-03-18 22:35:21 │ │ test │ News │ Ok │ 2021-04-05 09:59:03 │ journal du jour │ test │ News │ Ok │ 2021-04-05 09:59:05 │ │ test │ News │ Ok │ 2021-04-05 09:59:06 │ │ test │ jeux │ Ok │ 2021-04-01 22:22:15.113871 │ │ test │ jeux │ Ok │ 2021-04-01 22:22:57.189312 │ │ test │ jeux │ Ok │ 2021-04-01 22:23:21.049307 │ │ test │ jeux │ Ok │ 2021-04-01 22:23:48.350934 │ │ test │ jeux │ Ok │ 2021-04-01 22:24:15.721174 │ │ test │ android │ Ok │ 2021-04-01 22:24:47.324475 │ │ test │ android │ Ok │ 2021-04-01 22:25:09.740677 │ │ test │ vuejs │ Ok │ 2021-04-01 22:25:34.307735 │ │ test │ python │ Ok │ 2021-04-01 22:26:02.412688 │ │ test │ python │ Ok │ 2021-04-01 22:26:41.975564 │ │ test │ python │ Ok │ 2021-04-01 22:28:21.838166 │ │ test │ Python │ Ok │ 2021-04-01 22:28:47.804644 │ │ test │ Python │ Ok │ 2021-04-01 22:29:18.791661 │ │ test │ Python │ Ok │ 2021-04-01 22:29:44.568828 │ │ test │ Python │ Ok │ 2021-04-01 22:30:10.952486 │ │ test │ VueJS │ Ok │ 2021-04-01 22:30:34.507337 │ │ test │ Humour │ Ok │ 2021-04-01 22:31:03.458147 │ ───────────────────┴───────────┴─────────┴────────┴────────────────────────────┘

```

### switch the status of a trigger

switch the status of trigger to on/off

```bash
python run.py -a switch -trigger_id 1

여보세요 ! Switch
Successfully disabled Trigger 'Mon Blog'
```

and check it again to see the status moving

```bash
09:00 $ python run.py -a report
여보세요 !
Report
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ID ┃ Name ┃ Md Folder ┃ Tags ┃ Status ┃ Triggered ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1 │ Mon Blog │ test │ News │ Disabled │ 2021-05-15 09:00:27 │
│ 2 │ KBS Culture │ test │ News │ Ok │ 2021-04-05 09:59:03 │
│ 3 │ KBS journal du jour │ test │ News │ Ok │ 2021-04-05 09:59:05 │
│ 4 │ KBS Show biz │ test │ News │ Ok │ 2021-04-05 09:59:06 │
│ 5 │ Jux Video │ test │ jeux │ Ok │ 2021-04-01 22:22:15.113871 │
│ 6 │ PlayStation Blog │ test │ jeux │ Ok │ 2021-04-01 22:22:57.189312 │
│ 7 │ GameKult │ test │ jeux │ Ok │ 2021-04-01 22:23:21.049307 │
│ 8 │ Gameblog │ test │ jeux │ Ok │ 2021-04-01 22:23:48.350934 │
│ 9 │ NoFrag │ test │ jeux │ Ok │ 2021-04-01 22:24:15.721174 │
│ 10 │ Frandroid │ test │ android │ Ok │ 2021-04-01 22:24:47.324475 │
│ 11 │ Les Numeriques │ test │ android │ Ok │ 2021-04-01 22:25:09.740677 │
│ 12 │ VueJS News │ test │ vuejs │ Ok │ 2021-04-01 22:25:34.307735 │
│ 13 │ Cacktus Blog │ test │ python │ Ok │ 2021-04-01 22:26:02.412688 │
│ 14 │ Python News │ test │ python │ Ok │ 2021-04-01 22:26:41.975564 │
│ 15 │ nedbatchelder │ test │ python │ Ok │ 2021-04-01 22:28:21.838166 │
│ 16 │ Django News │ test │ Python │ Ok │ 2021-04-01 22:28:47.804644 │
│ 17 │ Python Insider │ test │ Python │ Ok │ 2021-04-01 22:29:18.791661 │
│ 18 │ PyCharm Blog │ test │ Python │ Ok │ 2021-04-01 22:29:44.568828 │
│ 19 │ Real Python │ test │ Python │ Ok │ 2021-04-01 22:30:10.952486 │
│ 20 │ VueJS │ test │ VueJS │ Ok │ 2021-04-01 22:30:34.507337 │
│ 21 │ Odieux Connard │ test │ Humour │ Ok │ 2021-04-01 22:31:03.458147 │
└────┴─────────────────────┴───────────┴─────────┴──────────┴────────────────────────────┘

```

## Migrations

if you had the version 0.5.0:

run

```sql
migrations/alter_table_trigger_add_telegram.sql
migrations/alter_table_trigger_add_wallabag.sql
```

if you had the version 0.4.0:

run

```sql
migrations/alter_table_trigger_add_webhook.sql
```

(Image credits to [Emojipedia](https://emojipedia.org/))

## Test

```bash
export TEST_DATABASE_URL=sqlite:///db_test.sqlite3
pytest
[...]
yeoboseyo/tests/test_models.py .. [ 66%]
yeoboseyo/tests/test_settings.py . [100%]

===================================================================== 3 passed in 0.52s ======================================================================
```

```bash
coverage run --source='.' -m pytest
coverage report -m
```