Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/podverse/podverse-ops
Deployment scripts and documentation for the Podverse ecosystem
https://github.com/podverse/podverse-ops
agplv3 ansible docker foss hacktoberfest open-source podcast podcasting podcasting20 podcasts rss v4v valueforvalue
Last synced: about 1 month ago
JSON representation
Deployment scripts and documentation for the Podverse ecosystem
- Host: GitHub
- URL: https://github.com/podverse/podverse-ops
- Owner: podverse
- License: agpl-3.0
- Created: 2018-06-18T04:00:33.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-04-29T03:08:48.000Z (8 months ago)
- Last Synced: 2024-05-01T11:37:03.912Z (8 months ago)
- Topics: agplv3, ansible, docker, foss, hacktoberfest, open-source, podcast, podcasting, podcasting20, podcasts, rss, v4v, valueforvalue
- Language: PLpgSQL
- Homepage: https://podverse.fm/about
- Size: 32.9 MB
- Stars: 9
- Watchers: 5
- Forks: 8
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# podverse-ops
Deployment scripts for the podverse ecosystem
## Getting started
### Local / Stage / Prod
To test the Docker containers locally, use the docker-compose.local.yml file, and replace \#\#\# with "local".
For stage deployment, use the docker-compose.stage.yml file, and replace \#\#\# with "stage".
For prod deployment, use the docker-compose.prod.yml file, and replace \#\#\# with "prod".
### Setup environment variables
Duplicate the config/podverse-api-\#\#\#.example.env file, rename it to config/podverse-api-\#\#\#.env, and update all of the environment variables to match what is needed for your environment. Repeat these steps for podverse-db-\#\#\#.env and podverse-web-\#\#\#.env.
Add your Google API key file to config/google/jwt.keys.json. Look in the config/google/jwt.keys.json.example for more info.
### SSL Setup (Stage/Prod)
Replace the LETSENCRYPT_HOST and LETSENCRYPT_EMAIL environment variables in the docker-compose.###.yml file.
### Create and start all containers
WARNING: If you use the letsencrypt for SSL on stage or prod, be careful
not to run the letsencrypt container too many times. Every time the container
starts it requests SSL certificates from letsencrypt, and if you reach that limit,
you won't be able to request new SSL certificates for a week. For that reason you may
want to remove podverse_letsencrypt_nginx from the following command while you're testing.
First you'll need to create the nginx-proxy network:
```
docker network create nginx-proxy
```
Then run the application containers:
```
docker-compose -f docker-compose.###.yml up -d podverse_nginx_proxy podverse_letsencrypt_nginx podverse_db podverse_api podverse_web
```
### Add podcast categories to the database
```
docker exec -d podverse_api_### npm --prefix /tmp run seeds:categories
```
### Add feed urls to the database
To add podcasts to the database, you first need to add feed urls to the
database, and then run the podcast parser with those feed urls.
You can pass multiple feed urls as a comma-delimited string parameter to the
`npm run scripts:addFeedUrls` command.
A list of sample podcast feed urls can be found in
[podverse-api/docs/sampleFeedUrls.txt](https://github.com/podverse/podverse-api/blob/develop/docs/sampleFeedUrls.txt).
```
docker exec podverse_api_### npm --prefix /tmp run scripts:addFeedUrls
```
### Parse feed urls to add podcasts and episodes to the database
Orphan feed urls do not have a podcast associated with them.
```
docker exec podverse_api_### npm --prefix /tmp run scripts:parseOrphanFeedUrls
```
To parse all non-orphan and public feed urls, you can run:
```
docker exec podverse_api_### npm --prefix /tmp run scripts:parsePublicFeedUrls
```
### Use SQS to add feed urls to a queue, then parse them
This project uses AWS SQS for its remote queue.
To add all orphan feeds to the queue:
```
docker exec podverse_api_### npm --prefix /tmp run scripts:addAllOrphanFeedUrlsToPriorityQueue
```
To add all non-orphan, public feeds to the queue:
```
docker exec podverse_api_### npm --prefix /tmp run scripts:addAllPublicFeedUrlsToQueue
```
To add all non-authority feedUrls (podcasts that do not have an authorityId):
```
docker exec podverse_api_### npm --prefix /tmp run scripts:addNonPodcastIndexFeedUrlsToPriorityQueue
```
To add all recently updated (according to Podcast Index), public feeds to the priority queue:
```
docker exec podverse_api_### npm --prefix /tmp run scripts:addRecentlyUpdatedFeedUrlsToPriorityQueue
```
After you have added feed urls to a queue, you can retrieve and then parse
the feed urls by running:
```
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue --
```
We also have a self-managed parsing queue, where we manually mark podcasts to be added to a separate queue for parsing at a regular cadence. The property is `Podcast.parsingPriority` and the `parsingPriority` is a value between 0-5. 0 is the default, and means the podcast should not be added to the self-managed queue. 1 is the most frequent, and 5 is the least frequent parsing.
At the time of writing this, 3 is the value we are using the most, which adds the feeds to the queue every 30 minutes.
The `offset` value is optional, and probably not needed.
```bash
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:addFeedsToQueueByPriority
```
Then to parse from the self-managed queue call:
```bash
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue
```
### Schedule parsing with cron
Below is a sample cron command for adding feeds to queues.
```
0 */6 * * * docker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run --rm podverse_api_parser_worker npm run scripts:addAllPublicFeedUrlsToQueue
```
Feed parsing happens in worker containers that run continuously, and after the worker
receives a "No messages found" response from the queue, the worker waits for a
timeout period before making another message request to the queue.
```
docker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run -d --name podverse_api_parser_worker podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue -- 1800000
```
### Request Google Analytics pageview data and save to database with cron
Below is a sample cron config for requesting unique pageview data from Google
Analytics, which is used throughout the site for sorting by popularity (not an ideal system for popularity sorting, but it's what we're using for now).
```
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips allTime
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes allTime
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts allTime
```
### Automated Database Backup
You can run the `scripts/db_backup.sh` file with cron to automatically backup the database. Follow steps 1 and 2 in [this tutorial by Rahul Kumar](https://tecadmin.net/install-postgresql-server-on-ubuntu/) to install pg_dump, and follow [this tutorial by Pranav Prakash](https://pranavprakash.net/2017/05/16/automated-postgresql-backups/) to configure the script.
Sample cron command:
```
0 0 * * * /home/mitch/podverse-ops/scripts/db_backup.sh
```