{"id":21097222,"url":"https://github.com/podverse/podverse-ops","last_synced_at":"2025-08-02T16:12:53.680Z","repository":{"id":42246404,"uuid":"137708207","full_name":"podverse/podverse-ops","owner":"podverse","description":"Deployment scripts and documentation for the Podverse ecosystem","archived":false,"fork":false,"pushed_at":"2025-07-23T02:14:59.000Z","size":34809,"stargazers_count":16,"open_issues_count":3,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-23T04:11:47.277Z","etag":null,"topics":["agplv3","ansible","docker","foss","hacktoberfest","open-source","podcast","podcasting","podcasting20","podcasts","rss","v4v","valueforvalue"],"latest_commit_sha":null,"homepage":"https://podverse.fm/about","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/podverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-18T04:00:33.000Z","updated_at":"2025-03-02T16:45:37.000Z","dependencies_parsed_at":"2025-07-14T23:28:22.875Z","dependency_job_id":null,"html_url":"https://github.com/podverse/podverse-ops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/podverse/podverse-ops","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podverse%2Fpodverse-ops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podverse%2Fpodverse-ops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podverse%2Fpodverse-ops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podverse%2Fpodverse-ops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/podverse","download_url":"https://codeload.github.com/podverse/podverse-ops/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podverse%2Fpodverse-ops/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268416633,"owners_count":24246945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agplv3","ansible","docker","foss","hacktoberfest","open-source","podcast","podcasting","podcasting20","podcasts","rss","v4v","valueforvalue"],"created_at":"2024-11-19T22:46:55.979Z","updated_at":"2025-08-02T16:12:53.647Z","avatar_url":"https://github.com/podverse.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# podverse-ops\n\nDeployment scripts for the podverse ecosystem\n\n## Getting started\n\n### Local / Stage / Prod\n\nTo test the Docker containers locally, use the docker-compose.local.yml file, and replace \\#\\#\\# with \"local\".\n\nFor stage deployment, use the docker-compose.stage.yml file, and replace \\#\\#\\# with \"stage\".\n\nFor prod deployment, use the docker-compose.prod.yml file, and replace \\#\\#\\# with \"prod\".\n\n### Setup environment variables\n\nDuplicate the config/podverse-api-\\#\\#\\#.example.env file, rename it to config/podverse-api-\\#\\#\\#.env, and update all of the environment variables to match what is needed for your environment. Repeat these steps for podverse-db-\\#\\#\\#.env and podverse-web-\\#\\#\\#.env.\n\nAdd your Google API key file to config/google/jwt.keys.json. Look in the config/google/jwt.keys.json.example for more info.\n\n### SSL Setup (Stage/Prod)\n\nReplace the LETSENCRYPT_HOST and LETSENCRYPT_EMAIL environment variables in the docker-compose.###.yml file.\n\n### Create and start all containers\n\nWARNING: If you use the letsencrypt for SSL on stage or prod, be careful\nnot to run the letsencrypt container too many times. Every time the container\nstarts it requests SSL certificates from letsencrypt, and if you reach that limit,\nyou won't be able to request new SSL certificates for a week. For that reason you may\nwant to remove podverse_letsencrypt_nginx from the following command while you're testing.\n\nFirst you'll need to create the nginx-proxy network:\n\n```\ndocker network create nginx-proxy\n```\n\nThen run the application containers:\n\n```\ndocker-compose -f docker-compose.###.yml up -d podverse_nginx_proxy podverse_letsencrypt_nginx podverse_db podverse_api podverse_web\n```\n\n### Add podcast categories to the database\n\n```\ndocker exec -d podverse_api_### npm --prefix /tmp run seeds:categories\n```\n\n### Add feed urls to the database\n\nTo add podcasts to the database, you first need to add feed urls to the\ndatabase, and then run the podcast parser with those feed urls.\n\nYou can pass multiple feed urls as a comma-delimited string parameter to the\n`npm run scripts:addFeedUrls` command.\n\nA list of sample podcast feed urls can be found in\n[podverse-api/docs/sampleFeedUrls.txt](https://github.com/podverse/podverse-api/blob/develop/docs/sampleFeedUrls.txt).\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:addFeedUrls \u003cfeed urls\u003e\n```\n\n### Parse feed urls to add podcasts and episodes to the database\n\nOrphan feed urls do not have a podcast associated with them.\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:parseOrphanFeedUrls\n```\n\nTo parse all non-orphan and public feed urls, you can run:\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:parsePublicFeedUrls\n```\n\n### Use SQS to add feed urls to a queue, then parse them\n\nThis project uses AWS SQS for its remote queue.\n\nTo add all orphan feeds to the queue:\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:addAllOrphanFeedUrlsToPriorityQueue\n```\n\nTo add all non-orphan, public feeds to the queue:\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:addAllPublicFeedUrlsToQueue\n```\n\nTo add all non-authority feedUrls (podcasts that do not have an authorityId):\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:addNonPodcastIndexFeedUrlsToPriorityQueue\n```\n\nTo add all recently updated (according to Podcast Index), public feeds to the priority queue:\n\n```\ndocker exec podverse_api_### npm --prefix /tmp run scripts:addRecentlyUpdatedFeedUrlsToPriorityQueue\n```\n\nAfter you have added feed urls to a queue, you can retrieve and then parse\nthe feed urls by running:\n\n```\ndocker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue -- \u003cretryTimeMS\u003e\n```\n\nWe also have a self-managed parsing queue, where we manually mark podcasts to be added to a separate queue for parsing at a regular cadence. The property is `Podcast.parsingPriority` and the `parsingPriority` is a value between 0-5. 0 is the default, and means the podcast should not be added to the self-managed queue. 1 is the most frequent, and 5 is the least frequent parsing.\n\nAt the time of writing this, 3 is the value we are using the most, which adds the feeds to the queue every 30 minutes.\n\nThe `offset` value is optional, and probably not needed.\n\n```bash\ndocker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:addFeedsToQueueByPriority \u003cparsingPriority\u003e \u003coffset\u003e\n```\n\nThen to parse from the self-managed queue call:\n\n```bash\ndocker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue\n```\n\n### Schedule parsing with cron\n\nBelow is a sample cron command for adding feeds to queues.\n\n```\n0 */6 * * * docker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run --rm podverse_api_parser_worker npm run scripts:addAllPublicFeedUrlsToQueue\n```\n\nFeed parsing happens in worker containers that run continuously, and after the worker\nreceives a \"No messages found\" response from the queue, the worker waits for a\ntimeout period before making another message request to the queue.\n\n```\ndocker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run -d --name podverse_api_parser_worker podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue -- 1800000\n```\n\n### Request Google Analytics pageview data and save to database with cron\n\nBelow is a sample cron config for requesting unique pageview data from Google\nAnalytics, which is used throughout the site for sorting by popularity (not an ideal system for popularity sorting, but it's what we're using for now).\n\n```\n0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips hour\n5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips day\n10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips week\n15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips month\n25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips year\n40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips allTime\n\n0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes hour\n5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes day\n10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes week\n15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes month\n25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes year\n40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes allTime\n\n0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts hour\n5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts day\n10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts week\n15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts month\n25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts year\n40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts allTime\n```\n### Automated Database Backup\n\nYou can run the `scripts/db_backup.sh` file with cron to automatically backup the database. Follow steps 1 and 2 in [this tutorial by Rahul Kumar](https://tecadmin.net/install-postgresql-server-on-ubuntu/) to install pg_dump, and follow [this tutorial by Pranav Prakash](https://pranavprakash.net/2017/05/16/automated-postgresql-backups/) to configure the script.\n\nSample cron command:\n\n```\n0 0 * * * /home/mitch/podverse-ops/scripts/db_backup.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpodverse%2Fpodverse-ops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpodverse%2Fpodverse-ops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpodverse%2Fpodverse-ops/lists"}