{"id":13596531,"url":"https://github.com/j0k3r/banditore","last_synced_at":"2025-04-10T05:41:27.753Z","repository":{"id":37432560,"uuid":"82027290","full_name":"j0k3r/banditore","owner":"j0k3r","description":"Banditore retrieves new releases from your starred GitHub repositories and generate an Atom feed with them.","archived":false,"fork":false,"pushed_at":"2024-05-28T03:24:51.000Z","size":4439,"stargazers_count":128,"open_issues_count":7,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-05-28T13:00:58.802Z","etag":null,"topics":["github","hacktoberfest","rabbitmq","release","rss","starred-repositories","symfony","symfony-application"],"latest_commit_sha":null,"homepage":"https://bandito.re","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j0k3r.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"j0k3r"}},"created_at":"2017-02-15T06:20:40.000Z","updated_at":"2024-05-30T06:26:31.924Z","dependencies_parsed_at":"2023-09-25T06:19:14.987Z","dependency_job_id":"88a4061f-845e-4f99-bc4b-8e837d17d9cc","html_url":"https://github.com/j0k3r/banditore","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j0k3r%2Fbanditore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j0k3r%2Fbanditore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j0k3r%2Fbanditore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j0k3r%2Fbanditore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j0k3r","download_url":"https://codeload.github.com/j0k3r/banditore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248166222,"owners_count":21058475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github","hacktoberfest","rabbitmq","release","rss","starred-repositories","symfony","symfony-application"],"created_at":"2024-08-01T16:02:32.909Z","updated_at":"2025-04-10T05:41:27.741Z","avatar_url":"https://github.com/j0k3r.png","language":"PHP","funding_links":["https://github.com/sponsors/j0k3r"],"categories":["PHP"],"sub_categories":[],"readme":"\u003cimg src=\"https://i.imgur.com/kAvg4w9.png\" align=\"right\" /\u003e\n\n# Banditore\n\n![CI](https://github.com/j0k3r/banditore/workflows/CI/badge.svg)\n[![Coveralls Status](https://coveralls.io/repos/github/j0k3r/banditore/badge.svg?branch=master)](https://coveralls.io/github/j0k3r/banditore?branch=master)\n![PHPStan level max](https://img.shields.io/badge/PHPStan-level%20max-brightgreen.svg?style=flat)\n\nBanditore retrieves new releases from your GitHub starred repositories and put them in a RSS feed, just for you.\n\n![](https://i.imgur.com/XDCWLJV.png)\n\n## Requirements\n\n - PHP \u003e= 8.2 (with `pdo_mysql`)\n - MySQL \u003e= 5.7\n - Redis (to cache requests to the GitHub API)\n - [RabbitMQ](https://www.rabbitmq.com/), which is optional (see below)\n - [Supervisor](http://supervisord.org/) (only if you use RabbitMQ)\n - [NVM](https://github.com/nvm-sh/nvm#install--update-script) \u0026 [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/) to install assets\n\n## Installation\n\n1. Clone the project\n\n    ```bash\n    git clone https://github.com/j0k3r/banditore.git\n    ```\n\n2. [Register a new OAuth GitHub application](https://github.com/settings/applications/new) and get the _Client ID_ \u0026 _Client Secret_ for the next step (for the _Authorization callback URL_ put `http://127.0.0.1:8000/callback`)\n\n3. Install dependencies using [Composer](https://getcomposer.org/download/) and define your parameter during the installation\n\n    ```bash\n    APP_ENV=prod composer install -o --no-dev\n    ```\n\n    If you want to use:\n     - **Sentry** to retrieve all errors, [register here](https://sentry.io/signup/) and get your dsn (in Project Settings \u003e DSN).\n\n4. Setup the database\n\n    ```bash\n    php bin/console doctrine:database:create -e prod\n    php bin/console doctrine:schema:create -e prod\n    ```\n\n5. Install assets\n\n    ```bash\n    nvm install\n    yarn install\n    ```\n\n6. You can now launch the website:\n\n    ```bash\n    php -S localhost:8000 -t public/\n    ```\n\n    And access it at this address: `http://127.0.0.1:8000`\n\n## Running the instance\n\nOnce the website is up, you now have to setup few things to retrieve new releases.\nYou have two choices:\n- using crontab command (very simple and ok if you are alone)\n- using RabbitMQ (might be better if you plan to have more than few persons but it's more complex) :call_me_hand:\n\n### Without RabbitMQ\n\nYou just need to define these 2 cronjobs (replace all `/path/to/banditore` with real value):\n\n```bash\n# retrieve new release of each repo every 10 minutes\n*/10  *   *   *   *   php /path/to/banditore/bin/console -e prod banditore:sync:versions \u003e\u003e /path/to/banditore/var/logs/command-sync-versions.log 2\u003e\u00261\n# sync starred repos of each user every 5 minutes\n*/5   *   *   *   *   php /path/to/banditore/bin/console -e prod banditore:sync:starred-repos \u003e\u003e /path/banditore/to/var/logs/command-sync-repos.log 2\u003e\u00261\n```\n\n### With RabbitMQ\n\n1. You'll need to declare exchanges and queues. Replace `guest` by the user of your RabbitMQ instance (`guest` is the default one):\n\n ```bash\n php bin/console messenger:setup-transports -vvv sync_starred_repos\n php bin/console messenger:setup-transports -vvv sync_versions\n ```\n\n2. You now have two queues and two exchanges defined:\n - `banditore.sync_starred_repos`: will receive messages to sync starred repos of all users\n - `banditore.sync_versions`: will receive message to retrieve new release for repos\n\n3. Enable these 2 cronjobs which will periodically push messages in queues (replace all `/path/to/banditore` with real value):\n\n ```bash\n # retrieve new release of each repo every 10 minutes\n */10  *   *   *   *   php /path/to/banditore/bin/console -e prod banditore:sync:versions --use_queue \u003e\u003e /path/to/banditore/var/logs/command-sync-versions.log 2\u003e\u00261\n # sync starred repos of each user every 5 minutes\n */5   *   *   *   *   php /path/to/banditore/bin/console -e prod banditore:sync:starred-repos --use_queue \u003e\u003e /path/banditore/to/var/logs/command-sync-repos.log 2\u003e\u00261\n```\n\n4. Setup Supervisor using the [sample file](data/supervisor.conf) from the repo. You can copy/paste it into `/etc/supervisor/conf.d/` and adjust path. The default file will launch:\n  - 2 workers for sync starred repos\n  - 4 workers to fetch new releases\n\n Once you've put the file in the supervisor conf repo, run `supervisorctl update \u0026\u0026 supervisorctl start all` (`update` will read your conf, `start all` will start all workers)\n\n### Monitoring\n\nThere is a status page available at `/status`, it returns a json with some information about the freshness of fetched versions:\n\n```json\n{\n    \"latest\": {\n        \"date\": \"2019-09-17 19:50:50.000000\",\n        \"timezone_type\": 3,\n        \"timezone\": \"Europe\\/Berlin\"\n    },\n    \"diff\": 1736,\n    \"is_fresh\": true\n}\n```\n\n- `latest`: the latest created version as a DateTime\n- `diff`: the difference between now and the latest created version (in seconds)\n- `is_fresh`: indicate if everything is fine by comparing the `diff` above with the `status_minute_interval_before_alert` parameter\n\nFor example, I've setup a check on [updown.io](https://updown.io/r/P7qer) to check that status page and if the page contains `\"is_fresh\":true`. So I receive an alert when `is_fresh` is false: which means there is a potential issue on the server.\n\n## Running the test suite\n\nIf you plan to contribute (you're awesome, I know that :v:), you'll need to install the project in a different way (for example, to retrieve dev packages):\n\n```bash\ngit clone https://github.com/j0k3r/banditore.git\ncomposer install -o\nphp bin/console doctrine:database:create -e=test\nphp bin/console doctrine:schema:create -e=test\nphp bin/console doctrine:fixtures:load --env=test -n\nphp bin/simple-phpunit -v\n```\n\nBy default the `test` connexion login is `root` without password. You can change it in [app/config/config_test.yml](app/config/config_test.yml).\n\n## How it works\n\nOk, if you goes that deeper in the readme, it means you're a bit more than interested, I like that.\n\n### Retrieving new release / tag\n\nThis is the complex part of the app. Here is a simplified solution to achieve it.\n\n#### New release\n\nIt's not as easy as using the `/repos/:owner/:repo/releases` API endpoint to retrieve latest release for a given repo. Because not all repo owner use that feature (which is a shame in my case).\n\nAll information for a release are available on that endpoint:\n- name of the tag (ie: v1.0.0)\n- name of the release (ie: yay first release)\n- published date\n- description of the release\n\n\u003e Check a new release of that repo as example: https://api.github.com/repos/j0k3r/banditore/releases/5770680\n\n#### New tag\n\nSome owners also use tag which is a bit more complex to retrieve all information because a tag only contains information about the SHA-1 of the commit which was used to make the tag.\nWe only have these information:\n- name of the tag (ie: v1.4.2)\n- name of the release will be the name of the tag, in that case\n\n\u003e Check tag list of swarrot/SwarrotBundle as example: https://api.github.com/repos/swarrot/SwarrotBundle/tags\n\nAfter retrieving the tag, we need to retrieve the commit to get these information:\n- date of the commit\n- message of the commit\n\n\u003e Check a commit from the previous tag list as example: https://api.github.com/repos/swarrot/SwarrotBundle/commits/84c7c57622e4666ae5706f33cd71842639b78755\n\n### GitHub Client Discovery\n\nThis is the most important piece of the app. One thing that I ran though is hitting the rate limit on GitHub.\nThe rate limit for a given authenticated client is 5.000 calls per hour. This limit is **never** reached when looking for new release (thanks to the [conditional requests](https://developer.github.com/v3/#conditional-requests) of the GitHub API) on a daily basis.\n\nBut when new user sign in, we need to sync all its starred repositories and also all their releases / tags. And here come the gourmand part:\n- one call for the list of release\n- one call to retrieve information of each tag (if the repo doesn't have release)\n- one call for each release to convert markdown text to html\n\nLet's say the repo:\n- has 50 tags: 1 (get tag list) + 50 (get commit information) + 50 (convert markdown) = 101 calls.\n- has 50 releases: 1 (get tag list) + 50 (get each release) + 50 (convert markdown) = 101 calls.\n\nAnd keep in mind that some repos got also 1.000+ tags (!!).\n\nTo avoid hitting the limit in such case and wait 1 hour to be able to make requests again I created the [GitHub Client Discovery class](src/AppBundle/Github/ClientDiscovery.php).\nIt aims to find the best client with enough rate limit remain (defined as 50).\n- it first checks using the GitHub OAuth app\n- then it checks using all user GitHub token\n\nWhich means, if you have 5 users on the app, you'll be able to make (1 + 5) x 5.000 = 30.000 calls per hour\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj0k3r%2Fbanditore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj0k3r%2Fbanditore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj0k3r%2Fbanditore/lists"}