{"id":27881815,"url":"https://github.com/src-d/rovers","last_synced_at":"2025-05-05T05:05:25.160Z","repository":{"id":36602096,"uuid":"40908251","full_name":"src-d/rovers","owner":"src-d","description":"Rovers is a service to retrieve repository URLs from multiple repository hosting providers.","archived":false,"fork":false,"pushed_at":"2019-07-02T15:50:32.000Z","size":22738,"stargazers_count":14,"open_issues_count":7,"forks_count":13,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-05-05T05:05:16.244Z","etag":null,"topics":["bitbucket","cgit","crawler","github"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/src-d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-17T15:59:59.000Z","updated_at":"2023-09-08T17:00:41.000Z","dependencies_parsed_at":"2022-08-30T13:20:15.229Z","dependency_job_id":null,"html_url":"https://github.com/src-d/rovers","commit_stats":null,"previous_names":["tyba/srcd-rovers"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Frovers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Frovers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Frovers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Frovers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/src-d","download_url":"https://codeload.github.com/src-d/rovers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252442485,"owners_count":21748451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitbucket","cgit","crawler","github"],"created_at":"2025-05-05T05:05:24.572Z","updated_at":"2025-05-05T05:05:25.147Z","avatar_url":"https://github.com/src-d.png","language":"HTML","readme":"# rovers\n[![Build Status](https://travis-ci.com/src-d/rovers.svg?branch=master)](https://travis-ci.com/src-d/rovers)\n[![codecov](https://codecov.io/gh/src-d/rovers/branch/master/graph/badge.svg)](https://codecov.io/gh/src-d/rovers)\n\n**rovers** is a service to retrieve repository URLs from multiple repository\nhosting providers and store the URLs and metadata to a PostgreSQL database while\nissuing messages to a RabbitMQ queue so you can connect other processes to it.\n\n# Quick start using docker images\n\n## Download docker images\n\nGet the latest rovers image:\n\n```bash\ndocker pull srcd/rovers\n```\n\n### Start everything using docker-compose\n\nInstall [docker-compose](https://docs.docker.com/compose/install/).\n\nStart Rabbit and Postgres\n\n```bash\ndocker-compose -d up rovers-postgres rovers-rabbitmq\n```\n\nExport as environment variables the API keys ([see section](#supported-providers)). Then, you can execute rovers:\n```bash\ndocker-compose up --no-deps rovers\n```\n\nIf you need to run just some of the available supported-providers, you can use this command replacing the flag `--provider` with those providers you want to use:\n```bash\ndocker-compose run --rm --no-deps --service-ports rovers /bin/sh -c \"rovers initdb; rovers repos --provider=github --provider=bitbucket\"\n```\n\nAfter that, rovers will generate a lot of 'mentions' (git repositories found on the internet), and sending them to the 'rovers' queue in Rabbit.\n\nTo stop and remove all the containers running `docker-compose down`.\n\nFinally, you can use [Borges](https://github.com/src-d/borges) to fetch the content of these repositories.\n\n# Supported Providers\n\nAll the supported providers are used by default. In case you need to run only some of them you must use the `--provider` flag:\n```bash\nrovers repos --provider=github --provider=bitbucket\n```\n\n## Generate needed API keys for the providers\n\nTo be able to fetch github and cgit repositories, you should create several API keys:\n\n- Get Github token: https://github.com/settings/tokens\n- Get Bing token (Bing is the search engine used to fetch cgit repositories from internet): https://azure.microsoft.com/en-us/pricing/details/cognitive-services/search-api/web/\n\n## GitHub\n\nUses the GitHub API to get new repositories. Requires a GitHub API token, which will only need repository read access. You can set the token through the environment variable:\n\n```bash\n$ export CONFIG_GITHUB_TOKEN=github-token\n```\n\n## Bitbucket\n\nUses the Bitbucket API to get new repositories as an anonymous user.\n\n**Note:** the first time this provider runs it may take a while to find git repositories, as the vast majority of repositories at the beginning of the API results are mercurial repositories.\n\n### cgit\n\ncgit is a popular service to create your own self-hosted Git repository provider.\nRovers uses Bing search to discover cgit instances online and tracks them to get\nnew repositories. Requires a Bing API key. You can set the key through the environment variable:\n\n```bash\n$ export CONFIG_BING_KEY=bing-api-key\n```\n\n# Installation\n\n```\ngo get -u github.com/src-d/rovers/...\n```\n\n# Usage\n\nRun `rovers --help` to get help about the supported commands and their options.\nThe commands `initdb`, `replay` and `repos` also have their own `--help` switch\nfor specific options related to them.\n\n\nTo initialize the database schemas. You need to run this command only once.\n```\nrovers initdb\n```\nTo start collecting repository URLs\n```\nrovers repos --provider=github\n```\n\nYou can configure rovers using environment variables:\n\nProviders:\n- `CONFIG_GITHUB_TOKEN` to set the github api key.\n- `CONFIG_BING_KEY` to set the cgit api key.\n\nBroker:\n- `CONFIG_BROKER_URL` to set the broker url, by default `amqp://guest:guest@localhost:5672`\n\nDatabase:\n- `CONFIG_DBUSER`: database username, by default if not set `testing`\n- `CONFIG_DBPASS`: database user password, by default if not set `testing`\n- `CONFIG_DBHOST`: database host, by default if not set `0.0.0.0`\n- `CONFIG_DBPORT`: database port, by default if not set `5432`\n- `CONFIG_DBNAME`: database name, by default if not set `testing`\n- `CONFIG_DBSSLMODE`: ssl mode to use, by default if not set `disable`\n- `CONFIG_DBTIMEOUT`: connection timeout, by default if not set `30s`\n- `CONFIG_DBAPPNAME`: application name\n\n# Development\n\n## Build\n\n- `rm Makefile.main; rm -rf .ci` to make sure you will have the last Makefile changes.\n- `make dependencies` to download vendor dependencies.\n- `make packages` to generate binaries for several platforms.\n\nYou will find the built binaries under `./build`.\n\n## Test\n\nThis service uses PostgreSQL and RabbitMQ.\n\nTo execute tests locally you need to run RabbitMQ and PostgreSQL.\n\n```bash\n  docker-compose up -d rovers-postgres rovers-rabbitmq\n  make test\n```\n\n# Running Rovers in Kubernetes\n\nYou can use the official [Helm](https://github.com/kubernetes/helm) [chart](https://github.com/src-d/charts/tree/master/stable/rovers) to deploy Rovers in your Kubernetes cluster.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Frovers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrc-d%2Frovers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Frovers/lists"}