{"id":19900070,"url":"https://github.com/m-lab/downloader","last_synced_at":"2025-05-02T22:32:10.105Z","repository":{"id":22642880,"uuid":"96916822","full_name":"m-lab/downloader","owner":"m-lab","description":"Service to download meta data sources for archive.","archived":false,"fork":false,"pushed_at":"2025-02-27T16:44:28.000Z","size":196,"stargazers_count":2,"open_issues_count":4,"forks_count":3,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-29T15:21:23.167Z","etag":null,"topics":["etl","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-07-11T16:59:23.000Z","updated_at":"2025-02-27T16:44:30.000Z","dependencies_parsed_at":"2024-04-15T06:55:53.907Z","dependency_job_id":"0993c2d1-67a8-4f3e-baa9-2fc075d0d76f","html_url":"https://github.com/m-lab/downloader","commit_stats":{"total_commits":145,"total_committers":12,"mean_commits":"12.083333333333334","dds":"0.37241379310344824","last_synced_commit":"39a1ce4e14258e77ac0e6e5d7318af03b9fd16c2"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fdownloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fdownloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fdownloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fdownloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/downloader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252116476,"owners_count":21697388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","pipeline"],"created_at":"2024-11-12T20:11:03.688Z","updated_at":"2025-05-02T22:32:09.772Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[downloader](https://github.com/m-lab/downloader) [![Version](https://img.shields.io/github/tag/m-lab/downloader.svg)](https://github.com/m-lab/downloader/releases) [![Build Status](https://travis-ci.org/m-lab/downloader.svg?branch=master)](https://travis-ci.org/m-lab/downloader) [![Coverage Status](https://coveralls.io/repos/m-lab/downloader/badge.svg?branch=master)](https://coveralls.io/github/m-lab/downloader?branch=master) [![GoDoc](https://godoc.org/github.com/m-lab/downloader?status.svg)](https://godoc.org/github.com/m-lab/downloader) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/downloader)](https://goreportcard.com/report/github.com/m-lab/downloader)\n\n# downloader\nThe Downloader tool runs on Google Container Engine and will scrape the Maxmind\nand Routeviews for new data. It will download them from the websites and place\nthem in Google Cloud Storage in the specefied bucket.\n\nIt takes one mandatory arguement: `--bucket=GCS-BUCKET-NAME`\n\n## Travis Deployment\nDownloader is designed to be deployed exclusively from Travis-CI. If you need to\nconfigure Travis to automatically deploy to GKE, then there are a couple things\nyou need to be sure to configure.\n\n### Service Account\nThe deployment scripts expect the service accounts needed for uploading the\ndocker image to GCS and deploying the image to GKE to be stored in the protected\nenvironment variable `GCLOUD_SERVICE_KEY_XXX`, where XXX is the environment\nyou're deploying to (BOX for sandbox, STG for staging, or PRD for production).\n\nThe service account key should be the base64 encoded version of the JSON keyfile\nfor a service account with the appropriate permissions to create/read/write\nobjects in GCS and administer clusters in GKE. You can base64 encode the json\nkey by `cat key.json | base64 -w 0` and putting the output from that command\ninto the protected environment variable, either through the Travis website or\nthrough encrypting it in the .travis.yml file.\n\nThe service account also needs the pubsub publisher and pubsub viewer roles.\n\n### Deployment Configuration\nIn addition to the service account, when setting up a new travis deployment, you\nneed to configure the project name you're deploying to, the cluster name within\nthat project, and the bucket name you want the data saved to. Those are all\nparameters passed into the deploy.sh command, along with the service account\nJSON key, encoded in base64 form. The deploy.sh command takes the form:\n``` shell\ndeploy.sh \u003cproject name\u003e \u003ccluster name\u003e \u003cbucket name\u003e \\\n    \u003cbase64 service account key text\u003e\n```\n\n## Kubernetes Secrets\nIn order for downloader to be able to connect to GCS, it needs to have a service\naccount with access to GCS. You can use the same service account used for travis\ndeployment, if you wish. But you need to store the key file in a kubernetes\nsecret, named downloader-app-key, so that the deployment config can find and\nmount it for use by the app. You can set it with: \n\n``` shell\nkubectl create secret generic \\\n    downloader-app-key --from-file=key.json=/path/to/key.json\n\n```\n\n## Cluster Creation\n\nThe default cluster size is enough for the downloader, but you need to be sure\nto give it read/write permissions for GCS when you create the cluster.\n\nSo, we create a dedicated node pool with storage-rw permissions. Ultimately, a\nlimited permission service account would be preferable. Initially, three nodes\nwill be allocated, but the autoscaler will shut down two after the downloader\nis deployed.\n\nThe cluster node-pool is managed using Terraform, defined in\n[terraform-support](https://github.com/m-lab/terraform-support).\n\nFor prometheus monitoring, you must make an extra node pool, also managed by\nterraform.\n\n## Pub/Sub Topic\nThe downloader also expects a pub/sub topic named \"downloader-new-files\" to\nexist. The topic must be created in the project that the downloader is running\nin, otherwise the downloader will not start.\n\n## Prometheus Monitoring\nMost of the work for prometheus monitoring is done in the prometheus-support\nrepository. The only things you need to be aware of is that downloader exports\nsome prometheus metrics on /metrics, the containers will have the label so that\nprometheus scrapes them, and that if you are creating a new cluster for\ndownloader, you must follow the setup instructions in the prometheus-support\nrepo's readme.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fdownloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fdownloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fdownloader/lists"}