{"id":21888692,"url":"https://github.com/swisscom/cf-scraper","last_synced_at":"2025-04-15T10:20:03.916Z","repository":{"id":42349634,"uuid":"188909211","full_name":"swisscom/cf-scraper","owner":"swisscom","description":"A simple app which scrapes information about Cloud Foundry orgs","archived":false,"fork":false,"pushed_at":"2024-10-16T15:16:36.000Z","size":67,"stargazers_count":1,"open_issues_count":5,"forks_count":3,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-04-15T10:19:40.882Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/swisscom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-27T21:07:04.000Z","updated_at":"2024-10-16T15:16:39.000Z","dependencies_parsed_at":"2024-10-18T07:12:34.946Z","dependency_job_id":null,"html_url":"https://github.com/swisscom/cf-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swisscom%2Fcf-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swisscom%2Fcf-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swisscom%2Fcf-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swisscom%2Fcf-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/swisscom","download_url":"https://codeload.github.com/swisscom/cf-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249048753,"owners_count":21204307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-11-28T11:16:41.480Z","updated_at":"2025-04-15T10:20:03.896Z","avatar_url":"https://github.com/swisscom.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cf-scraper\nA simple app which scrapes information about Cloud Foundry orgs\n\n## How to deploy\n\n### Prepare the service instances\n`cf-scraper` gets the credentials for accessing the Cloud Foundry API via a service of type `secret-store`. Furthermore, it gets the list input orgs from an S3 service instance and also writes the scrape output to that S3 service instance.\n\n1. Create a user in your Cloud Foundry instance with the role `cloud_controller.global_auditor`.\n````\nuaac user add $AUDITOR_USER_NAME --emails $AUDITOR_EMAIL;\nuaac member add cloud_controller.global_auditor $AUDITOR_USERNAME;\n````\n2. Create a secrets store called `cf-api-credentials`.\n````\ncf cs secrets-store json cf-api-credentials -c '{\"username\": \"'$AUDITOR_USER_NAME'\", \"password\": \"'$AUDITOR_PASSWORD'\"}'\n````\n3. Create an S3 service instance named `orgs-store`.\n````\ncf cs dynstrg-2 usage orgs-store\n````\n\n### Load the input orgs\nThe scraper uses a file called `input/input-orgs.json` in the `orgs-store` instance. The list can be changed at any time. The scraper will pick up the latest version when it starts the next run.\n\n4. Prepare the file `input-orgs.json` to contain an array of org names.\n````\n[\n  \"org-1\",\n  \"org-2\",\n  \"org-3\",\n  ...,\n  \"org-n\"\n]\n````\n\n5. Upload the file to `orgs-store/input`, for example using [mc](https://github.com/minio/mc).  \n\n### Adapt the schedule\nThe scraper runs as a scheduled task. The schedule is defined as a cron expression in the environment variable `SYNC_SCHEDULE`. \n\n6. Open `manifest.yml` and set `SYNC_SCHEDULE` to the desired cron expression (e.g. `*/15 * * * *` for \"at every 15th minute\").\n\n### Push the app\n7. Everything else is self configuring. Just push the app.\n````\ncf push\n````\n\n## Collect the scrape result\nThe scraper uploads the result of a scrape run to `orgs-store/output/scrape-result.json`. Before starting the upload, a backup copy of the previous result is made called `scrape-result-backup.json`.\n\n8. Download `orgs-store/output/scrape-result.json`, for example using [mc](https://github.com/minio/mc).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswisscom%2Fcf-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswisscom%2Fcf-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswisscom%2Fcf-scraper/lists"}