{"id":20102920,"url":"https://github.com/stackrox/external-network-pusher","last_synced_at":"2025-05-06T08:31:00.510Z","repository":{"id":57661349,"uuid":"304165709","full_name":"stackrox/external-network-pusher","owner":"stackrox","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-13T12:14:53.000Z","size":341,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-13T13:25:25.144Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackrox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-15T00:14:39.000Z","updated_at":"2024-11-13T12:14:57.000Z","dependencies_parsed_at":"2023-10-11T11:10:37.048Z","dependency_job_id":"f78c5dc8-175a-4769-9b36-36da16d53298","html_url":"https://github.com/stackrox/external-network-pusher","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fexternal-network-pusher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fexternal-network-pusher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fexternal-network-pusher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackrox%2Fexternal-network-pusher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackrox","download_url":"https://codeload.github.com/stackrox/external-network-pusher/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224495973,"owners_count":17321023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:33:29.621Z","updated_at":"2024-11-13T17:33:30.284Z","avatar_url":"https://github.com/stackrox.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# external-network-pusher\n\nThis repo crawls list of external network providers and publishes the info\nto a user specified Google bucket. The current list of Providers include:\n- Google Cloud\n- Amazon AWS\n- Microsoft Azure\n- Oracle OCI\n- Cloudflare\n\n# Code structure\ncmd/network-crawler.go\n- Contains the main script which crawls and publishes the external network info.\n\nimage\n- Contains the Dockerfile to package this script into a docker image\n\npkg/common\n- Contains some util functions and commons types/constants definitions. Crawler interface\n  definition is also defined here under types.go\n\npkg/crawlers\n- Contains provider specific implementations of crawler instances.\n\n## How to build and run external-network-pusher?\nTo build, we can simply run\n```bash\nmake build\n```\nThis would create a `.gobin` directory and build a binary suited to your OS platform.\n\nTo create an image, just do\n```bash\nmake image\n```\nor to push to StackRox registry under project `stackrox-hub`, do\n```bash\nmake push\n```\n\n## How to run crawlers?\nAfter building the binary, just do\n```bash\n.gobin/network-crawler --bucket-name \u003cGCS bucket name\u003e\n```\nwhere bucket name is the name of the GCS bucket you want to upload to. It is a required field.\n\nBy default it would crawl all the providers listed above, alternatively you can crawl specific set of providers by specifying providers to skip. For example, if you want to skip crawling for Google Cloud and Amazon AWS, do\n```bash\n.gobin/network-crawler --bucket-name \u003cGCS bucket name\u003e --skipped-providers Google,Amazon\n```\nPlease see `--help` for full list of options.\n\n\n### Output structure\nThis script uploads to the user specified bucket in the following manner. Under the bucket, you should see:\n\nexternal-networks/latest_metadata\n- File which contains metadata of the latest networks. All consumers of the crawled network data should only look at this file for the filename which contains the latest networks.\n\nexternal-networks/\\\u003ctimestamp\\\u003e_\\\u003cdynamic_uuid\\\u003e/networks\n- Main file that contains all the provider networks.\n\nexternal-networks/\\\u003ctimestamp\\\u003e_\\\u003cdynamic_uuid\\\u003e/checksum\n- Contains the checksum for the above networks file. `latest_metadata` file also contains this info for the latest network data.\n\nLater runs will not overwrite the previous run outputs. Instead, it will upload a new version of the network data, and change the content of the `latest_metadata` file.\n\nAs of now the script only keeps 10 run records in the bucket. If the script detected that there are more than 10 records, it starts deleting from the oldest one according to timestamp.\n\n### URL endpoints\nURL endpoints used by this crawler is defined in `pkg/common/constants.go`. Please check file for the URLs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackrox%2Fexternal-network-pusher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackrox%2Fexternal-network-pusher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackrox%2Fexternal-network-pusher/lists"}