{"id":18508437,"url":"https://github.com/ossf/package-feeds","last_synced_at":"2025-07-22T03:06:25.622Z","repository":{"id":38069194,"uuid":"317620857","full_name":"ossf/package-feeds","owner":"ossf","description":"Feed parsing for language package manager updates","archived":false,"fork":false,"pushed_at":"2024-12-04T14:35:33.000Z","size":585,"stargazers_count":79,"open_issues_count":33,"forks_count":24,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-25T10:03:37.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ossf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-01T17:41:52.000Z","updated_at":"2025-04-26T12:56:20.000Z","dependencies_parsed_at":"2023-02-18T07:45:30.609Z","dependency_job_id":"9cc2751e-a9ba-48a1-b882-ba67f353f144","html_url":"https://github.com/ossf/package-feeds","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ossf/package-feeds","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf%2Fpackage-feeds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf%2Fpackage-feeds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf%2Fpackage-feeds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf%2Fpackage-feeds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ossf","download_url":"https://codeload.github.com/ossf/package-feeds/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossf%2Fpackage-feeds/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266417114,"owners_count":23925301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:14:22.110Z","updated_at":"2025-07-22T03:06:25.590Z","avatar_url":"https://github.com/ossf.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Package Feeds\n\nThe binary produced by [cmd/scheduled-feed/main.go](cmd/scheduled-feed/main.go) can be used to monitor various\npackage repositories for changes and publish data to external services for further processing.\n\nAdditionally, the repo contains a few subprojects to aid in the analysis of these open source packages, in particular to look for malicious software.\n\nThese are:\n\n[Feeds](./pkg/feeds/) to watch package registries (PyPI, NPM, etc.) for changes to packages\nand to make that data available via a single standard interface.\n\n[Publisher](./pkg/publisher/) provides the functionality to push package details from feeds towards\nexternal services such as GCP Pub/Sub. Package details are formatted inline with a versioned\n[json-schema](./package.schema.json).\n\nThis repo used to contain several other projects, which have since been split out into\n[github.com/ossf/package-analysis](https://github.com/ossf/package-analysis).\n\nThe goal is for all of these components to work together and provide extensible, community-run\ninfrastructure to study behavior of open source packages and to look for malicious software.\nWe also hope that the components can be used independently, to provide package feeds or runtime\nbehavior data for anyone interested.\n\n# Configuration\n\nA YAML configuration file can be provided with the following format:\n\n```\nfeeds:\n- type: pypi\n- type: npm\n- type: goproxy\n- type: rubygems\n- type: crates\n\npublisher:\n  type: 'gcp_pubsub'\n  config:\n    url: \"gcppubsub://foobar.com\"\n\nhttp_port: 8080\n\npoll_rate: 5m\n\ntimer: false\n```\n\n`poll_rate` string formatted for [duration parser](https://golang.org/pkg/time/#ParseDuration).This is used as an initial value to generate a cutoff point for feed events relative to the given time at execution, with subsequent events using the previous time at execution as the cutoff point.\n`timer` will configure interal polling of the `feeds` at the given `poll_rate` period, individual feeds configured with a `poll_rate` will poll on an interval regardless of these options. To specify this configuration file, define its path in your environment under the `PACKAGE_FEEDS_CONFIG_PATH` variable.\n\nAn event handler can be configured through the `events` field, this is documented in the [events README](./pkg/events/README.md).\n\n## FeedOptions\n\nFeeds can be configured with additional options, not all feeds will support these features. Check [feeds/README.md](./pkg/feeds/README.md) for more information on feed specific configurations.\n\nBelow is an example of such options with pypi being configured to poll a specific set of packages\n\n```\nfeeds:\n- type: pypi\n  options:\n    packages:\n    - fooPackage\n    - barPackage\n```\n\n## Legacy Configuration\n\nLegacy configuration methods are still supported. By default, without a configuration file all feeds will be enabled. The environment variable `OSSMALWARE_TOPIC_URL` can be used to select the GCP pubsub publisher and `PORT` will configure the port for the HTTP server.\nThe default `poll_rate` is 5 minutes, it is assumed that an external service is dispatching requests to the configured HTTP server at this frequency.\n\n# Running Locally\n\nTo start an instance of `package-feeds` running on a local machine, run the\nfollowing commands:\n\n```shell\n$ docker build . -t local-package-feeds        ## Build the container\n$ mkdir /tmp/feedconfig/                       ## Create the feeds.yml config\n$ cat \u003c\u003c EOF \u003e\u003e /tmp/feedconfig/feeds.yml\npublisher:\n  type: stdout\npoll_rate: 5m\nhttp_port: 8080\nEOF\n$ docker run -v /tmp/feedconfig:/config/ \\\n  -e \"PACKAGE_FEEDS_CONFIG_PATH=/config/feeds.yml\" \\\n  -p 8080:8080 --rm -ti local-package-feeds    ## Start the container\n```\n\n# Contributing\n\nIf you want to get involved or have ideas you'd like to chat about, we discuss this project in the [OSSF Securing Critical Projects Working Group](https://github.com/ossf/wg-securing-critical-projects) meetings.\n\nSee the [Community Calendar](https://calendar.google.com/calendar?cid=czYzdm9lZmhwNWk5cGZsdGI1cTY3bmdwZXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) for the schedule and meeting invitations.\n\nPRs are linted using `golangci-lint` with the following [config file](./.golangci.yml). If you wish to run this locally, see the [install docs](https://golangci-lint.run/usage/install/#local-installation).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossf%2Fpackage-feeds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fossf%2Fpackage-feeds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossf%2Fpackage-feeds/lists"}