{"id":21629789,"url":"https://github.com/mozilla-services/foxsec-pipeline","last_synced_at":"2025-04-11T13:51:42.272Z","repository":{"id":33491799,"uuid":"151310686","full_name":"mozilla-services/foxsec-pipeline","owner":"mozilla-services","description":"Log analysis pipeline utilizing Apache Beam","archived":false,"fork":false,"pushed_at":"2023-07-05T20:59:23.000Z","size":14200,"stargazers_count":25,"open_issues_count":27,"forks_count":6,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-25T10:04:59.764Z","etag":null,"topics":["apache-beam","dataflow","log-analysis","security"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mozilla-services.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-02T19:23:37.000Z","updated_at":"2023-08-30T04:01:39.000Z","dependencies_parsed_at":"2024-06-19T00:15:04.577Z","dependency_job_id":"2d633a22-e14c-45da-97d6-ea3326211c88","html_url":"https://github.com/mozilla-services/foxsec-pipeline","commit_stats":{"total_commits":1093,"total_committers":10,"mean_commits":109.3,"dds":"0.28362305580969804","last_synced_commit":"7b880fda824ca99c9a612dc4aacf76484bcaa7f8"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Ffoxsec-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Ffoxsec-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Ffoxsec-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Ffoxsec-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mozilla-services","download_url":"https://codeload.github.com/mozilla-services/foxsec-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248411966,"owners_count":21099036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","dataflow","log-analysis","security"],"created_at":"2024-11-25T02:08:50.015Z","updated_at":"2025-04-11T13:51:42.252Z","avatar_url":"https://github.com/mozilla-services.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# foxsec-pipeline\n\n[![Build Status](https://circleci.com/gh/mozilla-services/foxsec-pipeline/tree/master.svg?style=svg)](https://circleci.com/gh/mozilla-services/foxsec-pipeline/tree/master)\n[![Documentation](https://img.shields.io/badge/documentation-link-purple.svg)](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/)\n\n[Apache Beam](https://beam.apache.org/) pipelines for analyzing log data.\n\n## Documentation\n\n* [secops-beam Java documentation](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/)\n\njavadoc documentation is currently updated manually and although should be up to date, may not be current\nwith master.\n\n## Introduction to Beam\n\nTo get familiar with developing pipelines in Beam, this repository also contains a small workshop that\nprovides some guidance on building basic pipelines. The introduction document can be found\n[here](docs/beam-intro/INTRO.md).\n\n## Tests\n\nTests can be executed locally using Docker.\n\n### Run all tests\n\n```bash\ndocker build -f Dockerfile-base -t foxsec-pipeline-base:latest .\nbin/m test\n```\n\n### Run a specific test\n\n```bash\ndocker build -f Dockerfile-base -t foxsec-pipeline-base:latest .\nbin/m test -Dtest=ParserTest\n```\n\n## CLI Usage\n\n### Pipeline [RuntimeSecrets](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/crypto/RuntimeSecrets.html)\n\nPipeline runtime secrets can be generated locally using the main method in the [`RuntimeSecrets`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/crypto/RuntimeSecrets.html) class.\n\n```bash\nbin/m compile exec:java -Dexec.mainClass=com.mozilla.secops.crypto.RuntimeSecrets -Dexec.args='-i testkey -k dataflow -p my-gcp-dataflow-project -r dataflow'\n```\n\nRun the class with no options to see usage information. Note that in this case, the key ring name and key name\nare being specified as `dataflow`. The existing `RuntimeSecrets` class requires the keys to be accessible\nusing these identifiers when the pipeline is executing.\n\nThe output of the command can be prefixed with `cloudkms://` in an option to enable runtime decryption of the secrets\nduring pipeline execution.\n\n### Interacting with Minfraud\n\nReputation data can be fetched from Minfraud locally using the main method in the [`Minfraud`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/Minfraud.html) class.\n\nYou must provide the accountid and licensekey plus the IP and/or email you want to get reputation data for. `--accountid` and `--licensekey` can either be provided directly or provided as RuntimeSecrets (`cloudkms://...`).\n\n```bash\nbin/m exec:java \\\n  -Dexec.mainClass=\"com.mozilla.secops.Minfraud\" \\\n  -Dexec.args=\"-p my-gcp-dataflow-project --accountid 'cloudkms://...' --licensekey 'cloudkms://...' --ip '8.8.8.8' --email 'example@example.com'\"\n```\n\n### Creating Watchlist entries\n\nWatchlist entries can be created locally using the main method in the [`Watchlist`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/Watchlist.html) class.\n\nYou must also prefix your command with `WITHOUT_DAEMONS=true` so that the entry won't be submitted to the Datastore emulator running within the container.\n\n```\nusage: Watchlist\n -c,--createdby \u003carg\u003e\n -ne,--neverexpires     Watchlist entry never expires (compared to default\n                        of 2 weeks)\n -o,--object \u003carg\u003e      Object to watch. Can be an IP or email.\n -p,--project \u003carg\u003e     GCP project name (required if submitting to\n                        Datastore)\n -s,--severity \u003carg\u003e    Severity of Watchlist entry. Can be 'info',\n                        'warn', or 'crit'\n -su,--submit           Submit Watchlist entry to Datastore rather than\n                        emit json\n -t,--type \u003carg\u003e        Type of object to watch. Can be 'ip' or 'email'\n```\n\n#### Example of creating entry without submitting to Datastore\n```bash\n$ bin/m exec:java -Dexec.mainClass=\"com.mozilla.secops.Watchlist\" -Dexec.args=\"--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info'\"\n\n{\"type\":\"ip\",\"severity\":\"info\",\"expires_at\":\"2020-02-26T17:45:01.399Z\",\"created_by\":\"example@example.com\",\"object\":\"127.0.0.1\"}\n```\n\n#### Example of submitting to Datastore\n```bash\n$ WITHOUT_DAEMONS=true bin/m exec:java -Dexec.mainClass=\"com.mozilla.secops.Watchlist\" -Dexec.args=\"--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info' --project foxsec-pipeline-nonprod --submit\"\n\nFeb 12, 2020 5:41:44 PM com.mozilla.secops.state.State initialize\nINFO: Initializing new state interface using com.mozilla.secops.state.DatastoreStateInterface\nFeb 12, 2020 5:41:45 PM com.mozilla.secops.state.StateCursor set\nINFO: Writing state for 127.0.0.1\nFeb 12, 2020 5:41:45 PM com.mozilla.secops.state.State done\nINFO: Closing state interface com.mozilla.secops.state.DatastoreStateInterface\nSuccessfully submitted watchlist entry to foxsec-pipeline-nonprod\n{\"type\":\"ip\",\"severity\":\"info\",\"expires_at\":\"2020-02-26T17:41:43.919Z\",\"created_by\":\"example@example.com\",\"object\":\"127.0.0.1\"}\n```\n\n## Contributing\n\nSee the [contributing guidelines](./CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-services%2Ffoxsec-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmozilla-services%2Ffoxsec-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-services%2Ffoxsec-pipeline/lists"}