https://github.com/mozilla-services/foxsec-pipeline
Log analysis pipeline utilizing Apache Beam
https://github.com/mozilla-services/foxsec-pipeline
apache-beam dataflow log-analysis security
Last synced: about 1 year ago
JSON representation
Log analysis pipeline utilizing Apache Beam
- Host: GitHub
- URL: https://github.com/mozilla-services/foxsec-pipeline
- Owner: mozilla-services
- License: mpl-2.0
- Created: 2018-10-02T19:23:37.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2023-07-05T20:59:23.000Z (almost 3 years ago)
- Last Synced: 2025-03-25T10:04:59.764Z (over 1 year ago)
- Topics: apache-beam, dataflow, log-analysis, security
- Language: Java
- Homepage:
- Size: 13.5 MB
- Stars: 25
- Watchers: 13
- Forks: 6
- Open Issues: 27
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# foxsec-pipeline
[](https://circleci.com/gh/mozilla-services/foxsec-pipeline/tree/master)
[](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/)
[Apache Beam](https://beam.apache.org/) pipelines for analyzing log data.
## Documentation
* [secops-beam Java documentation](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/)
javadoc documentation is currently updated manually and although should be up to date, may not be current
with master.
## Introduction to Beam
To get familiar with developing pipelines in Beam, this repository also contains a small workshop that
provides some guidance on building basic pipelines. The introduction document can be found
[here](docs/beam-intro/INTRO.md).
## Tests
Tests can be executed locally using Docker.
### Run all tests
```bash
docker build -f Dockerfile-base -t foxsec-pipeline-base:latest .
bin/m test
```
### Run a specific test
```bash
docker build -f Dockerfile-base -t foxsec-pipeline-base:latest .
bin/m test -Dtest=ParserTest
```
## CLI Usage
### Pipeline [RuntimeSecrets](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/crypto/RuntimeSecrets.html)
Pipeline runtime secrets can be generated locally using the main method in the [`RuntimeSecrets`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/crypto/RuntimeSecrets.html) class.
```bash
bin/m compile exec:java -Dexec.mainClass=com.mozilla.secops.crypto.RuntimeSecrets -Dexec.args='-i testkey -k dataflow -p my-gcp-dataflow-project -r dataflow'
```
Run the class with no options to see usage information. Note that in this case, the key ring name and key name
are being specified as `dataflow`. The existing `RuntimeSecrets` class requires the keys to be accessible
using these identifiers when the pipeline is executing.
The output of the command can be prefixed with `cloudkms://` in an option to enable runtime decryption of the secrets
during pipeline execution.
### Interacting with Minfraud
Reputation data can be fetched from Minfraud locally using the main method in the [`Minfraud`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/Minfraud.html) class.
You must provide the accountid and licensekey plus the IP and/or email you want to get reputation data for. `--accountid` and `--licensekey` can either be provided directly or provided as RuntimeSecrets (`cloudkms://...`).
```bash
bin/m exec:java \
-Dexec.mainClass="com.mozilla.secops.Minfraud" \
-Dexec.args="-p my-gcp-dataflow-project --accountid 'cloudkms://...' --licensekey 'cloudkms://...' --ip '8.8.8.8' --email 'example@example.com'"
```
### Creating Watchlist entries
Watchlist entries can be created locally using the main method in the [`Watchlist`](https://mozilla-services.github.io/foxsec-pipeline/secops-beam/com/mozilla/secops/Watchlist.html) class.
You must also prefix your command with `WITHOUT_DAEMONS=true` so that the entry won't be submitted to the Datastore emulator running within the container.
```
usage: Watchlist
-c,--createdby
-ne,--neverexpires Watchlist entry never expires (compared to default
of 2 weeks)
-o,--object Object to watch. Can be an IP or email.
-p,--project GCP project name (required if submitting to
Datastore)
-s,--severity Severity of Watchlist entry. Can be 'info',
'warn', or 'crit'
-su,--submit Submit Watchlist entry to Datastore rather than
emit json
-t,--type Type of object to watch. Can be 'ip' or 'email'
```
#### Example of creating entry without submitting to Datastore
```bash
$ bin/m exec:java -Dexec.mainClass="com.mozilla.secops.Watchlist" -Dexec.args="--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info'"
{"type":"ip","severity":"info","expires_at":"2020-02-26T17:45:01.399Z","created_by":"example@example.com","object":"127.0.0.1"}
```
#### Example of submitting to Datastore
```bash
$ WITHOUT_DAEMONS=true bin/m exec:java -Dexec.mainClass="com.mozilla.secops.Watchlist" -Dexec.args="--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info' --project foxsec-pipeline-nonprod --submit"
Feb 12, 2020 5:41:44 PM com.mozilla.secops.state.State initialize
INFO: Initializing new state interface using com.mozilla.secops.state.DatastoreStateInterface
Feb 12, 2020 5:41:45 PM com.mozilla.secops.state.StateCursor set
INFO: Writing state for 127.0.0.1
Feb 12, 2020 5:41:45 PM com.mozilla.secops.state.State done
INFO: Closing state interface com.mozilla.secops.state.DatastoreStateInterface
Successfully submitted watchlist entry to foxsec-pipeline-nonprod
{"type":"ip","severity":"info","expires_at":"2020-02-26T17:41:43.919Z","created_by":"example@example.com","object":"127.0.0.1"}
```
## Contributing
See the [contributing guidelines](./CONTRIBUTING.md).