Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/zomato/vinifera

A GitHub recon/monitoring tool for finding internal leaks belonging to your organisation.
https://github.com/zomato/vinifera

github recon security

Last synced: about 1 month ago
JSON representation

A GitHub recon/monitoring tool for finding internal leaks belonging to your organisation.

Host: GitHub
URL: https://github.com/zomato/vinifera
Owner: Zomato
License: apache-2.0
Created: 2021-02-10T11:31:14.000Z (almost 4 years ago)
Default Branch: master
Last Pushed: 2024-01-07T09:48:28.000Z (12 months ago)
Last Synced: 2024-05-11T21:33:53.139Z (8 months ago)
Topics: github, recon, security
Language: Ruby
Homepage:
Size: 477 KB
Stars: 88
Watchers: 10
Forks: 14
Open Issues: 9
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Vinifera - Monitor Internal Leaks on Github

#### Github Monitoring Tool :robot:

![Vinifera Logo](docs/img/vinifera.png "Vinifera Logo")

We have been using Vinifera in production since Dec 2019 and has helped us prevent security incidents.
Vinifera started out as an internal project to ensure Security hygiene of our public contributions and monitor potential leaks on Github.
We believe this will help other companies to strengthen their security hygiene when it comes to public sources like Github.

![Stats](docs/img/stats.png "Production Stats")

## What is Vinifera?
Vinifera allows Companies/Organizations to monitor public assets to find references to internal code leaks and potential breaches.

Sometimes developers might leak internal code and credentials by accident. Vinifera aims to help companies detect those breaches in due time and respond to the incident.

## How does it work?

Vinifera monitors developers belonging to the organization, monitors and scans public contributions to look for potential violations and breach of internal/secret/proprietary code by looking for references defined.

* Vinifera works by syncing org users.
* For each user, all the public assets are registered (if not already tracked).
* Each asset (Repo, Gist) is then scanned for any differences.
* Each difference is then scanned, stored, and reported (if contains any leaks)

![Vinifera Workflow](docs/img/workflow.png "Vinifera Workflow")

## What does the name mean?

During the development and inception of the tool, the Security team consumed lots of grapes, so we named it after the fruit we love :)

Vinifera is inspired from the [Bionomial name of Grapes](https://en.wikipedia.org/wiki/Vitis_vinifera)

## Setup and Usage.

### Requirements
Vinifera requires the installation of the following tools:
1. PostgreSQL
2. Redis
3. Docker
4. Ruby (Install via rbenv/rvm )

### Setup

##### Github Access Token
To scan your organization members, Vinifera requires a token with the ability to read Organization members.

Generate a new token [https://github.com/settings/tokens/new](https://github.com/settings/tokens/new) with no special scope

You would want to use the token of an admin user (with no special scope), since the admin can list all users of an organization.

[https://docs.github.com/en/rest/reference/orgs#list-organization-members](https://docs.github.com/en/rest/reference/orgs#list-organization-members)

> If the authenticated user is also a member of this organization then both concealed and public members will be returned.

#### Docker

* Rename `.docker_env.example` to `.docker_env`

```bash
mv .docker_env.example .docker_env
```

* Update `.docker_env` with needed variables

```bash
GITHUB_ACCESS_TOKEN=
VINIFERA_ORG_NAME=

RAILS_MAX_THREADS=60 # This also controls DB pool

RAILS_MASTER_KEY=

# set the environment
RAILS_ENV=

# set the following if RAILS_ENV is set to production
SECRET_KEY_BASE= # min 32 bits key is allowed

# add slack incoming webhook URLs for respective slack channels
SLACK_UPDATES_GROUP_URL=https://hooks.slack.com/services/
SLACK_TARGETS_GROUP_URL=https://hooks.slack.com/services/
SLACK_USER_TRACKING_GROUP_URL=https://hooks.slack.com/services/
SLACK_VINIFERA_VIOLATION_GROUP_URL=https://hooks.slack.com/services/
SLACK_ERROR_GROUP_URL=https://hooks.slack.com/services/

# By default fork and big fork scanning is disabled
VINIFERA_ENABLE_FORK_SCANNING=false
VINIFERA_ENABLE_BIG_FORK_SCANNING=false
```

* Build

```bash
docker-compose build
```
* Run

```bash
docker-compose up
```

* Re-building after any changes

```bash
docker-compose up --build
```

* Sync Github Org Users

```bash
docker exec -it vinifera_sidekiq bundle exec rake periodic_syncs:sync_github_users
```

[Docker Compose Commands for Reference](https://docs.docker.com/compose/reference/)

#### Manual

* Install required dependencies
```bash
cd
bundle install
```

* Setup DB and migrations
```bash
bundle exec rails db:create
bundle exec rails db:migrate
```

* Setup Environment Variables and Slack WebHook
Sample env file is available at `.example_env`
```bash
GITHUB_ACCESS_TOKEN=
VINIFERA_ORG_NAME= # Name of org for which token was generated.

VINIFERA_DATABASE_HOST=
VINIFERA_DATABASE_PASSWORD=

RAILS_MAX_THREADS=60 # This also controls DB pool

RAILS_MASTER_KEY=

SLACK_UPDATES_GROUP_URL=https://hooks.slack.com/services/
SLACK_TARGETS_GROUP_URL=https://hooks.slack.com/services/
SLACK_USER_TRACKING_GROUP_URL=https://hooks.slack.com/services/
SLACK_VINIFERA_VIOLATION_GROUP_URL=https://hooks.slack.com/services/
SLACK_ERROR_GROUP_URL=https://hooks.slack.com/services/

# Add this only if you are using Docker over TLS, recommended way, if, on the same host as vinifera, you may skip it

DOCKER_CLIENT_CERT_PATH=/home/deployer/.docker
DOCKER_HOST=tcp://:

# By default fork and big fork scanning is disabled
VINIFERA_ENABLE_FORK_SCANNING=false
VINIFERA_ENABLE_BIG_FORK_SCANNING=false

REDIS_URL=redis://
SIDEKIQ_REDIS_URL=redis://

# For PagerDuty Integration
DEFAULT_PD_INTEGRATION_KEY=
ENABLE_PAGER_DUTY_TRIGGER=true
```

* Setup Cron Jobs
```bash
bundle exec whenever --update-crontab
```
* Setup your custom toml rules for Gitleaks. [For more info](https://github.com/zricethezav/gitleaks#rules-summary)
```toml
[[rules]]
# Insert your rules here
# description = "Internal References"
# regex = '''(?i)((.*)(.*))'''
# tags = ["internal", "company","references"]
```

* Start Sidekiq
```bash
bundle exec sidekiq
```

##### Datadog

Additionally, to get the metrics on Datadog like in the above screenshot, you can use the DataDog agent - [https://docs.datadoghq.com/agent/](https://docs.datadoghq.com/agent/)

##### PagerDuty

To ensure the team never misses any violation, PagerDuty integration option is there:

![PagerDuty Incident](docs/img/pager_duty_incident.png "PagerDuty Incident")

For PagerDuty integration, following environment variable needs to be set to `true`

```bash
ENABLE_PAGER_DUTY_TRIGGER=true
```

Then create a new service and a integration key as described in following doc - https://support.pagerduty.com/docs/services-and-integrations#create-a-new-service

![PagerDuty Service](docs/img/pager_duty_service.png "PagerDuty Service")

```bash
DEFAULT_PD_INTEGRATION_KEY=
```

## Contributing

We are open to contributions/bug fixes/performance improvements to our project :)

## Donations

If you appreciate the tool we have built, feel free to contribute/donate to the projects on the top of which Vinifera was built :)

Vinifera is built on top of other open-source software:
1. [Rails](https://github.com/rails/rails) (Our Favourite Web Framework)
2. [Sidekiq](https://github.com/mperham/sidekiq) (Handles Job LifeCycle, Scheduling, and Retries)
3. [Gitleaks](https://github.com/zricethezav/gitleaks) (Gitleaks powers the code scanning via Docker Images)
4. [Docker](https://github.com/docker) (For running Scans in an isolated environment)
5. [Docker-api](https://github.com/swipely/docker-api) (Ruby Client to interact with Docker Remote API)
6. [Sidekiq Throttled ](https://github.com/sensortower/sidekiq-throttled) (For Throttling workers)
7. [Octokit](https://github.com/octokit/octokit.rb) (Ruby Toolkit for Github API)
8. [Whenever](https://github.com/javan/whenever) (Cron Jobs in Ruby)

.... (list will go on ..., you get the gist ;) )

You can also donate to [Feeding India](https://www.feedingindia.org/)

## Disclaimer

Neither Zomato nor the developers of this tool are responsible for any damage caused by this tool or usage of the same.
Use responsibly. Refer to [LICENSE](LICENSE) for more details.

## License

Vinifera is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text.