zomato/vinifera

A GitHub recon/monitoring tool for finding internal leaks belonging to your organisation.

Vinifera - Monitor Internal Leaks on Github

Github Monitoring Tool 🤖

---

We have been using Vinifera in production since Dec 2019 and has helped us prevent security incidents. Vinifera started out as an internal project to ensure Security hygiene of our public contributions and monitor potential leaks on Github. We believe this will help other companies to strengthen their security hygiene when it comes to public sources like Github.

What is Vinifera?

Vinifera allows Companies/Organizations to monitor public assets to find references to internal code leaks and potential breaches.

Sometimes developers might leak internal code and credentials by accident. Vinifera aims to help companies detect those breaches in due time and respond to the incident.

How does it work?

Vinifera monitors developers belonging to the organization, monitors and scans public contributions to look for potential violations and breach of internal/secret/proprietary code by looking for references defined.

• Vinifera works by syncing org users.
• For each user, all the public assets are registered (if not already tracked).
• Each asset (Repo, Gist) is then scanned for any differences.
• Each difference is then scanned, stored, and reported (if contains any leaks)

What does the name mean?

During the development and inception of the tool, the Security team consumed lots of grapes, so we named it after the fruit we love :)

Vinifera is inspired from the Bionomial name of Grapes

Setup and Usage.

Requirements

Vinifera requires the installation of the following tools:

1. PostgreSQL
2. Redis
3. Docker
4. Ruby (Install via rbenv/rvm )

Setup

Github Access Token

To scan your organization members, Vinifera requires a token with the ability to read Organization members.

Generate a new token https://github.com/settings/tokens/new with no special scope

You would want to use the token of an admin user (with no special scope), since the admin can list all users of an organization.

https://docs.github.com/en/rest/reference/orgs#list-organization-members

If the authenticated user is also a member of this organization then both concealed and public members will be returned.

Docker

• Rename .docker_env.example to .docker_env

mv .docker_env.example .docker_env

• Update .docker_env with needed variables

GITHUB_ACCESS_TOKEN=<REDACTED>
VINIFERA_ORG_NAME=<Your_org_name>

RAILS_MAX_THREADS=60 # This also controls DB pool

RAILS_MASTER_KEY=<ADD_YOUR_MASTER_KEY_HERE>

# set the environment
RAILS_ENV=<development|production>

# set the following if RAILS_ENV is set to production
SECRET_KEY_BASE=<YOUR_SECRET_KEY> # min 32 bits key is allowed

# add slack incoming webhook URLs for respective slack channels
SLACK_UPDATES_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_TARGETS_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_USER_TRACKING_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_VINIFERA_VIOLATION_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_ERROR_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>

# By default fork and big fork scanning is disabled
VINIFERA_ENABLE_FORK_SCANNING=false
VINIFERA_ENABLE_BIG_FORK_SCANNING=false

• Build

docker-compose build

• Run

docker-compose up

• Re-building after any changes

docker-compose up --build

• Sync Github Org Users

docker exec -it vinifera_sidekiq bundle exec rake periodic_syncs:sync_github_users

---

Docker Compose Commands for Reference

---

Manual

• Install required dependencies

cd <location_of_cloned_repo>
bundle install

• Setup DB and migrations

bundle exec rails db:create
bundle exec rails db:migrate

• Setup Environment Variables and Slack WebHook Sample env file is available at .example_env

GITHUB_ACCESS_TOKEN=<REDACTED>
VINIFERA_ORG_NAME=<Your_org_name> # Name of org for which token was generated.

VINIFERA_DATABASE_HOST=<localhost>
VINIFERA_DATABASE_PASSWORD=<YOUR_PASSWORD>

RAILS_MAX_THREADS=60 # This also controls DB pool

RAILS_MASTER_KEY=<REDACTED>


SLACK_UPDATES_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_TARGETS_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_USER_TRACKING_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_VINIFERA_VIOLATION_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>
SLACK_ERROR_GROUP_URL=https://hooks.slack.com/services/<YOUR_CONFIG_HERE>



# Add this only if you are using Docker over TLS, recommended way, if, on the same host as vinifera, you may skip it

DOCKER_CLIENT_CERT_PATH=/home/deployer/.docker
DOCKER_HOST=tcp://<IP>:<PORT>

# By default fork and big fork scanning is disabled
VINIFERA_ENABLE_FORK_SCANNING=false
VINIFERA_ENABLE_BIG_FORK_SCANNING=false

REDIS_URL=redis://<REDIS_URI>
SIDEKIQ_REDIS_URL=redis://<REDIS_URI>

# For PagerDuty Integration
DEFAULT_PD_INTEGRATION_KEY=<xxxxxxxx>
ENABLE_PAGER_DUTY_TRIGGER=true

• Setup Cron Jobs

bundle exec whenever --update-crontab

• Setup your custom toml rules for Gitleaks. For more info

[[rules]]
# Insert your rules here
# description = "Internal References"
# regex = '''(?i)((.*)<COMPANY_INTERNAL_REFERENCES>(.*))'''
# tags = ["internal", "company","references"]

• Start Sidekiq

bundle exec sidekiq

Datadog

Additionally, to get the metrics on Datadog like in the above screenshot, you can use the DataDog agent - https://docs.datadoghq.com/agent/

PagerDuty

To ensure the team never misses any violation, PagerDuty integration option is there:

For PagerDuty integration, following environment variable needs to be set to true

ENABLE_PAGER_DUTY_TRIGGER=true

Then create a new service and a integration key as described in following doc - https://support.pagerduty.com/docs/services-and-integrations#create-a-new-service

DEFAULT_PD_INTEGRATION_KEY=<xxxxxxxx>

Contributing

We are open to contributions/bug fixes/performance improvements to our project :)

Donations

If you appreciate the tool we have built, feel free to contribute/donate to the projects on the top of which Vinifera was built :)

Vinifera is built on top of other open-source software:

1. Rails (Our Favourite Web Framework)
2. Sidekiq (Handles Job LifeCycle, Scheduling, and Retries)
3. Gitleaks (Gitleaks powers the code scanning via Docker Images)
4. Docker (For running Scans in an isolated environment)
5. Docker-api (Ruby Client to interact with Docker Remote API)
6. Sidekiq Throttled (For Throttling workers)
7. Octokit (Ruby Toolkit for Github API)
8. Whenever (Cron Jobs in Ruby)

.... (list will go on ..., you get the gist ;) )

You can also donate to Feeding India

Disclaimer

Neither Zomato nor the developers of this tool are responsible for any damage caused by this tool or usage of the same. Use responsibly. Refer to LICENSE for more details.

License

Vinifera is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.