cloudfoundry-community/logsearch-boshworkspace

The fastest way to get started with logsearch, cf-release and bosh-lite

logsearch-boshworkspace

The fastest way to deploy Logsearch in combination with Cloud Foundry onto bosh-lite.

Preparation

To get started you will need a running bosh-lite. Get yours by following the instructions here

Next step is setting up this repository

git clone https://github.com/cloudfoundry-community/logsearch-boshworkspace.git
cd logsearch-boshworkspace
bundle install

Filters

The logsearch community has created some pre-built filters. They are distributed as addon repositories (such as https://github.com/logsearch/logsearch-filters-cf).

A rake task is provided to fetch a subset of these addon filters and update the templates/ to include the logstash filters.

rake addons:update

Currently some addon's test suites fail - you can continue with:

rake addons:update_templates

Open issues for failing test suites for addons:

• 
• 

Deploy

This section includes instructions for configuration and deployment of Logsearch.

Deploy Logsearch to bosh-lite

bosh deployment logsearch-warden
bosh prepare deployment
bosh deploy

Since the deployment characteristics of Cloud Foundry on bosh-lite/warden are well known (static IPs, etc) you do not need to modify the deployments/logsearch-warden.yml file for this to work.

Deploy Logsearch to AWS VPC

Copy the logsearch-aws-vpc.yml or logsearch-aws-vpc-cf-route.yml to a new file that you will edit:

cp deployments/logsearch-aws-vpc-cf-route.yml deployments/my-logsearch.yml

Edit deployments/my-logsearch.yml with your:

• director UUID (run bosh status --uuid and populate into director_uuid)
• NATS host (run bosh vms --dns and populate into meta.cf.nats_servers.host)
• Network subnet (populate the subnet ID into meta.zones.z1.subnet_id and update other meta.zones.z1.* fields as necessary)
• Security group (use the same security groups being used for Cloud Foundry and populate into meta.security_groups)

bosh deployment my-logsearch
bosh prepare deployment
bosh deploy

Update Cloud Foundry with Syslog enabled

You can now re-deploy Cloud Foundry with syslog emitting to your Logsearch Add the following to your Cloud Foundry deployment manifest.

First, get the IP or hostname for ingestor/0 VM:

$ bosh vms --dns
Deployment `logsearch-aws-test'

+------------+---------+---------------+------------+-------------------------------------------------+
| Job/index  | State   | Resource Pool | IPs        | DNS A records                                   |
+------------+---------+---------------+------------+------------+------------------------------------+
| ingestor/0 | running | common        | 10.10.6.7  | 0.ingestor.default.logsearch-aws-test.microbosh |
...

In the example above, use either 10.10.6.7 or 0.ingestor.default.logsearch-aws-test.microbosh

Add the following to your Cloud Foundry deployment manifest (global properties as it is for all job templates):

properties:
  syslog_daemon_config:
    address: 10.10.6.7
    port: 5515
    transport: relp

Now redeploy Cloud Foundry:

$ bosh deploy

Access Kibana UI via Cloud Foundry router

Instead of logsearch-warden, use logsearch-warden-cf-route to make Kibana UI accessible via the public CF router (logs.DOMAIN)

Play time

Now you can browse to the Kibana dashboard here

logsearch-boshworkspace's Issues

bosh-lite command

From the readme:

bosh prepare deployment

is not a valid bosh-lite command

Error running rake plugins:update

I get this error running rake plugins:update:

dcallao@ip-10-202-86-106:/data/workspace/logsearch$ sudo rake plugins:update
sudo: unable to resolve host ip-10-202-86-106
rake aborted!
Don't know how to build task 'plugins:update'

(See full trace by running task with --trace)

CF filters addons missing

On file https://github.com/cloudfoundry-community/logsearch-boshworkspace/blob/master/Rakefile
at line 33:
logstash_filters: 'https://raw.githubusercontent.com/logsearch/logsearch-filters-cf/new-logsearch-addon-structure/target/logstash-filters-default.conf

The address return Not found, and the curl returns a file with 'Not found' text inside it

path 'logsearch-for-cloudfoundry-boshrelease' does not exist in the given tree (Rugged::TreeError)

ubuntu@bastion:~/workspace/deployments/logsearch-boshworkspace$ bosh prepare deployment

Fetching release 'logsearch' to satisfy template references
Version '19' has been checkout into:

• /home/ubuntu/workspace/deployments/logsearch-boshworkspace/.releases/logsearch
  Fetching release 'logsearch-for-cloudfoundry' to satisfy template references
  /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/workspace/release.rb:86:in path': the path 'logsearch-for-cloudfoundry-boshrelease' does not exist in the given tree (Rugged::TreeError) from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/workspace/release.rb:86:inreleases_tree'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/workspace/release.rb:94:in final_releases' from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/workspace/release.rb:120:inrelease'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/workspace/release.rb:16:in update_repo' from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/cli/commands/prepare.rb:32:inblock in prepare_release_repos'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/cli/commands/prepare.rb:28:in each' from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/cli/commands/prepare.rb:28:inprepare_release_repos'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.1/lib/bosh/cli/commands/prepare.rb:18:in prepare' from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.3098.0/lib/cli/command_handler.rb:57:inrun'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.3098.0/lib/cli/runner.rb:56:in run' from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.3098.0/bin/bosh:19:in<top (required)>'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/bosh:23:in load' from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/bosh:23:in'
  from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in eval' from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in'

logsearch-filters-common/bin/install_deps.sh: No such file or directory

$ rake filters:update

...

Adding community plugins ... (takes a few minutes)
./bin/install_deps.sh
./bin/install_deps.sh: line 4: /home/ubuntu/workspace/deployments/logsearch-boshworkspace/tmp/logsearch-filters-cf/vendor/logsearch-filters-common/bin/install_deps.sh: No such file or directory
rake aborted!
Command failed with status (127): [./bin/install_deps.sh...]
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:28:in `block (3 levels) in <top (required)>'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:38:in `block (3 levels) in in_filter_dirs'
/home/ubuntu/.rvm/gems/ruby-2.1.5@global/gems/bundler-1.7.6/lib/bundler.rb:236:in `block in with_clean_env'
/home/ubuntu/.rvm/gems/ruby-2.1.5@global/gems/bundler-1.7.6/lib/bundler.rb:223:in `with_original_env'
/home/ubuntu/.rvm/gems/ruby-2.1.5@global/gems/bundler-1.7.6/lib/bundler.rb:229:in `with_clean_env'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:38:in `block (2 levels) in in_filter_dirs'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:37:in `chdir'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:37:in `block in in_filter_dirs'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:36:in `each'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:36:in `in_filter_dirs'
/home/ubuntu/workspace/deployments/logsearch-boshworkspace/Rakefile:28:in `block (2 levels) in <top (required)>'
/home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => filters:update => filters:install_deps
(See full trace by running task with --trace)