splunk/splunk-app-splunkgit

GitHub App

Splunk > Splunkgit

This application needs an awesome software called Splunk.

Currently the application is tested and the documentation written for Mac OS X 10.7

Stuff in this repository are mentioned on a four part blog series.

• part 1
• part 2
• part 3
• part 4

Released v1.3.0.1 (12/19/2012)

• Edited README.md!!

Released v1.3! (12/19/2012, because of it's been a long time)

• Fixes problem with grep -P, where -P is not supported by all grep versions.
• Fixes git author page for Splunk 5.x
• Some extractions of code, to hopefully make somethings easier to understand.
• Splunking commit messages (No dashboards with this data yet)
• Can splunk any source files (Non are enabled and there are no dashboards yet)

Released v1.2.1! (02/15/2012)

• All repositories are now non-flash, aka mobile supported.
• Increased the days shown in multiple repositories from last 2 weeks to last 30 days.

Released v1.2! (01/16/2012)

• You can now watch multiple repositories in semi real-time!
• Less configuration
• Faster updating scripts

Getting started

Installing splunk

Generic instructions

• Download Splunk for your platform.
• Unpack/Install Splunk by running the downloaded files.
• Follow the instructions on screen
• When done continue to Installing Splunkgit App section

<SPLUNK_ROOT> will from now on refer to the direcotry where you've extracted splunk.

Installing splunkgit

• Make sure splunk is not running
• Open Terminal app
• Goto <SPLUNK_ROOT>: cd <SPLUNK_ROOT>
• Goto app directory: cd etc/apps
• Download the app: git clone git://github.com/splunk/splunk-app-splunkgit.git
  You can also download a released version from the tags page.

Configuration

• Make sure splunk is not running
• Open Terminal app
• Goto <SPLUNK_ROOT>/etc/apps/splunk-app-aplunkgit
• Edit local/splunkgit.conf with a text editor (open -e local/splunkgit.conf) and assign the following values:
  • repo_addresses= The addresses to the repos, use the read-only address. Ex: git://github.com/splunk/splunk-app-splunkgit.git. You can have one or multiple repositories, space separated
  • user= Splunk user login so our scripts can search in Splunk
  • password= Splunk password for the user

Configurating multiple repositories in semi real-time

• Edit local/splunkgit.conf and assign repo_addresses= with multiple repositories by separating the repositories with a space. Ex: repo_address=git://github.com/splunk/splunk-app-splunkgit.git git://github.com/splunk/splunk-sdk-java.git git://github.com/splunk/splunk-sdk-python.git

• Copy default/inputs.conf to the local directory

• Set the interval value of the fetch_git_repo_data.sh script to a low value. Ex: 20 The git repositores will now be updated each 20 seconds. The views in multiple repositories dashboard will be updated whenever there's more data.

Changing repository

• Make sure splunk is not running

• Run the following command to wipe all app data from splunk:

    splunk clean eventdata -f -index splunkgit
  

• Change the splunkgit.conf file, as described in Configuration section, to point to the new repo.

Starting and stopping Splunk

• Open Terminal
• Goto <SPLUNK_ROOT>: cd <SPLUNK_ROOT>
• Start splunk bin/splunk start
  • On you web browser goto http://localhost:8000
  • If asked enter your name and user name (default value is admin:changeme)
  • If you change the password, you also need the change the configuration file to match this.
• Stop splunk: bin/splunk stop

Third party libraries

• httplib2
• joblib

Known issues

• If you clone this repository, install the app and start up Splunk without configurating your own splunkgit.conf (as explained in Changing repository) splunk will get git repository data from this repositories .git directory.

License

Copyright 2012 Splunk, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

splunk-app-splunkgit's Issues

The local folder/conf is missing

When you install this app the splunkgit.conf is missing in local which is kinda confusing since you need to edit that.

Git/Repository view shouldn't requires flash

Unable to connect to private repository

@petterik

Hello @petterik

I've just installed the splunkgit app on our internal splunk instance, and while it seems to work okay connecting to public repositories, when I provide a read-only account git url it doesn't seem to like this.

I'm perusing the code base to see if you are able to support private repositories or not, although the wiki seemed to mention support of both..

cc @jumanjiman

Don't force a user to enter his password in plain text

Currently you require a user to enter a password into a conf file.

Instead of performing searches when fetching data for deduping you should have a search that runs which pipes that info to a csv-file instead and read that file when the script is run.

Get rid of sed

Use the filtering built in to awk instead.

$> cat file.txt  | sed '/^$/d'

can be done as:

$>  cat file.txt | awk '$0!~/^$/ {print $0}'

(i.e. if the whole line is not empty, print the whole line)

That way you can get rid of sed, and put the whole awk program into a file, add any additional conditions for which lines to care about or ignore, format the code nicely and use it as an arg to awk:

$> cat file.txt | awk -f print_non-empty_lines.awk 

Thankyou robinsp for pointing it out.

Github/Repo statistics view shouldn't requires flash

Separate git_fetch_repo_data.sh and multi_repository.xml generation

Right now git_fetch_repo_data.sh is doing both fetching and generating the multi_repository.xml.

These should be separated into two scripts.
They should also be set to different timers in inputs.conf.

The multi_repository generating script should only have to be run at start-up/configuration change.
git_fetch_repo_data.sh could be run all the time, or intervals of maybe 20-30secs?

<option name="charting.axisY">numeric</option> is not supported with HTML5

Did: Went to look at the multi-git//repositories view on my iPad
Expected: Everything to be viewed since Splunk 4.3 is not flash anymore.
Happened: Did not see any of the graphs.

What to do:
Remove numeric from the views, since it's not HTML5 supported. Splunk then rollsback to a flash view instead.

Git/Files view shouldn't requires flash

Splunk 6.5 and accessing git using https

I have this installed in a test 6.5.2 server and made a simple minded chnage to point this to our local git repo with a https URL - before I start debugging this do you have any advice or suggestions about how to make this work?

splunk cmd ./fetch_git_repo_data.sh
repository does not exist!
fetching git repo data for repository: https://xxxxxxx:[email protected]/repo/foo
Cloning into bare repository '/opt/splunk/etc/apps/splunk-app-splunkgit/git-repositories/foo'...
fatal: unable to access 'https://xxxxxxx:[email protected]/repo/foo/': Failed connect to server.foo.comt:443; Connection timed out
Unable to clone repository: https://xxxxxxx:[email protected]/repo/foo
repository does not exist!

Splunk host and port should be configurable

Did: Started Splunk on port 10000 and 10089 instead of the default 8000 and 8089.
Expected: All views to show.
Happened: Multi-Git never appeared.

Multi-Git/repository view is generated by a script and then calls a REST endpoint when it's done. This REST endpoint is hardcoded to be 8089. Also the host is hardcoded to localhost.

These should be configurable and the whole code should be searched for localhost:8000 and localhost:8089.