fatherlinux / petit Goto Github PK

Log analysis program for use from the command line

License: Other

Shell 1.34% Makefile 0.17% Python 13.08% GLSL 84.61% Roff 0.80%

petit's Introduction

Download and Information
===============================================================================
http://crunchtools.com/software/petit/

Why
===============================================================================
Log analysis is something that all systems administrators know they need to do. 
Many of us come to this point, either because there is a problem, there is a 
security requirement from the organization, or it keeps you up all night 
wanting to know what is going on in all of that data.

Looking for best practices for log analysis on this Internet is difficult at 
best. Many years ago, I discovered a script that hashed log files by removing 
all of their numbers and replacing them with "#" characters. The results of 
this simple algorithm were phenomenal, logs could be reduced by a factor of 
ten. This was much more readable, yet left much of the quality data that I 
needed to determine if there was a problem.

In the years since I discovered that simple algorithm, I have come to discover 
many techniques on text analysis which are commonly used in linguistics and 
anthropology to analyze natural languages. This has led me to develop very 
simple best practices for analyzing logs.

The Basics
===============================================================================
 1. Logs are made up of output which are programmed by human beings. There 
are no real restraints on what is output, other than, some cultural rules on 
being professional. This makes the output from programs very much a natural i
language. This also makes the output of someones program an approximation of 
the reality of what is happening inside a program. This is important to 
remember, logs are not perfect.

 2. When a systems administrator analyzes logs by changing them, he is 
creating an approximation of an approximation of reality in side a working 
program. This is not necessarily a bad thing, especially, when the programmer 
never gives you better than his approximation of reality anyway.

 3. In practice logs are made up of certainty and uncertainty. For example, I 
know what OpenSSH puts in the log during a login, because it is common. On the 
other hand, I do not now what a Compaq DL380 G3 will put in the log when it has 
a disk controller error. This is important to remember.

 4. The basic log analysis algorithm in Petit works to remove certainty, while 
leaving uncertainty. Stated another way, Petit quantitatively removes certainty, 
thereby leaving uncertainty, which by necessity requires qualitative analysis 
from a systems administrator

 5. After the algorithm has been applied, the output must be read by a systems 
administrator to determine if it is a normal or abnormal. Then abnormal entries 
can be acted on, hopefully before there is noticeable impact to your system. 


Installing/Uninstalling
===============================================================================
Installation can be done with RPM, DEB or TAR

INSTALL
========================================

TAR
 make install

RPM
 rpm -ivh petit.rpm

DEB
 dpkg -i petit.deb

UNINSTALL
========================================

TAR
 make uninstall

RPM
 rpm -e petit

DEB
 dpkg -r petit


Building Packages
===============================================================================
Two forms of building are maintained for sanity, deb and rpm. The build scripts
for these two package managers are distributed with petit for convenience and 
are used internally by the project  as part of a larger script to help 
distribute snapshots on our site. Usage is simple, currently no make install is
supported, but may be part of future distributions.

RPM
 make rpm

DEB
 make deb


Routine Operations
===============================================================================

Hash a syslog, removing reboots and all standard filters. By default petit will 
show a sample for all entries which are found three or less times.

	petit --hash --fingerprint /var/log/messages

Hash an Apache log

	petit --hash /var/log/httpd/access_log

Get a daemons report

	petit --daemon /var/log/messages

Get a host report

	petit --host /var/log/messages

Find qualitatively important words in your log. This is especially useful to 
help determine what should be monitored in swatch.

	petit --wordcount /var/log/messages

Graph first 60 seconds in a syslog

	petit -sgraph /var/log/messages

Track a special work you are interested in by minute

	cat /var/log/messages | grep error | petit --mgraph

Show samples for each entry

	petit --hash --allsample /var/log/messages

Special Operations
===============================================================================

Create an on the fly driver for a nonstandard file format, then pipe it to Petit. 
Petit can hash files of non-standard types ok, but graphing requires the time 
values to be in the correct columns.

	cat /var/log/httpd/error_log | awk '{$1="";$5="";print}' | lt --sgraph

petit's People

Contributors

Stargazers

Watchers

Forkers

bbneo rubbsdecvik jmccar wwalker cgasp dittoalex jgru iranzo chazzrobbz

petit's Issues

Unsupported timestamp format: 2020-09-27T00:00:03.766699+00:00

If I swap the T with a space in this timestamp convention, parsing works - can this format with T be supported?

cat test1.txt

2020-09-27T00:00:03.766699+00:00 hostxxxxx su: pam_unix(su:session): session closed for user yyyy

petit --sgraph ./test1.txt

Cannot parse values on line: 0

cat test2.txt

2020-09-27 00:00:03.766699+00:00 hostxxxxx su: pam_unix(su:session): session closed for user yyyy

petit --sgraph ./test2.txt

############################################################
01 31 00

Start Time: 1900-01-01 01:01:01 Minimum Value: 0
End Time: 1900-01-01 01:02:00 Maximum Value: 1
Duration: 60 seconds Scale: 0.166666666667

petit: traceback if reading file fails due to permission errors

Hi,

A Debian user (Michael Prokop <[email protected]>) encountered a bug with Petit : 

-----8<-----

Package: petit
Version: 1.0.3-1
Severity: normal


% petit --hash  ~log/syslog
Traceback (most recent call last):
  File "/usr/bin/petit", line 348, in <module>
    get_options(sys.argv[1:])
  File "/usr/bin/petit", line 135, in get_options
    hash(filename)
  File "/usr/bin/petit", line 164, in hash
    log = Log(args)
  File "/usr/share/petit/crunchtools.py", line 620, in __init__
    buffer = self.open_file(filename)
  File "/usr/share/petit/crunchtools.py", line 649, in open_file
    f = open(filename)                         
IOError: [Errno 13] Permission denied: '/var/log/syslog'

According exception handling without displaying the
traceback to the user would be nice. :)

regards,
-mika-

-----8<-----
The complete Debian bug report is available here : 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588900

Best regards,
Carl Chenet - Debian maintainer

Blog: http://carlchenet.wordpress.com
Identi.ca: http://identi.ca/carlchenet

Original issue reported on code.google.com by [email protected] on 15 Jul 2010 at 1:30

Error in Debian package: Version string '1.1.1_1': invalid character in version number

dpkg -i petit-current.deb
dpkg: error processing petit-current.deb (--install):
 parsing file '/var/lib/dpkg/tmp.ci/control' near line 2 package 'petit':
 error in Version string '1.1.1_1': invalid character in version number
Errors were encountered while processing:
 petit-current.deb

Debian 7 x64

Original issue reported on code.google.com by [email protected] on 27 Jul 2014 at 11:00

Feature request : package petit on Pypi

Hi,

It would be very handy to package petit as a package on Pypi.

This should only require to add a setup.py and would make installing petit without sudo rights a breeze using pip.

Traceback when there's no file permissions

Traceback (most recent call last):
  File "/home/iranzo/.local/bin/petit", line 446, in <module>
    get_options()
  File "/home/iranzo/.local/bin/petit", line 233, in get_options
    eval(options.mode + "()")
  File "<string>", line 1, in <module>
  File "/home/iranzo/.local/bin/petit", line 246, in mode_hash
    log = CrunchLog(filename)
  File "/home/iranzo/.local/venvs/petitlog/lib/python3.7/site-packages/crunchtools/CrunchLog.py", line 68, in __init__
    self.f = open(filename)
PermissionError: [Errno 13] Permission denied: '/var/log/messages'
Exception ignored in: <function CrunchLog.__del__ at 0x7f4b68b4c290>
Traceback (most recent call last):
  File "/home/iranzo/.local/venvs/petitlog/lib/python3.7/site-packages/crunchtools/CrunchLog.py", line 98, in __del__
    self.f.close()
AttributeError: 'CrunchLog' object has no attribute 'f'

Crashes on large files

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

[root@zoltan ~]# (zcat /var/log/snmpd.log.[1-9].gz; cat
/var/log/snmpd.log) | grep `date -d 'last month' | awk '{print $2}'`|
grep "snmpd\[" | petit --hash
Traceback (most recent call last):
File "/usr/bin/petit", line 348, in ?
get_options(sys.argv[1:])
File "/usr/bin/petit", line 135, in get_options
hash(filename)
File "/usr/bin/petit", line 164, in hash
log = Log(args)
File "/usr/share/petit/crunchtools.py", line 631, in __init__
self.append(Entry(line))
File "/usr/share/petit/crunchtools.py", line 90, in __init__
self.month = str(time.strptime(self.month,"%b")[1])
File "/usr/lib/python2.4/_strptime.py", line 291, in strptime
found = format_regex.match(data_string)
MemoryError

Please use labels and text to provide additional information.
Scott submitted this, I know what it is

Original issue reported on code.google.com by [email protected] on 28 Apr 2011 at 6:23

Add graphing features

Action Items
* Re-factor graphing code
* Add auto-scaling feature to automatically determine which graph to use
* Add new switch to specify exact amount of time to graph over allowing 
arbitrary granularity, limited by screen size.

Original issue reported on code.google.com by [email protected] on 2 Jun 2011 at 3:01

Petit crashes when parsing /var/log/syslog on Debian

Hi,

One of the Debian users has the following issue : 

---8<---
Hi,

I have a crash while using --hash with the /var/log/syslog file :

bureau:~# petit --hash /var/log/syslog
Traceback (most recent call last):
  File "/usr/bin/petit", line 348, in <module>
    get_options(sys.argv[1:])
  File "/usr/bin/petit", line 135, in get_options
    hash(filename)
  File "/usr/bin/petit", line 164, in hash
    log = Log(args)
  File "/usr/share/petit/crunchtools.py", line 631, in __init__
    self.append(Entry(line))
  File "/usr/share/petit/crunchtools.py", line 87, in __init__
    self.hour, self.minute, self.second = clocktime.split(":") 
ValueError: need more than 1 value to unpack

Bye,
Carl Chenet
---8<---

The original bug report is available at 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=589691

Bye,
Carl Chenet - Debian maintainer

Blog: http://carlchenet.wordpress.com
Identi.ca: http://identi.ca/carlchenet

Original issue reported on code.google.com by [email protected] on 20 Jul 2010 at 2:30

move to github

that would make forking, providing patches much easier :)

Original issue reported on code.google.com by [email protected] on 29 Jul 2014 at 4:43

Feature: Add real time graph

Features?
1. Real time graph (perhaps one minute/five minute increments)

Steps
1. Implement chunking of data input (open file and read chunks, process, etc.)

Original issue reported on code.google.com by [email protected] on 28 Oct 2010 at 7:09

Add the hacktoberfest topics to the repo so that PR's count towards it

Add hacktoberfest as topic to the repo to get contributions from participants

Debian/Ubuntu install problem

-------- Original Message --------
Subject:    error when installing petit
Date:   Thu, 12 May 2011 20:11:38 -0400
From:   Joshua Williams <[email protected]>
To:     [email protected]


hi Scott,

When downloading the .deb package for petit from this link:

http://crunchtools.com/wp-content/files/petit/petit-current.deb

I got the following error message:

dpkg: error processing petit-current.deb (--install):
 parse error, in file '/var/lib/dpkg/tmp.ci/control' near line 2 package 'petit':
 error in Version string '1.1.1_1': invalid character in version number
Errors were encountered while processing:
 petit-current.deb

when installing it on Debain Squeeze.  Just an FYI.

Thanks,

-Josh

Original issue reported on code.google.com by [email protected] on 2 Jun 2011 at 1:25

ImportError: No module named crunchtools.CrunchLog

What steps will reproduce the problem?
1. wget http://crunchtools.com/wp-content/files/petit/petit-current.deb; sudo 
dpkg -i petit-current.deb
2. petit --wordcount /var/log/messages

What is the expected output? What do you see instead?

Traceback (most recent call last):
  File "/usr/bin/petit", line 41, in <module>
    from crunchtools.CrunchLog import Log
ImportError: No module named crunchtools.CrunchLog


What version of the product are you using? On what operating system?
Petit Version: 1.1.0

Distributor ID: Ubuntu
Description:    Ubuntu 9.04
Release:    9.04
Codename:   jaunty

Please provide any additional information below.

Same symptom's on Ubuntu 10.10, crunchtools.CrunchLog module not packaged in 
.deb?

Original issue reported on code.google.com by [email protected] on 28 Feb 2011 at 11:18

Attachments:

file