simagix/hatchet

MongoDB JSON Log Analyzer

Hatchet - MongoDB JSON Log Analyzer and Viewer

Hatchet is a powerful and sophisticated logs analyzer and viewer specifically designed for MongoDB JSON logs. It provides advanced features for logs processing, aggregation and storage of the processed data. To make the data accessible and usable for its users, Hatchet utilizes an embedded SQLite3 database. This database allows for the storage of processed and aggregated data and makes it possible to offer RESTful APIs and a web interface to users.

The web interface of Hatchet is highly interactive and user-friendly, providing a seamless experience for searching logs and navigating through reports and charts. The intuitive design and easy-to-use interface makes it simple for users to find what they need, when they need it. Additionally, with the embedded database, Hatchet provides fast access to data and a high level of performance, making it the ideal solution for logs analysis and management. Further design details can be found at .

Change Log

• v0.5.0, August, 2023

Build

Clone and run the build.sh script; gcc is required to support CGO.

git clone --depth 1 https://github.com/simagix/hatchet.git
cd hatchet ; ./build.sh

An executable hatchet is output to the directory dist/. Note that the script also works and tested on Windows x64 using MingGW and Git Bash.

Quick Start

Use the command below to process a log file, mongod.log.gz and start a web server listening to port 3721. The default database is SQLite3.

./dist/hatchet -web logs/sample-mongod.log.gz

Load a file within a defined time range:

./dist/hatchet -web -from "2023-09-23T20:25:00" -to "2023-09-23T20:26:00" logs/sample-mongod.log.gz

Load multiple files and process them individually:

./dist/hatchet -web rs1/mongod.log rs2/mongod.log rs3/mongod.log

Load multiple files and process them collectively:

./dist/hatchet -web -merge rs1/mongod.log rs2/mongod.log rs3/mongod.log

Use the URL http://localhost:3721/ in a browser to view reports and charts. Alternatively, you can use the in-memory mode without persisting data, for example:

./dist/hatchet -url in-memory logs/sample-mongod.log.gz

if you choose to view in the legacy format without a browser, use the command below:

./dist/hatchet -legacy logs/sample-mongod.log.gz

For additional usages and integration details, see developer's guide.

A Smart Log Analyzer

How smart Hatchet is? A picture is worth a thousand words.

Other Usages

Other than its ability to read from files, Hatchet offers additional functionality that includes reading from S3 and web servers, as well as MongoDB Atlas. This means that users can use Hatchet to conveniently access and download data from these sources, providing a more versatile and efficient data analysis experience.

Web Servers

The tool supports reading from web servers using both the http:// and https:// protocols. The -user flag is optional when using basic authentication.

hatchet [-user {username}:{password}] https://{hostname}/{log name}

Atlas

To download logs directly from MongoDB Atlas, you will need to use the -user and -digest flags and provide the necessary information for both. These flags are used to authenticate and authorize your access to the database.

hatchet -user {pub key}:{private key} -digest https://cloud.mongodb.com/api/atlas/v1.0/groups/{group ID}/clusters/{hostname}/logs/mongodb.gz

AWS S3

Hatchet has the ability to download files from AWS S3. When downloading files, Hatchet will automatically retrieve the Region and Credentials information from the configuration files located at ${HOME}/.aws. This means that there's no need to provide this information manually each time you download files from AWS S3 using Hatchet.

hatchet -s3 [--endpoint-url {test endpoint}] {bucket}/{key name}

Logs Obfuscation

Use Hatchet to obfuscate logs. It automatically obfuscates the values of the matched patterns under the "attr" field, such as SSN, credit card numbers, phone numbers, email addresses, IP addresses, FQDNs, port numbers, namespaces, and other numbers. Note that, for example, replacing "host.example.com" with "rose.taipei.com" in the log file will consistently replace all other occurrences of "host.example.com" with "rose.taipei.com". To obfuscate logs and redirect them to a file, use the following syntax:

hatchet -obfuscate {log file} > {output file}

License

Apache-2.0 License

hatchet's Issues

cannot find package "github.com/simagix/hatchet

Hi Team, We are getting below Error

[root@RCPPPMPOSAPPN3 hatchet]# ./build.sh
main/hatchet.go:9:2: cannot find package "github.com/simagix/hatchet" in any of:
/usr/lib/golang/src/github.com/simagix/hatchet (from $GOROOT)
/root/go/src/github.com/simagix/hatchet (from $GOPATH)

root@RCPPPMPOSAPPN3 hatchet]# go version
go version go1.9.4 linux/amd64

unhandled data type primitive.Decimal128, 0 error on parsing mongod log

There's another unhandled data type I discovered in a new log:

$ hatchet -web mongodb.log
2023/03/01 12:42:30 processing mongodb.log
2023/03/01 12:42:30 hatchet name is ssssad_0d2ef4
2023/03/01 12:42:30 fast counting mongodb.log ...
...
...
2023/03/01 12:42:30 unhandled data type primitive.Decimal128, 0
2023/03/01 12:42:30 unhandled data type primitive.Decimal128, 0
...

I isolated the first two lines that were being processed which threw this error (attached), however unilke the data type error seen on minKey, this "primitive" isn't explicitly listed as "Decimal128" in the log lines, e.g.:

[{"t":{"$date":"2023-01-30T04:41:31.770+00:00"},"s":"I",  "c":"WRITE",    "id":51803,   "ctx":"conn2780","msg":"Slow query","attr":{"type":"update","ns":"STFPIMSQS.PurchaseOrder","command":{"q":{"_id":{"$oid":"63d32d03e6b07c1af9168e5b"}},"u":{"$set":{"_id":{"$oid":"63d32d03e6b07c1af9168e5b"},"orderNumber":"8230628","orderDate":"2020-06-16T00:00","orderStatusCode":"I","orderStatusDescription":"RECEIVED","mpnOrdered":"C419406-17","mpnReceived":"C419406-17","mepn":"78-5010-3-0104","isAOG":false,"stationCode":"D00","shipToStation":"1","supplierNumber":"131997","supplierSequence":"1","buyerCode":"555","expediteIndicator":"","isSpec2000S1Booked":true,"exceptionQuantity":0,"newPartIndicator":false,"hasBlanketOrder":false,"orderQuantity":6,"receivedQuantity":6,"reopenQuantity":0,"underReceiptQty":0,"queueBypassIndicator":"B","planner":"100599","receipts":2,"receiptDate":"2020-07-07T00:00","requiredDate":"2020-07-12T00:00","aircraftNumber":"0","scheduledShipQuantity":0,"requistionNumber":"0","receivedDate":"2020-11-13T00:00","minimumOrderAmount":0,"averageLeadTime":"0","carrier":"","waybill":"FEDX              ","packingSlip":"1537778","averageCost":{"$numberDecimal":"0"},"metadata":{"id":{"$oid":"63d32d03e6b07c1af9168e5b"},"tms":{"$date":"2023-01-30T04:41:24.722Z"},"user":"po-data-processor","sourceTms":{"$date":"2023-01-30T04:41:24.722Z"}}}},"multi":false,"upsert":true},"planSummary":"IDHACK","keysExamined":0,"docsExamined":0,"nMatched":0,"nModified":0,"nUpserted":1,"keysInserted":2,"numYields":0,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":8676}},"FeatureCompatibilityVersion":{"acquireCount":{"w":8676}},"ReplicationStateTransition":{"acquireCount":{"w":8677}},"Global":{"acquireCount":{"w":8676}},"Database":{"acquireCount":{"w":8676}},"Collection":{"acquireCount":{"w":8676}},"Mutex":{"acquireCount":{"r":8676}}},"flowControl":{"acquireCount":4338,"timeAcquiringMicros":5145},"readConcern":{"level":"local","provenance":"implicitDefault"},"storage":{"data":{"bytesRead":227014,"timeReadingMicros":318}},"remote":"192.168.254.9:2436","durationMillis":363}}

Assuming it's similar to Go's Decimal128.BigInt() type, but can't discern from above lines which field is the culprit.

Stats by appName

Is it possible to add Stats by appName like Stats by IPs?

multiple applications can run on same server, appName can be a real differentiator here rather than IP.

Search queries report total records

In the search screen of the log results, it would be great if you could perform a search and, in addition to seeing a set of paginated rows returned, get a total count of how many total records match the search terms.

Intermittent 'log format not supported' issues

Hi, thank you for your work on Keyhole and Hatchet, both tools have proven incredibly valuable for me when identify areas of improvement within Mongo.

I have setup a Github actions pipeline to download server logs periodically using the atlas cli:

atlas logs download <host> mongodb.gz

After downloading the log, I attempt to generate a Hatchet report, and then run the web server to collect data from the api:

# Generate the hatchet, run report build to completion.
hatchet /tmp/mongodb.log.gz
# Once done run the API server.
hatchet -web &

However, every so often the hatchet fails to process a log file at random with the following error:

# 'broken' is the name of the file that I pulled from Github as an artifact
2023/03/28 17:04:00 Downloads/hatchet v0.3.4-20230327
2023/03/28 17:04:00 processing broken-00-01-mongodb.log.gz
2023/03/28 17:04:00 hatchet name is broken_00_01_mongodb_169960
2023/03/28 17:04:00 log format not supported

I understand that this is likely an issue with the file generation, or contents of the file itself, meaning that there is a chance that the issue lies in the Atlas CLI.
However, I was wondering if there was any way to get more specific information about this error, as running in verbose mode did not yield any extra information.

Any help appreciated if you happen to have ran into this issue before.

Thanks

Versions

$ hatchet -version
Downloads/hatchet v0.3.4-20230327

$ go version
go version go1.20.2 linux/amd64

$ atlas process list
ID                                                 REPLICA SET NAME       SHARD NAME   VERSION
atlas-xxx-i-00-00.yyy.mongodb.net:27017       atlas-xxx-shard-0                4.4.19
atlas-xxx-shard-00-00.yyy.mongodb.net:27017   atlas-xxx-shard-0                4.4.19
atlas-xxx-shard-00-01.yyy.mongodb.net:27017   atlas-xxx-shard-0                4.4.19
atlas-xxx-shard-00-02.yyy.mongodb.net:27017   atlas-xxx-shard-0                4.4.19

Stats by IPs - Closed connection info not available

In the front page can you add closed connection next to accepted connection column under Stats by IPs?

panic: interface conversion: interface {} is string, not primitive.D

MongoDB server version: 4.4.14

[root@centos test]# docker images | grep hatchet
simagix/hatchet             latest          bb124fd475a7   5 weeks ago    33.8MB

[root@centos test]# docker run --rm --network=host -v /tmp/test:/home/simagix   simagix/hatchet /hatchet -web /home/simagix/mongod_error.log
2023/06/08 08:49:00 simagix/hatchet v0.4.2-20230502
2023/06/08 08:49:00 using database ./data/hatchet.db
2023/06/08 08:49:00 processing /home/simagix/mongod_error.log
2023/06/08 08:49:00 hatchet name is mongod_error_04522b
2023/06/08 08:49:00 fast counting /home/simagix/mongod_error.log ...
2023/06/08 08:49:00 counted 4 lines
2023/06/08 08:49:00 creating hatchet mongod_error_04522b
panic: interface conversion: interface {} is string, not primitive.D

goroutine 1 [running]:
github.com/simagix/hatchet.AddLegacyString(0xc0000fc960)
        /github.com/simagix/hatchet/legacy.go:114 +0x17ba
github.com/simagix/hatchet.(*Logv2).Analyze(0xc000319950, {0x7ffc0342bf6b, 0x1e})
        /github.com/simagix/hatchet/logv2.go:235 +0xb72
github.com/simagix/hatchet.Run({0xc000098760, 0x1f})
        /github.com/simagix/hatchet/hatchet.go:115 +0x10cc
main.main()
        /github.com/simagix/hatchet/main/hatchet.go:20 +0x10e

Hi, each line in the following mongod_error.log will get the above error when parsing the log. Can you help to find out the reason and fix it ? Thanks a lot.

mongod_error.log

cannot find package "github.com/brianvoe/gofakeit/v6"

[root@xxxxxxx hatchet]# ./build.sh
../go/src/github.com/simagix/hatchet/bios.go:19:2: cannot find package "github.com/brianvoe/gofakeit/v6" in any of:
/usr/lib/golang/src/github.com/brianvoe/gofakeit/v6 (from $GOROOT)
/root/go/src/github.com/brianvoe/gofakeit/v6 (from $GOPATH)
../go/src/github.com/simagix/hatchet/audit_template.go:16:2: cannot find package "golang.org/x/text/language" in any of:
/usr/lib/golang/src/golang.org/x/text/language (from $GOROOT)
/root/go/src/golang.org/x/text/language (from $GOPATH)
../go/src/github.com/simagix/hatchet/audit_template.go:17:2: cannot find package "golang.org/x/text/message" in any of:
/usr/lib/golang/src/golang.org/x/text/message (from $GOROOT)
/root/go/src/golang.org/x/text/message (from $GOPATH)
[root@xxxxxxx hatchet]#

near "s": syntax error

Facing below error after scanning the whole log file downloaded from Atlas.

Error - 1 -
2023/09/28 23:58:31 mtools/hatchet v0.5.1-20230926 2023/09/28 23:58:31 using database ./data/hatchet.db 2023/09/28 23:58:31 processing ./../crm-standard-logs.gz 2023/09/28 23:58:31 hatchet name is crm_standard_logs_d80230 2023/09/28 23:58:31 fast counting ./../crm-standard-logs.gz ... 2023/09/28 23:58:32 counted 987200 lines 2023/09/28 23:58:32 creating hatchet crm_standard_logs_d80230 2023/09/28 23:58:32 CREATE TABLE IF NOT EXISTS hatchet ( name text not null primary key, version text, module text, arch text, os text, start text, end text ); 2023/09/28 23:58:32 DROP TABLE IF EXISTS crm_standard_logs_d80230; CREATE TABLE crm_standard_logs_d80230 ( id integer not null primary key, date text, severity text, component text, context text, msg text, plan text, type text, ns text, message text collate nocase, op text, filter text, _index text, milli integer, reslen integer ); 2023/09/28 23:58:32 DROP TABLE IF EXISTS crm_standard_logs_d80230_audit; CREATE TABLE crm_standard_logs_d80230_audit ( type text, name text, value integer ); 2023/09/28 23:58:32 DROP TABLE IF EXISTS crm_standard_logs_d80230_clients; CREATE TABLE crm_standard_logs_d80230_clients ( id integer not null primary key, ip text, port text, conns integer, accepted integer, ended integer, context text ); 2023/09/28 23:58:32 DROP TABLE IF EXISTS crm_standard_logs_d80230_drivers; CREATE TABLE crm_standard_logs_d80230_drivers ( id integer not null primary key, ip text, driver text, version text ); 2023/09/28 23:58:32 DROP TABLE IF EXISTS crm_standard_logs_d80230_ops; CREATE TABLE crm_standard_logs_d80230_ops ( op text, count integer, avg_ms numeric, max_ms integer, total_ms integer, ns text, _index text, reslen integer, filter text ); 2023/09/28 23:58:32 using 9 threads 2023/09/28 23:53:45 near "s": syntax error