roma-glushko/tango

Tango is a command-line tool for analyzing access logs 💃

Tango

Tool to get insights from the server access logs

---

Tango is a dependency-free command-line tool for analyzing access logs 💃

Currently, work on this project is in progress. However, a few pre-releases are ready available to use 🎉

Table of Contents

• Installation
• Usage
• Filters
• Report Commands
• Misc Commands
• Config File

Installation

macOS

Tango can be installed on macOS via Homebrew:

brew tap roma-glushko/tango
brew install roma-glushko/tango/tango

To upgrade, try to run:

brew upgrade tango

Linux

Tango is available on Linux via Snapcraft. This means that Tango can be installed on:

• Ubuntu
• Debian
• CentOS
• openSUSE
• Linux Mint
• Fedora
• Kubuntu
• elementary OS
• Arch Linux
• KDE Neon
• Manjaro

To upgrade, try to run:

snap refresh tango

Windows

Tango can be installed on Windows via Scoop:

scoop bucket add tango https://github.com/roma-glushko/scoop-tango.git
scoop install tango

To upgrade, try to run:

scoop update tango

Usage

List of available commands:

tango help

Tango Version:

tango -v

Global Options

Filters

// IP filters
tango --ip-filter "127.0.0.1" custom -l access-log.log -r custom.csv
tango --keep-ip-filter "8.8.8.8" custom -l access-log.log -r custom.csv

// URI filters
tango --uri-filter "/test-page" custom -l access-log.log -r custom.csv
tango --keep-uri-filter "/admin/" custom -l access-log.log -r custom.csv

// Time Frame filter
tango --keep-time-filter "2019-09-15 04:16:00 -0400" --keep-time-filter "2019-09-15 04:35:00 -0400" custom -l access-log.log -r custom.csv

// User Agent filters
tango --ua-filter "iPhone OS 12_3_1 like Mac OS X" custom -l access-log.log -r custom.csv
tango --keep-ua-filter "iPhone OS 12_3_1 like Mac OS X" custom -l access-log.log -r custom.csv

// Asset filter
tango --asset-filter "/pub/static/" --asset-filter "/pub/media/" custom -l access-log.log -r custom.csv

// System IP filter
tango --system-ips "127.0.0.1"  --system-ips "1.2.3.4" custom -l access-log.log -r custom.csv

Other

// Base URL info
tango --base-url "https://example.com/" custom -l access-log.log -r custom.csv

Report Commands

Custom Reports

tango --keep-uri-filter "/newsletter/subscriber/new/" custom -l access-log.log -r custom.csv

Use cases:

• generate a report with all requests from a certain IP
• generate a report with all requests to a certain URL

Geo Reports

tango geo -l access-log.log -r custom.csv

Geo Report uses MaxMind Geo lib to get Geo information. See Geo Lib command for more info.

Use cases:

• collects geo information about all IPs that requested the website
• get request distribution by IP with geo information
• see all IPs sorted by countries/continents/cities

Example of the report:

Example of the report

IP	Country	City	Continent	Sample Request	Browser Agent	Count of Requests
46.229.173.68	United States	Ashburn	North America	/robots.txt	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)	362
40.77.167.91	United States	Boydton	North America	/contact-us	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)	3
178.154.171.62	Russia		Europe	/	Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)	34

Browser Reports

tango browser -l access-log.log -r custom.csv

Use cases:

• check how many requests were sent by crawlers
• check what kind of browsers requested the website
• check bandwith that was transmitted to all kind of browsers
• check what crawlers requested the website

Example of the report

Category	Browser	Requests	Bandwith	Sample URL	User Agents
Crawlers	bingbot	629	28.8 MB	/black-bag-product	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Chrome	Chrome	131998	1.3 GB	/gears/bags?p=3	Mozilla/5.0 (Linux; Android 8.0.0; G8441) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.136 Mobile Safari/537.36 Mozilla/5.0 (Linux; Android 9; SM-G960F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.136 MobileSafari/537.36 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36

Request Reports

tango request -l access-log.log -r custom.csv

Use cases:

• check how many requests were sent to a certain URL
• check all URLs that were responded with 404/50X code
• find requests from security scanners (sort by response codes and look at 404/50X codes which were requested only 1 time)

Example of the report

Path	Requests	Response Code	Referer URLs
/media/catalog/product/black-bag.jpg	20	200	/black-bag
/admin/sales/order/view/order_id/1234	4	200	/admin/sales/order/index/order_id/123
/test321	1	404	/

Pace Reports [Experimental]

tango pace -l access-log.log -r custom.csv

Use cases:

• check which IPs and how many requests they made during a certain time frame
• check count of requests per minutes/hours

Example of the report

Hour Group	Minute Group	IP	Browser	Pace (req/min)	Pace (req/hour)
2020-02-10 04 h					35
	2020-02-10 04:06			15
		51.15.191.180	Barkrowler/0.9 (+https://babbar.tech/crawler)	10
		54.36.150.167	Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)	5
	2020-02-10 04:06			15
	2020-02-10 04:07			20
		66.249.76.89	Googlebot-Image/1.0	20
	2020-02-10 04:07			20
2020-02-10 04 h					35

Journey Reports [Experimental]

tango journey -l access-log.log -r custom.csv

Misc Commands

Geo Lib

// Install geo library to be able to generate geo reports
tango geo-lib

Tango uses the MaxMind GeoLite2-City database and stores it under:

• macOS - /Users/[username]/.tango/GeoLite2-City.mmdb

To be able to manage the Geo lib, you need to generate acceses under MaxMind Account page

Config File

Put the similar content to a .tango.yaml file under your working directory where you analyze logs:

"asset-filter":
  - "/pub/static/"
  - "/pub/media/"
  - "/media/"
  - "/static/"
"ip-filter":
  - "127.0.0.1"
"system-ips":
  # Fastly IPs
  - "23.235.32.0/20"
  - "43.249.72.0/22"
  - "103.244.50.0/24"
  - "103.245.222.0/23"
  - "103.245.224.0/24"
  - "104.156.80.0/20"
  - "151.101.0.0/16"
  - "157.52.64.0/18"
  - "167.82.0.0/17"
  - "167.82.128.0/20"
  - "167.82.160.0/20"
  - "167.82.224.0/20"
  - "172.111.64.0/18"
  - "185.31.16.0/22"
  - "199.27.72.0/21"
  - "199.232.0.0/16"

tango's Issues

[CI] Try to use Github Actions instead of Travis

Description

We need to evaluate if there are any benefits of using Github Actions instead of TravisCI.

GoRelease supports GA now: https://goreleaser.com/actions/

Expected behavior

• Run tests on every pull requests
• Run release actions on every new tag

[Logs] Multithread access log analyze

Description

It should be possible to analyze access logs in multiple threads to speed up the tool.

Benefits

Better user experience and less time to wait during huge log analysis

[Report] Add the referral report

Description

Add a report to collect information about referral websites from which visitors came to the website.

[Config] Multiple BaseUrls in the same config file

Description

Some projects can have multi-domain setup. Magento and Wordpress support that out-of-box. I will be handy if it's needed only one config file to process access logs for all project domains.

Expected behavior

It's possible to specify a couple of BaseUrls in the config files.

Benefits

Make possible to use one config for all project domains.

[Journey] Journey Report miss template while distributing Tango as a binary file

Preconditions

1. Any version of Tango
2. Any OS

Steps to reproduce

1. Run journey report

Expected result

1. Journey report is generated without errors

Actual results

There is an error on saving the report:

Error on reading journey report template: open template/journey-report.tmpl: no such file or directory

[CI] Travis Deployment is failed on Win OS

Travis Deployment seems to be failed on Win OS only. Deployment Log:

travis_fold:start:worker_info
�[0K�[33;1mWorker information�[0m
hostname: [e-mail]-production-2
version: v6.2.1 https://github.com/travis-ci/worker/tree/4e3246c044eb4915c2378ffacd0b3d3ed0136bba
instance: travis-job-7916f5f6-ffc3-4ad2-a694-e898577ea52f travis-ci-onion-1803-containers-1542208204-ad01dca (via amqp)
startup: 7.394070823s
travis_fold:end:worker_info
�[0KSecret environment variables are not obfuscated on Windows, please refer to our documentation: https://docs.travis-ci.com/user/best-practices-security�[0m
travis_time:start:0bd629e8
�[0Ktravis_time:end:0bd629e8:start=1568011450964546800,finish=1568011451005869300,duration=41322500,event=check_unsupported
�[0Ktravis_time:start:0049e318
�[0Ktravis_time:end:0049e318:start=1568011451051118100,finish=1568011451090383000,duration=39264900,event=apt_get_update
�[0Ktravis_time:start:068886f0
�[0K�[33;1mDisabling Windows Defender�[0m
$ powershell -Command Set-MpPreference -DisableArchiveScanning \$true
$ powershell -Command Set-MpPreference -DisableRealtimeMonitoring \$true
$ powershell -Command Set-MpPreference -DisableBehaviorMonitoring \$true
travis_time:end:068886f0:start=1568011451136593500,finish=1568011454032883700,duration=2896290200,event=disable_windows_defender
�[0KUpdating gimme
travis_time:start:001e260c
�[0K
travis_fold:start:git.checkout
�[0Ktravis_time:start:0d1a397c
�[0K$ git clone --depth=1 --branch=0.0.0-rc10 https://github.com/roma-glushko/tango.git roma-glushko/tango
Cloning into 'roma-glushko/tango'...
travis_time:end:0d1a397c:start=1568011483080530200,finish=1568011484843065800,duration=1762535600,event=checkout
�[0K$ cd roma-glushko/tango
$ git checkout -qf 0.0.0-rc10
travis_fold:end:git.checkout
�[0K
travis_time:end:0d1a397c:start=1568011483080530200,finish=1568011485091393600,duration=2010863400,event=checkout
�[0K$ travis_export_go 1.12.x github.com/roma-glushko/tango
travis_time:start:106f5b02
�[0K
�[33;1mSetting environment variables from .travis.yml�[0m
$ export PATH=/snap/bin:$PATH
$ export GO111MODULE=on

travis_time:end:106f5b02:start=1568011485185325200,finish=1568011485308841100,duration=123515900,event=env
�[0Ktravis_time:start:1a884014
�[0K$ travis_setup_go
Note: checking out 'e13541b555075c431e1215cac670f0f322540627'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

$ export GOPATH="/c/Users/travis/gopath"
$ export PATH="/c/Users/travis/gopath/bin:/c/Users/travis/.gimme/versions/go1.12.9.windows.amd64/bin:/snap/bin:/c/Users/travis/bin:/bin:/usr/bin:/c/tools/ruby25/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/c/Program Files/CMake/bin:/c/Program Files/Git/cmd:/c/Program Files/LLVM/bin:/c/Program Files/dotnet:/c/Users/travis/AppData/Local/Microsoft/WindowsApps:/c/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin"
$ export GO111MODULE="on"
go version go1.12.9 windows/amd64
travis_time:end:1a884014:start=1568011485359253600,finish=1568011518400310300,duration=33041056700,event=
�[0K$ gimme version
v1.5.3
$ go version
go version go1.12.9 windows/amd64
travis_fold:start:go.env
�[0K$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\travis\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:/Users/travis/gopath
set GOPROXY=
set GORACE=
set GOROOT=C:\Users\travis\.gimme\versions\go1.12.9.windows.amd64
set GOTMPDIR=
set GOTOOLDIR=C:\Users\travis\.gimme\versions\go1.12.9.windows.amd64\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\travis\gopath\src\github.com\roma-glushko\tango\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\travis\AppData\Local\Temp\go-build318432194=/tmp/go-build -gno-record-gcc-switches
travis_fold:end:go.env
�[0Ktravis_fold:start:install
�[0Ktravis_time:start:0fbdf0c9
�[0K$ travis_install_go_dependencies 1.12.x -v
go get: -t flag is a no-op when using modules
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding github.com/oschwald/geoip2-golang v1.3.0
go: finding github.com/urfave/cli v1.21.0
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: finding github.com/oschwald/maxminddb-golang v1.3.1
go: finding golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
Fetching https://gopkg.in/yaml.v2?go-get=1
go: finding github.com/BurntSushi/toml v0.3.1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
go: finding golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/fatih/color v1.7.0
go: finding github.com/mattn/go-colorable v0.1.2
go: finding github.com/mattn/go-runewidth v0.0.4
go: finding github.com/stretchr/objx v0.1.0
Fetching https://gopkg.in/check.v1?go-get=1
Parsing meta tags from https://gopkg.in/check.v1?go-get=1 (status code 200)
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at https://gopkg.in/check.v1?go-get=1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/mattn/go-isatty v0.0.8
go: finding golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223
go: downloading github.com/urfave/cli v1.21.0
go: downloading github.com/oschwald/geoip2-golang v1.3.0
go: downloading github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: extracting github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: downloading github.com/mattn/go-runewidth v0.0.4
go: extracting github.com/urfave/cli v1.21.0
go: extracting github.com/oschwald/geoip2-golang v1.3.0
go: downloading github.com/oschwald/maxminddb-golang v1.3.1
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading gopkg.in/yaml.v2 v2.2.2
go: extracting github.com/mattn/go-runewidth v0.0.4
go: extracting github.com/oschwald/maxminddb-golang v1.3.1
go: extracting github.com/BurntSushi/toml v0.3.1
go: downloading golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
go: extracting gopkg.in/yaml.v2 v2.2.2
go: extracting golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
tango/internal/usecase/config
github.com/urfave/cli
github.com/BurntSushi/toml
gopkg.in/yaml.v2
github.com/mattn/go-runewidth
github.com/cheggaaa/pb
tango/internal/domain/entity
github.com/urfave/cli/altsrc
tango/internal/usecase/mapper
tango/internal/usecase/processor
tango/internal/cli/factory
tango/internal/usecase
golang.org/x/sys/windows
tango/internal/infrastructure/reader
tango/internal/cli/component
tango/internal/usecase/filter
github.com/oschwald/maxminddb-golang
github.com/oschwald/geoip2-golang
tango/internal/usecase/report
tango/internal/infrastructure/writer
tango/internal/di
tango/internal/cli/command
tango/internal/cli
tango
travis_time:end:0fbdf0c9:start=1568011519825266700,finish=1568011549226993700,duration=29401727000,event=install
�[0Ktravis_fold:end:install
�[0Ktravis_fold:start:before_script
�[0Ktravis_time:start:0840730e
�[0K$ go get -t -v ./...
go get: -t flag is a no-op when using modules
travis_time:end:0840730e:start=1568011549281840200,finish=1568011550391542600,duration=1109702400,event=before_script
�[0Ktravis_fold:end:before_script
�[0Ktravis_time:start:01d1a41c
�[0K$ go build
travis_time:end:01d1a41c:start=1568011550442358800,finish=1568011553653541700,duration=3211182900,event=script
�[0K�[32;1mThe command "go build" exited with 0.�[0m
travis_time:start:2dd6fcf2
�[0K$ go test
?   	tango	[no test files]
travis_time:end:2dd6fcf2:start=1568011553704821000,finish=1568011567879185200,duration=14174364200,event=script
�[0K�[32;1mThe command "go test" exited with 0.�[0m

travis_fold:start:dpl_0
�[0Ktravis_time:start:2713b210
�[0K$ ruby -S gem install dpl
Successfully installed dpl-1.10.12
Parsing documentation for dpl-1.10.12
Installing ri documentation for dpl-1.10.12
Done installing documentation for dpl after 0 seconds
1 gem installed
travis_time:end:2713b210:start=1568011568571112900,finish=1568011571759897300,duration=3188784400,event=after_success
�[0Ktravis_fold:end:dpl_0
�[0Ktravis_time:start:24551688
�[0KSuccessfully installed dpl-script-1.10.12
Parsing documentation for dpl-script-1.10.12
Installing ri documentation for dpl-script-1.10.12
Done installing documentation for dpl-script after 0 seconds
1 gem installed
The system cannot find the path specified.
The system cannot find the path specified.
main: line 10: rev: command not found
main: line 12: rev: command not found
travis_fold:start:dpl.1
�[33mInstalling deploy dependencies�[0m

travis_fold:end:dpl.1
travis_fold:start:dpl.2
�[33mPreparing deploy�[0m

travis_fold:end:dpl.2
travis_fold:start:dpl.3
�[33mDeploying application�[0m

travis_fold:end:dpl.3
failed to deploy
Script failed with status 127

Error:

Parsing documentation for dpl-script-1.10.12
Installing ri documentation for dpl-script-1.10.12
Done installing documentation for dpl-script after 0 seconds
1 gem installed
The system cannot find the path specified.
The system cannot find the path specified.
main: line 10: rev: command not found
main: line 12: rev: command not found
travis_fold:start:dpl.1

[CI] Upload new snaps on Snapcraft

We need to add an ability to release Tango's builds on Snapcraft to let Ubuntu users use the tool.

References:

• https://goreleaser.com/snapcraft/
• https://snapcraft.io/first-snap/golang/macos/package

[UX] Report Time Format should be configurable

Description

Currently, Tango generates reports without the ability to specify output time format. I'm aware at least about 2 time formats that would be nice to have:

• EU format: Sep 11th, 2019 (month-first)
• US format: 11th September 2019 (day-first)

Also, it would be nice to have the ability to specify a custom time format in case it's needed.

Expected behavior

There is a way to specify two defined templates for the time (EU/US formats) or pass completely custom format based on Golang possibilities:

• https://programming.guide/go/format-parse-string-time-date-example.html

Benefits

• reduce the frustration of users that would read reports

[UX] Generate Report Filename and make it an optional parameter

Description

We need to reduce time to run/options to configure number to the minimum. Right now, report-file option is a required argument and user needs to type it manually. We can make it optional and generate the report file name based on some criteria.

Expected behavior

It's not required to type report-file option manually. Tango should be able to generate meaningful report file name based on input information:

• log filename
• filter to apply
• report type
• (maybe) add a timestamp

Benefits

• less frustration to work with Tango
• easier and quicker to generate reports

[Report] Add query param report

Description

Add a query param report to collect all unique params which happen in the access logs.

Benefits

This may help to setup and keep healthy website FPC caching engine (like Varnish)

[Anonymizer] Add ability to anonymize data in reports

Description

Access log anonymizing can be a super helpful feature to have in Tango. Usually, there are needs to pass/show access logs to 3rd-parties, but not always it's possible to show customer/visitor data

Expected behavior

A new CLI command that would process the data according to passed anonymizer params and would save the access log file in the same raw format (not CSV as Tango normally does).

Benefits

• zero effort to make access log data anonymized
• possibility to share access log data with 3rd-parties without fear to have legal issues

Additional information

The following data seems to be useful to anonymize:

• visitor IPs
• access URIs
• user agents
• timezone

It may be useful to add the ability to replace some specific lines in the access logs (like googlable vendor names).

Also, anonymized of data should show visitor actions as close as possible to the original file. This means that would be great to keep for IP and URI, User-Agent of the visitor the same along with the whole access log.

Example1. If we anonymize IP 1.1.1.1 to 250.2.3.50, IP 250.2.3.50 should be used everywhere where we need to close information about IP 1.1.1.1.

Example2. If we want to hide information about URI "/automobile-windows" and anonymize it to "/category100", then we need to use "/category100" everywhere where it comes to the original URI.

Such a strategy helps to keep relations in the access log and pass it to other parties.

[Report] Add a new security report

Description

It's super useful to be able to track and report anomaly URLs that may be a result of the following attacks:

• XSS
• SQL Injections
• The rest of OWASP 10 vulnerabilities

Expected behavior

There is a new report type security which can automatically detect and report anomaly and malicious requests.

Benefits

Security report type should allow tracking manual hacking attempts without reviewing the whole logfile as well as help to track automated security scannings.

Additional information

We have a couple of real-world samples of unauthorized security scannings.

[Config] Tango requires creation of .tango.yaml file

Description
Tango requires the creation of .tango.yaml file to be able to generate reports.

Expected Result
This should be optional and only for those who want to optimize and don't remember useful repeatable configs.

Actual Result
Tango requires .tango.yaml to be able to execute commands even if the file is empty

[Management] Improve Changelog Policy

Description

It's nice to have a clean and tidy list of changes released during each new version. The current changelog is not informative and helpful.

Benefits

It will be less frustrating to track bug fixes and improvements that come with new versions.

Additional information

Gorelease supports already changelog generating, but we need to create a flow to use it in a right way

[Bug] Geo Report can throw an error on saving

There can be an issue during Geo report generation.

Preconditions

• The latest version of Tango
• Any OS (reproduced on macOS)

Steps to reproduce

• Run the latest version of Tango like this:

tango geo -l 2019-10-13-transfer.log -r 10-13-geo-report.csv

Expected result

Geo report is generated in any case and saved correctly to the file system.

Actual results

There is an error in the console:

💃 Tango is on the scene!
💃 started to generate a geo report...
💃 reading access logs...
 433.17 MiB / 433.97 MiB [============================================================]  99.81% 1m7s
💃 saving the geo report...
2019/10/14 12:03:05 ipAddress passed to Lookup cannot be nil

No reports are saved.

[Config] Main Config File and Config files for specific cases

Description

Currently, we have a possibility to persist most of the arguments in the config file. This kind of important feature as no one wants to remember/retype long params for CLI commands.

As access log analysis is a on-going task and people come back to this task after some time, there is a different reasons and purpose of the analysis.
Example1. Today some performance degradations happened to the website. There is a need to analyze incoming traffic
Example2. Tomorrow, there is a security accident and malicious activity was detected. Need to analyze access logs from another angle.
Example3. Other days, some customer reports some issues with the payment gateways during morning time. Again there is a need to cover log analysis from a different side.

Example4. Now let's assume that case 1, 2, 3 happens relatively frequently and on a couple of projects.

There is an idea to have a solution to cover such workflow which includes less copy-pasting and let's focus on the problem.

Expected behavior

Tango supports the concept of main config file and case-specific config files.

Main config file is a file where all main project-specific information (like base-url, asset patterns, system-ips etc)

Case-specific config file is a file which can override the main config file of the same project and contains configs which are needed to work on the current case/task.

Benefits

• order access log checking workflow

[Docs] Document usage of the current functionality

The following parts of the Tango CLI should be documented:

• usage of filters
• report types
• examples of report outputs
• usage of .tango.yaml config file

[Logs] Support reading access log data from the stream

Description

It should be possible to read access logs not only from files but also from streams like

zcat access.log.gz | tango custom -r custom.csv

Benefits

It would be possible to analyze compressed logs on the fly without unarchiving them.

Installation: Calling bottle :unneeded is deprecated! There is no replacement

During installation Homebrew shows the following warning and in my case eventually fails:

Running `brew update --preinstall`...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
bash-language-server                     mold
fypp                                     yaml-language-server
grafana-agent                            zsh-fast-syntax-highlighting
libgrape-lite
==> Updated Formulae
Updated 71 formulae.
==> New Casks
fastmarks                  kdrive                     only-switch
==> Updated Casks
Updated 79 casks.

Warning: Calling bottle :unneeded is deprecated! There is no replacement.
Please report this issue to the roma-glushko/tango tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/roma-glushko/homebrew-tango/Formula/tango.rb:6

==> Downloading https://github.com/roma-glushko/tango/releases/download/1.1.0/ta
==> Downloading from https://objects.githubusercontent.com/github-production-rel
######################################################################## 100.0%
==> Installing tango from roma-glushko/tango
Warning: Calling bottle :unneeded is deprecated! There is no replacement.
Please report this issue to the roma-glushko/tango tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/roma-glushko/homebrew-tango/Formula/tango.rb:6

Error: Your Command Line Tools are too outdated.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/all/.
You should download the Command Line Tools for Xcode 13.1.

Error: An exception occurred within a child process:
  SystemExit: exit

[Logs] Support multiple logs parsing at the same time

Description

This may be helpful to support analyzing a group of logs at the same time to include their data to the same report.

Benefits

Streamline analyzing of big amount of access logs

[Report] Add support of Elastic Stack for reporting

Description

This may be helpful at scale if Tango supports Elastic Stack to save reports and render them in Kibana dashboards. This integration should be investigated to understand how Tango can be helpful in that case and clarify acceptance criteria.

Expected behavior

TBU

Benefits

• add an ability to use Tango with Elastic Stack which is a super popular framework for website monitoring
• expand Tango's use cases for scalable projects

Additional information

TBU

• https://www.linode.com/docs/databases/elasticsearch/visualize-apache-web-server-logs-using-elastic-stack-on-debian-8/
• https://www.elastic.co/products/logstash

Permission problems with the snap

Preconditions

10:46:21 ~ -1- $ tango -v
Tango version 1.1.1 (21f4d3dbc8836d4200b0fefca28f982847620c9a)
11:13:31 ~ -1- $

Steps to reproduce

$ tango journey -l /tmp/access.log -r /tmp/access.log.journey

or

$ tango journey -l /mnt/data/access.log -r /mnt/data/access.log.journey

Expected result

Should run

Actual results

either permission denied or file not found.

10:44:17 ~ -1- $ tango journey -l /tmp/roundcube.access -r TMP/roundcube.access.journey
💃 Tango is on the scene!
💃 started to generate a visitor's journey report...
💃 reading access logs...
2023/03/22 10:46:21 open /tmp/roundcube.access: no such file or directory
11:13:31 ~ -1- $ ls /tmp/roundcube.access
-rwxr-xr-x 1 ychaouche ychaouche 3.4M Mar 22 10:22 /tmp/roundcube.access
11:17:42 ~ -1- $


09:35:27 ~ -1- $ tango journey -l DATA/roundcube.access -r DATA/roundcube.access.journey
💃 Tango is on the scene!
💃 started to generate a visitor's journey report...
💃 reading access logs...
2023/03/22 09:35:28 open DATA/roundcube.access: permission denied
09:35:28 ~ -1- $ ls DATA/roundcube.access -r
-rwxrwxrwx 1 root root 3.4M Mar 22 09:26 DATA/roundcube.access
09:35:34 ~ -1- $

commentary

The problem seems to come from apparmor.
Here's what's in my syslog

Mar 22 09:35:11 ychaouche-PC kernel: [ 2484.149862] audit: type=1400 audit(1679474111.364:138): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=8910 comm="tango" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Mar 22 09:35:28 ychaouche-PC kernel: [ 2501.473736] audit: type=1400 audit(1679474128.688:139): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=8958 comm="tango" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Mar 22 09:37:00 ychaouche-PC dbus[1082]: [system] Activating service name='org.freedesktop.systemd1' (using servicehelper)
Mar 22 09:37:00 ychaouche-PC dbus[1082]: [system] Successfully activated service 'org.freedesktop.systemd1'
Mar 22 09:38:15 ychaouche-PC kernel: [ 2668.112283] audit: type=1400 audit(1679474295.329:140): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=9044 comm="tango" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Mar 22 09:42:39 ychaouche-PC kernel: [ 2932.648803] audit: type=1400 audit(1679474559.875:141): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=9158 comm="tango" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Mar 22 09:42:59 ychaouche-PC kernel: [ 2951.837491] audit: type=1400 audit(1679474579.064:142): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=9196 comm="tango" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Mar 22 09:44:07 ychaouche-PC kernel: [ 3019.986808] audit: type=1400 audit(1679474647.216:143): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=9528 comm="tango" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Mar 22 09:44:24 ychaouche-PC kernel: [ 3036.962988] audit: type=1400 audit(1679474664.192:144): apparmor="DENIED" operation="open" profile="snap.tango.tango" name="/mnt/partage_local/DATA/roundcube.access" pid=9566 comm="tango" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Looking at the apparmor profile,
(/var/lib/snapd/apparmor/profiles/snap.tango.tango)
I could find that there's a read acces to /tmp/,
but also read a comment about /tmp/ being created for each specific snap,
so I'm not sure which is which

  # The ubuntu-core-launcher creates an app-specific private restricted /tmp
  # and will fail to launch the app if something goes wrong. As such, we can
  # simply allow full access to /tmp.
  /tmp/   r,
  /tmp/** mrwlkix,

[Docs] The gif in the `README.md` isn't very helpfull

Description

The first gif in the readme directly caught my attention. I've been staring at it for 30 sec, to wait for the "demo" to start. But instead, it just starts over. An animation of a progress bar isn't really helpful.

I think it would be more awesome if it could show the output. Provide a glimpse of what this project provides. What happens once the logs have been "read"?

Additional information

[UX] Introduce a debug mode

Description

It will be helpful if Tango supports a debug mode which will verbose all useful information to a debug file.

Expected behavior

Debug mode will be disabled by default and it will be possible to enable it via an additional option. A file will contain all information about how processing went for given logs/arguments.

Benefits

Debug mode will bring more clarity on errors and bugs during working with Tango binary.

[Geo] MaxMind changed a way to access Geo DB

Steps to reproduce

1. Geo Lib installing test is executed on the TravisCI:
   https://travis-ci.org/roma-glushko/tango/builds/645156066

Expected result

All tests are executed without errors.

Actual results

There is an error during execution

This post might be helpful:

• https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
• https://github.com/maxmind/geoipupdate

[Info] Incorrect Usage: flag provided but not defined: -filter-name

Preconditions

1. Any version of Tango

Steps to reproduce

1. Run Tango CLI:

tango custom -l access.log -r custom.csv --keep-uri-filter "/newsletter/subscriber/new/"

Expected result

Tango generates a needed report.

Actual results

There is an error during execution.

How to Fix

Global filters should be specified before the report type. So the command from the steps to reproduce should look like this:

tango --keep-uri-filter "/newsletter/subscriber/new/" custom -l access.log -r custom.csv

[CI] Pass Tango CLI app version from git tag

Description

Right now it's super easy to forget to adjust version in Tango CLI application:

tango/internal/cli/tango-cli.go

Line 186 in 8cc5d30

cliApp.Version = "1.0.3 [beta]"

Expected behavior

The version is passed via CI and always equal to the version we have specified in git tag

Benefits

Users are sure which version of Tango they downloaded, installed and use

[Geo] Automatically install MaxMind Geo library

We need to make usage of Tango geo reports zero configured and don't require people to install and use MaxMind Geo library.

Reference:

• https://dev.maxmind.com/geoip/geoip2/geolite2/ - list of zipped libraries

[Bug] Request report is broken

Preconditions

Log File - transfer-ssl-2019-11-21.log

Steps to reproduce

Run Report report:

tango request -l transfer-ssl-2019-11-21.log -r 11-21.request.csv

Expected result

Request Report is generated successfully.

Actual results

Error is thrown, no report is generated:

💃 Tango is on the scene!
💃 started to generate a request report...
💃 reading access logs...
 71.73 MiB / 71.91 MiB [===============================================================]  99.74% 17s
💃 saving the request report...
panic: parse %.: invalid URL escape "%."

goroutine 1 [running]:
tango/internal/usecase/report.(*RequestReportUsecase).GenerateReport(0xc0001437d0, 0x7ffeefbff600, 0x11, 0xc00b42c000, 0x233a9, 0x2771c)
	/Users/glushko/Documents/Projects/etc/tango/internal/usecase/report/request-report-usecase.go:65 +0x655
tango/internal/cli/command.RequestReportCommand(0xc0000d0b00, 0x0, 0xc0000e2b60)
	/Users/glushko/Documents/Projects/etc/tango/internal/cli/command/request-report-command.go:26 +0x3c0
github.com/urfave/cli.HandleAction(0x14469e0, 0x151dc80, 0xc0000d0b00, 0xc0000d0b00, 0x0)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/app.go:514 +0xbe
github.com/urfave/cli.Command.Run(0x14e7643, 0x7, 0x0, 0x0, 0xc0000ada60, 0x1, 0x1, 0x14f7118, 0x28, 0x0, ...)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/command.go:171 +0x4cd
github.com/urfave/cli.(*App).Run(0xc00011a1c0, 0xc0000b8120, 0x6, 0x6, 0x0, 0x0)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/app.go:265 +0x72f
tango/internal/cli.(*TangoCli).Run(0xc000143f48, 0xc0000b8120, 0x6, 0x6)
	/Users/glushko/Documents/Projects/etc/tango/internal/cli/tango-cli.go:208 +0x50
main.main()
	/Users/glushko/Documents/Projects/etc/tango/main.go:11 +0x5d

[Logs] Support huge access log analysis

Description

Currently, Tango is only working with memory during log analysis. On the large scale projects, it may be an issue to analyze access logs as they Gb-sized.

Expected behavior

It's possible to use disk space to store information during access log analysis.

Benefits

Better support of large-scale projects with a demand to analyze huge access logs

Additional information

B+Tree can be used here.

[Log] Nginx Log has spaces in the end of the log record

Preconditions

1. Nginx Log in the combined format like this:

servername.com 1.1.1.1 - - [19/Nov/2019:02:55:46 +0100] "GET /server-status HTTP/1.1" 200 216081 "-" "libwww-perl/6.15" 
10.10.10.10 2.2.2.2 - - [19/Nov/2019:02:56:46 +0100] "GET /administrator HTTP/1.1" 301 245 "-" "Mozilla/5.0 (Linux; rv:55.0) Gecko/20100101 Firefox/55.0" 
10.10.10.10 3.3.3.3 - - [19/Nov/2019:02:56:46 +0100] "GET /administrator/ HTTP/1.1" 200 3314 "-" "Mozilla/5.0 (Linux; rv:55.0) Gecko/20100101 Firefox/55.0" 
servername.com 4.4.4.4 - - [19/Nov/2019:03:00:46 +0100] "GET /server-status HTTP/1.1" 200 216194 "-" "libwww-perl/6.15" 

Steps to reproduce

1. Run any report with any params on the log

Expected result

Log file is parsed and processed

Actual results

Nothing is parsed, all information is empty. However, regex pattern is still actual

[Coverage] Cover Recently Closed Bugs by Integration Tests

Description

We have recently fixed the following issues:

• #18: [Log] Nginx Log has spaces at the end of the log record
• #17: [Bug] Geo Report can throw an error on saving
• #28: [Bug] Request report is broken

We need to create integration tests to cover these cases.

Expected behavior

New integration tests are added and they cover fixed edge cases.

Benefits

Automatically check and prevent founded edge cases.

[Browser] Add an ability to specify browser classification from the config file

Description

It's good to have a way to specify a custom browser classification from the config file.

Expected behavior

It's possible to specify that SuperCustomChCrawler2.0 is a part of the Crawler group.

Benefits

Let users better control their browser reports