trackit/trackit

Trackit helps you understand and improve your use of AWS

TrackIt is a tool to optimize your AWS cloud usage and spending.

Features

• Easy account setup

• AWS Cost Breakdown

• AWS Tags overview

• Events alerts

How to use

With Docker Compose

You can start using TrackIt by using the docker-compose.yml template available in this repository. It will pull Docker images from Docker Registry.

$> docker-compose up -d

You can also build locally the needed Docker images by using the docker-compose.yml file available in docker/ folder.

$> docker-compose up -d -f docker/docker-compose.yml

Manually

0. Be sure all requirements below are met

• Docker
• Docker Compose
• AWS CLI and configure your credentials
• Golang
• Govender

1. Clone this repository

$> mkdir -p $HOME/go/src/github.com/trackit
$> cd $HOME/go/src/github.com/trackit
$> git clone https://github.com/trackit/trackit
$> cd trackit

2. Check out dependencies

$> govendor sync -v

3. Start TrackIt

$> ./start.sh

Note: On most operating systems, you will need to increase the mmap limit to allow elasticsearch to run properly:

$> sudo sysctl -w vm.max_map_count=262144

4. Now you can use TrackIt

TrackIt API is now listening on localhost:8080

Web UI

A Web UI made with React is available here: TrackIt Client

API documentation

The API exposes its own documentation on the GET /docs route, in JSON format. Also, the documentation for each route can be retrieved by an OPTIONS request. We are working on an actual viewer for this.

Recommendation plugins

Trackit uses a plugin system to easily implement new recommendation checks. Information on how to write plugins is available in a README in the plugins directory.

trackit's Issues

Add more resources report (for Tagbot)

Add more report generation of some resources that are often used.

External IDs on subaccounts cannot be changed and are empty by default

When TrackIt add subaccounts, it leaves external ids empty. The issue is that we would like to, at least, get an external id for each subaccount. The best would be to have the possibility to set an external id for each subaccount.

Elasticsearch index.max_inner_result_window parameter default value to 100

The elasticsearch is actually creating indexes with a parameter index.max_inner_result_window to 100 by default. We need to increase this limit to an integer max value on all indexes and on indexes that will be created in the future to avoid getting elasticsearch response errors.

Too specific path for the AWS_CREDENTIALS_FILE var

https://github.com/trackit/trackit2/blob/3572ddbe0a316667a30de4b17eedc30934bbe99c/scripts/awsenv#L8

This config is specific to the user schube, and should be changed to something more global, such as ~/.aws/credentials

Document and test code added by #19

#19 will be accepted without the code being properly tested and documented because it is blocking for many other features. This issue tracks the addition of tests and proper documentation.

• Document the bill ingestion
• Test the bill ingestion
• Fix the ElasticSearch mapping for tags
• Support incremental ingestion of bills, not only at the end of the month
• Run the updates periodically

Create the TrackIt task worker

The worker will be able to retrieve tasks from the SQS queue and achieve those ones.

Specifications:

• add a worker flag (--worker=true by example), if the flag is true then run the worker instead of the API
• check if ES is available
• listens to the SQS queue to retrieve tasks
• run the task
• handle task success (send acknowledge to the queue) or error (send changes on visibility timeout to ~15min to re-run the job later)
• send logs on cloudwatch groups logs (separated by task by account: ${task_name}/${account_id}

$GOPATH not set, also , cannot find package "github.com/satori/go.uuid

% ./start.sh
+ [[ ! -f server/main ]]
+ pushd server
~/trackit2/server ~/trackit2
+ ./buildstatic.sh
server.go:21:2: cannot find package "github.com/satori/go.uuid" in any of:
	/usr/lib/go-1.6/src/github.com/satori/go.uuid (from $GOROOT)
	($GOPATH not set)
server.go:22:2: cannot find package "github.com/trackit/jsonlog" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/jsonlog (from $GOROOT)
	($GOPATH not set)
server.go:24:2: cannot find package "github.com/trackit/trackit2/aws" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/aws (from $GOROOT)
	($GOPATH not set)
server.go:25:2: cannot find package "github.com/trackit/trackit2/aws/s3" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/aws/s3 (from $GOROOT)
	($GOPATH not set)
server.go:26:2: cannot find package "github.com/trackit/trackit2/config" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/config (from $GOROOT)
	($GOPATH not set)
server.go:27:2: cannot find package "github.com/trackit/trackit2/routes" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/routes (from $GOROOT)
	($GOPATH not set)
server.go:28:2: cannot find package "github.com/trackit/trackit2/users" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/users (from $GOROOT)
	($GOPATH not set)

so I set $GOPATH to $HOME

% go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/vagrant"
GORACE=""
GOROOT="/usr/lib/go-1.6"
GOTOOLDIR="/usr/lib/go-1.6/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

now when i run start.sh

 % ./start.sh
+ [[ ! -f server/main ]]
+ pushd server
~/trackit2/server ~/trackit2
+ ./buildstatic.sh
server.go:21:2: cannot find package "github.com/satori/go.uuid" in any of:
	/usr/lib/go-1.6/src/github.com/satori/go.uuid (from $GOROOT)
	/home/vagrant/src/github.com/satori/go.uuid (from $GOPATH)
server.go:22:2: cannot find package "github.com/trackit/jsonlog" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/jsonlog (from $GOROOT)
	/home/vagrant/src/github.com/trackit/jsonlog (from $GOPATH)
server.go:24:2: cannot find package "github.com/trackit/trackit2/aws" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/aws (from $GOROOT)
	/home/vagrant/src/github.com/trackit/trackit2/aws (from $GOPATH)
server.go:25:2: cannot find package "github.com/trackit/trackit2/aws/s3" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/aws/s3 (from $GOROOT)
	/home/vagrant/src/github.com/trackit/trackit2/aws/s3 (from $GOPATH)
server.go:26:2: cannot find package "github.com/trackit/trackit2/config" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/config (from $GOROOT)
	/home/vagrant/src/github.com/trackit/trackit2/config (from $GOPATH)
server.go:27:2: cannot find package "github.com/trackit/trackit2/routes" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/routes (from $GOROOT)
	/home/vagrant/src/github.com/trackit/trackit2/routes (from $GOPATH)
server.go:28:2: cannot find package "github.com/trackit/trackit2/users" in any of:
	/usr/lib/go-1.6/src/github.com/trackit/trackit2/users (from $GOROOT)
	/home/vagrant/src/github.com/trackit/trackit2/users (from $GOPATH)

Email is not validated at registration

The email field provided at registration time is never validated and anything can be input. We need an e-mail validation workflow to ensure the e-mail is valid and active.

Page cannot open

I have installed the trackit on a Centos 7 VM and I can see all docker container are up and running.But whenever i tried to connecto the api page via locahost, it shows page cannot connect or open

curl localhost:8580
404 page not found

Can someone please help me out here

Thank You

Issue with govender

When running this command
govendor sync -v

the github asks for the credentials to clone a repository
can you please let me know which credentials should be used there

message":"Failed to connect to SQL database

api_1 Failed to connect to SQL database, getting connect: connection timed out.

Multiple "Query execution failed, ES index does not exists" errors

I'm using the provided docker-compose.yml and the trackit-ui.
I've successfully completed the setup a few days ago and AWS is creating its cost file to an S3 bucket.

Now some parts of the UI are (partialy) working, others don't work at all with errors like:

• No event available (Data not available yet. Please check again in few hours.)
• No data available for this timerange
• Error while getting data (Data not available yet. Please check again in few hours.)
• No reports available (Reports bucket not configured)

The API outputs errors like:
{"level":"error","time":"2022-01-27T10:41:42.3852915Z","message":"Failed to wrap one of the bill repositories","data":{"error":"sql: no rows in result set"}}

{"level":"warning","time":"2022-01-27T09:37:28.1269098Z","message":"Query execution failed, ES index does not exists","data":{"error":"elastic: Error 404 (Not Found): no such index [type=index_not_found_exception]","index":"000001-elasticache-reports"},"context":{"requestId":"bddc05cb-7f54-11ec-9c68-02420a135405"}}

{"level":"warning","time":"2022-01-27T09:37:28.1171287Z","message":"Query execution failed, ES index does not exists","data":{"error":"elastic: Error 404 (Not Found): no such index [type=index_not_found_exception]","index":"000001-rds-reports"},"context":{"requestId":"bddb883f-7f54-11ec-9c68-02420a135405"}}

{"level":"warning","time":"2022-01-27T09:37:28.1219073Z","message":"Query execution failed, ES index does not exists","data":{"error":"elastic: Error 404 (Not Found): no such index [type=index_not_found_exception]","index":"000001-lambda-reports"},"context":{"requestId":"bddc044e-7f54-11ec-9c68-02420a135405"}}

{"level":"warning","time":"2022-01-27T09:37:28.1232164Z","message":"Query execution failed, ES index does not exists","data":{"error":"elastic: Error 404 (Not Found): no such index [type=index_not_found_exception]","index":"000001-es-reports"},"context":{"requestId":"bddb8920-7f54-11ec-9c68-02420a135405"}}

Did i miss something during setup?

Thank you in advance
Thomas

Error while registering user

Hi Team,

I have issue while registering a new user, it throws an error saying "An error has occurred" . Any help would be appreciated.