logrange / logrange Goto Github PK

View Code? Open in Web Editor NEW

193.0 12.0 16.0 70.85 MB

High performance data aggregating storage

License: Apache License 2.0

Go 98.67% Makefile 0.05% Shell 1.28%

stream-processing streams logging pipelines golang go database log-aggregation

logrange's People

Stargazers

Watchers

Forkers

ilya-zz cloudxo jorgheymans minxuancao thinkstylestudio backwardn b-xiang isgasho recorvery dongbin86 andypeng2015 adasari awesomedatatool ylwb ajunlonglive thebarton

logrange's Issues

Gravity Adapter: API adapter (repeat wstail functionality)

Collector: Use RPC client API to access to LA

k8s: create Graviational specific helm charts

Collector: Persist its state on to filesystem (review and refactor)

Collector: Support runtime status interface (review and refactor)

LR: add language grammar instead of regexp

LA: data truncation. API to truncate disk space used

k8s: CRD LA service (executable)

We need a CRD specific executable, which will provide LA interface via k8s CRD to an end user and which will be a LA client to provide CRD data to LA. It listens for CRD events to send them to LA

Gravity Adapter: create a separate repo and executable

Fwd: Persist state to filesystem

LA: when multiple sources are used (in tags condition or just no condition), streamming stops working

Collector: File parser (review and refactor)

Change journal Id generating procedure, so the last 2 digits selected randomically

Otherwise all journal goes into one folder :(

LR: Add cmd

Implement basic client. Put it in cmd/lr. Using LA RPC API client for connecting to the LA.

Collector: (suspision) offset can be broken if simultaneous writes happen to a log file

If a log file is written simultaneously from different sources, it looks like it can cause problems at least with fixed formats like json. It seems some lines could be mixed and we may end up with the wrong offset or something like that, so the next time we'll start reading from position where there is no valid json (e.g. in the middle of json line). And we never we'll move one for that file since we always start from that 'broken' position...

Fwd: Create cmd/fwd

to create basic executable which will connect to LA using RPC API client.

Add escape/unescape for names to FS functionality

in logrange/pkg/util 2 functions were defined:
// EscapeToFileName receives a name and turn it to a file-system file name.
// it is expected to escap (substitute) slashes '/' at least.
func EscapeToFileName(name string) string {
// TODO
return fname
}

// UnescapeFileName receives a file name and un-escape it. It supposes that the
// file name was escaped by EscapeToFileName() before
func UnescapeFileName(fname string) string {
//TODO
return fname
}

it needs to implement them.

Logrange: 1 click installation

We need to have the following scripts for 1 click installation:

standalone
k8s
docker
website visitor will have a chance to copy-paste a command and have the logrange up and running within seconds.

LA: Design logrange server RPC API (for collector and Fwd)

TDB

LA: Don't return error if there is no data, just return empty data "no sources for the expression the query..."

subj.

Setup default TLS according to modern compat

Most often then not security audits always require strict approved set of ciphers/protocols. The easiest way to pass all audits wrt to protocol is to always use mTLS and modern compat ciphers:

https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility

Fwd: Configuration reader

Read configuration from file system. Support configuration re-read (without restart) if config file is updated.

Quick start: Nice to have for Ubuntu users

I tried the quick start guide on Ubuntu 18.10 and have some minor comments. My assumptions the quick start is a set of commands to run from a live console and try the product quickly.

Command curl -s http://get.logrange.io/install | bash -s logrange doesn't work for me because Permission denied there is my console output:
$ curl -s http://get.logrange.io/install | bash -s logrange
Downloading from http://get.logrange.io/binaries/master/linux/x86_64/logrange...
mv: cannot move './logrange' to '/usr/local/bin/logrange': Permission denied
Done! Try running 'logrange'...
As an option please consider adding sudo:
curl -s http://get.logrange.io/install | **sudo** bash -s logrange
Also it would be nice to check that mv ./${CMD} /usr/local/bin is actually copied the file and provide recommendation to use sudo if the file was not copied because of lack of permissions (or consider an another folder). So, the console output can be:
$ curl -s http://get.logrange.io/install | bash -s logrange
Downloading from http://get.logrange.io/binaries/master/linux/x86_64/logrange...
Moving './logrange' to '/usr/local/bin/logrange'
mv: cannot move './logrange' to '/usr/local/bin/logrange': Permission denied
Please run with sudo: curl -s http://get.logrange.io/install | sudo bash -s logrange
Optionally, the destination folder may not exist for some reasons so better to check it is there.
Same experience with curl -s http://get.logrange.io/install | bash
Command logrange start --journals-dir /tmp does not work for the same permissions denied reason. 'sudo logrange start --journals-dir /tmp &' works but all the index files belongs to root. Also without & it eats my console.
Please consider changing Done! Try running 'lr'... to 'Done! Try running 'lr collect'' and also adding 'Then try running 'lr shell''
lr shell might print 'Type help for list of supported commands' after the logo, also the version of lr would be useful.
Also, It might be useful to add a section for 'How I can get rid of the tool if I want to?'

logrange / logrange Goto Github PK

logrange's People

Stargazers

Watchers

Forkers

logrange's Issues

Recommend Projects

Recommend Topics

Recommend Org