kuoss/lethe

Lightweight flat log DB

lethe

Contributors

lethe's Issues

check licenses

What would you like to be added?

licenses check

Why is this needed?

licenses check

GNU find

What would you like to be added?

GNU find

Why is this needed?

• https://unix.stackexchange.com/questions/475020/difference-between-find-and-gnu-find

search filter keyword with quotes

What would you like to be added?

search filter keyword with quotes

Why is this needed?

현재는 검색 키워드에 따옴표 없이 되는데 따옴표가 있는 것을 표준으로 하고자 합니다.

remove storage driver interface break point

Describe the bug
fileservice 패키지 코드들 중 driver 인터페이스를 이용하지 않고 파일을 핸들링 하는 경우가 존재합니다.
로그 저장소의 변경을 위해서 모든 파일 핸들링은 driver 인터페이스를 통해서 이루어져야 합니다.

os 패키지의 함수로 직접 파일을 핸들링하고 있습니다.

lethe/storage/fileservice/clean.go

Line 28 in 3442627

err := os.Remove(file)

lethe/storage/fileservice/fileservice.go

Line 23 in 3442627

err = os.MkdirAll(cfg.LogDataPath(), 0755)

lethe/storage/fileservice/list.go

Line 80 in 3442627

files, err := os.ReadDir(logDir.Fullpath)

To Reproduce

Expected behavior
-driver 인터페이스를 이용하여 구현, 필요시 인터페이스 확장

Screenshots

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context

Export log statistics metrics

Export log statistics metrics

list files panic

What happened?

rotator 작동 중 ListFiles에서 panic(nil pointer dereference) 발생

What did you expect to happen?

개선방안

• 표준 error handling 적용 https://github.com/kuoss/lethe/blob/v0.2.0-alpha.1/logs/rotator/list.go#L47
• 기동 시 config 내용 출력(info level)

How can we reproduce it (as minimally and precisely as possible)?

lethe.yml에서 신규 옵션 storage: 누락

Lethe version

lethe:v0.2.0-alpha.1

basic auth

What would you like to be added?

prometheus case
basic auth feature docs
https://prometheus.io/docs/guides/basic-auth/

auth config --web.config.file docs
https://prometheus.io/docs/prometheus/latest/configuration/https/

Why is this needed?

supports basic authentication

DeleteBySize 오동작

시간 순서대로 지워지지 않는다는 제보가 있었어요.

단위 테스트를 만들고 통과하도록 바꿔야겠습니다.

예를 들어 5개를 남긴다면...

node/node1	node/node2	pod/ns1	pod/ns2
	0000-00-00_00.log ❌
	1900-13-01_01.log ❌
	2000-01-01_01.log ❌
		2022-11-31_23.log ❌
2022-12-31_23.log ❌	2022-12-31_23.log ❌	2022-12-31_23.log ❌
2023-01-01_00.log ❌		2023-01-01_00.log ❌
2023-02-21_23.log ❌	2023-02-21_23.log ❌	2023-02-21_23.log ❌	2023-02-21_23.log ❌
2023-02-22_01.log ❌	2023-02-22_01.log ❌	2023-02-22_01.log ❌	2023-02-22_01.log ❌
	2023-02-22_02.log ❌	2023-02-22_02.log ✔️	2023-02-22_02.log ✔️
	2023-02-22_03.log ✔️	2023-02-22_03.log ✔️	2023-02-22_03.log ✔️

multiple operators

multiple operators

dependabot issues 2023-11

dependabot issues

• https://github.com/kuoss/lethe/security/dependabot/1
• https://github.com/kuoss/lethe/security/dependabot/2

bump

kuoss/helm-charts#32

error: not StringLiteral

What happened?

pod{namespace="kube-system"} |= "hello"

{"error":"not StringLiteral rhs: \u0026parser.StepInvariantExpr{Expr:(*parser.StringLiteral)(0xc0004323c0)}","errorType":"queryError","status":"error"}

"io/ioutil" deprecated - SA1019

What happened?

staticcheck Error

Error: routes_test.go:5:2: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details.  (SA1019)
Error: storage/driver/filesystem/driver.go:9:2: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details.  (SA1019)

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Lethe version

OS version

Coveralls 추가

What would you like to be added?

Coveralls 추가

Why is this needed?

현재 Codecov가 적용되어 있는데, Coveralls도 적용하여 사용해보고

나중에 장단점을 비교해보면 좋겠습니다.

DeleteBySize too slow or buggy

DeleteBySize

• sizingStrategy: file - 파일이 매우 많을 경우 너무 느림 (1분 이상 소요)
• sizingStrategy: disk - 제대로 측정이 안됨 마이너스로 나옴

일단 성능과 안정성을 고려하여
기존 sizingStrategy: disk 로직을 복원하고 기본값으로 했으면 합니다.

---

라고 생각했으나,

용량을 계산하면서 삭제할 파일 목록을 작성한 후에, 지우는 것으로 선회

error unwrapped

What would you like to be added?

외부 패키지에서 리턴 받는 에러들 중 래핑되지 않은 경우가 많습니다.

Why is this needed?

전체 에러 스택을 알기 위해서 래핑 해야 합니다.

refactoring log and test enviroment

What would you like to be added?

테스트 환경 세팅 개선

Why is this needed?

Deprecated 함수 제거

lethe writer mock

What would you like to be added?

for testing with mock in workspaces

Why is this needed?

for testing with mock in workspaces

error on handling file retention

GetAllFiles:" no files err=...
GetAllFilesWithSize: no files err=...

rotation_interval config

What would you like to be added?

rotation_interval config 도입

Why is this needed?

retention:
  rotation_interval: 20m ####
  size: 100m
  time: 1d
storage:
  driver: filesystem
  rootdirectory: /data/log

go-test-coverage

What would you like to be added?

go-test-coverage

Why is this needed?

go-test-coverage

staticcheck failed on import

Which jobs are failing?

staticcheck 과정 중
import "internal/cpu"
import "internal/goarch"
부분에서 자주 실패합니다.

Which tests are failing?

github action staticcheck logs...

-: cannot import "internal/cpu" (unexpected escape sequence in export data), possibly version skew - reinstall package (compile)
-: cannot import "internal/goarch" (unknown bexport format version -1

Since when has it been failing?

언젠가부터... ㅎㅎ

Reason for failure (if possible)

No response

Anything else we need to know?

No response

default port 6060

What would you like to be added?

• port config
• default port 6060

workflow go-test 실패

Which jobs are failing?

PR workflow Failing Test

Which tests are failing?

go test with coverage

https://github.com/kuoss/lethe/actions/runs/4312183815/jobs/7522427736

Since when has it been failing?

The beginning

Reason for failure (if possible)

need to find out

Anything else we need to know?

none

logformat json

What would you like to be added?

query?logformat=json

Why is this needed?

logformat=string

# curl lethe/api/v1/query?logformat=string (default, legacy)
{
  "status" : "success"
  "data": {
    "resultType": "logs",
    "result": [
      "2023-03-21T08:19:53.786445361Z[ns|pod1|nginx] hello nginx",
      "2023-03-21T08:19:53.786445361Z[ns|pod1|redis] hello redis 1",
      "2023-03-21T08:19:53.786445361Z[ns|pod1|redis] hello redis 2"
    ]
  }
}

logformat=json

# curl lethe/api/v1/query?logformat=json
{
  "status" : "success"
  "data": {
    "resultType": "logs",
    "result": [
      {"time":"2023-03-21T08:19:53.786445361Z", "namespace":"ns", "pod":"pod1", "container":"nginx", "log":"hello nginx"  },
      {"time":"2023-03-21T08:19:53.786445361Z", "namespace":"ns", "pod":"pod1", "container":"redis", "log":"hello redis 1" },
      {"time":"2023-03-21T08:19:53.786445361Z", "namespace":"ns", "pod":"pod1", "container":"redis", "log":"hello redis 2" }
    ]
  }
}

thousands separator

time="2023-07-17T07:02:27Z" level=info msg="DeleteBySize(23216655413 < 375809638400): DONE" file="delete.go:85"

time="2023-07-17T07:02:27Z" level=info msg="DeleteBySize(23,216,655,413 < 375,809,638,400): DONE" file="delete.go:85"

return warrnings pattern

return warrnings pattern

ndjson log testdata

ndjson log testdata

resultType streams alpha

resultType streams alpha

support sizingStrategy disk for windows

What would you like to be added?

support sizingStrategy disk for windows

#26 (comment)

Why is this needed?

Currently, it is supported only on Linux.

using common logger

What would you like to be added?

using common logger

Why is this needed?

lethe와 venti에서 동일한 logger 사용

improve workflow

What would you like to be added?

improve workflow

Why is this needed?

improve workflow

prometheus handler sugar

What would you like to be added?

기존에 지원하던 api들 중 prometheus 쿼리로 가능한 것들은 query로 전환하려하는데 web에서 호출하는 부분 수정 가능할까요?

/prometheus/time -> /prometheus/query?expr=time()
/prometheus/namespaces -> /prometheus/query?expr=kube_namespace_created
/prometheus/pods/:namespace -> /prometheus/query?expr=kube_pod_created{namespace="namespace01"}
/prometheus/nodes -> /prometheus/query?expr=kube_node_created

### Why is this needed?

/query 인터페이스에 param으로 호출 가능한 결과들이 편의성을 위해서 하드코딩된 인터페이스르 제공하고있다고 생각합니다.

go fmt에 맞지 않음

https://github.com/kuoss/lethe/actions/runs/4231307570/jobs/7349665637 참고

clean error

time="2023-07-17T07:02:27Z" level=warning msg="glob err: walk err: walkFunc err: lstat /data/log/data/log/host.*: no such file or directory, prefix: host" file="clean.go:17"

too slow go-licenses (1h+)

What would you like to be added?

too slow go-licenses (1h+)

https://github.com/kuoss/lethe/actions/runs/4276112104

Why is this needed?

Make the test faster

query API parameter limit

/api/v1/query?limit=100

goimports

goimports check 도입

walkFunc err: lstat /data/log/data/log/host.*

glob err: walk err: walkFunc err: lstat /data/log/data/log/host.*

https://github.com/kuoss/lethe/blob/v0.2.2/storage/driver/filesystem/driver.go#L177-L194

dependabot issues 2023-11 v2

dependabot issues 2023-11 v2

actions annotations warnings

What happened?

https://github.com/kuoss/lethe/actions/runs/5102453005

Annotations 5 warnings

goimports
The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

go-vet
Restore cache failed: Dependencies file is not found in /home/runner/work/lethe/lethe. Supported file pattern: go.sum

go-licenses
Restore cache failed: Dependencies file is not found in /home/runner/work/lethe/lethe. Supported file pattern: go.sum

go-test-failfast
Restore cache failed: Dependencies file is not found in /home/runner/work/lethe/lethe. Supported file pattern: go.sum

go-test-coverage
Restore cache failed: Dependencies file is not found in /home/runner/work/lethe/lethe. Supported file pattern: go.sum

lstat /data/log: no such file or directory

What happened?

time="2023-05-31T05:36:31Z" level=error msg="deleteByAge err: listFiles err: walk err: &{%!e(string=walk err: walkFunc err: lstat /data/log: no such file or directory) %!e(*fmt.wrapError=&{walkFunc err: lstat /data/log: no such file or directory 0xc000466030})}" file="rotator.go:35"
time="2023-05-31T05:36:31Z" level=error msg="deleteBySize err: listFiles err: walk err: &{%!e(string=walk err: walkFunc err: lstat /data/log: no such file or directory) %!e(*fmt.wrapError=&{walkFunc err: lstat /data/log: no such file or directory 0xc000466240})}" file="rotator.go:39"

How can we reproduce it (as minimally and precisely as possible)?

new clean pvc

Anything else we need to know?

.

Lethe version

v0.2.0

multiple filters feature

What would you like to be added?

multiple filters feature

Why is this needed?

multiple filters feature

unexpected files

/data/log/host..service
/data/log/kube.var.log.containers.-.log

go report card issues

What would you like to be added?

• gocyclo 95%
• misspell 98%

Why is this needed?

https://goreportcard.com/report/github.com/kuoss/lethe

gocyclo 95%

Gocyclo calculates cyclomatic complexities of functions in Go source code. The cyclomatic complexity of a function is calculated according to the following rules: 1 is the base complexity of a function +1 for each 'if', 'for', 'case', '&&' or '||' Go Report Card warns on functions with cyclomatic complexity > 15.

letheql/parser/ast.go
Line 356: warning: cyclomatic complexity 18 of function Children() is high (> 15) (gocyclo)
letheql/parser/parse.go
Line 436: warning: cyclomatic complexity 74 of function (*parser).checkAST() is high (> 15) (gocyclo)
letheql/parser/lex.go
Line 350: warning: cyclomatic complexity 47 of function lexStatements() is high (> 15) (gocyclo)
Line 483: warning: cyclomatic complexity 17 of function lexInsideBraces() is high (> 15) (gocyclo)
letheql/engine.go
Line 207: warning: cyclomatic complexity 24 of function preprocessExprHelper() is high (> 15) (gocyclo)

misspell 98%

Misspell Finds commonly misspelled English words

storage/logservice/match/label.go
Line 18: warning: "unknwon" is a misspelling of "unknown" (misspell)

테스트 환경 정비

테스트용 로그 파일들을 Go 코드로 처리하는 것이 비효율적으로 보여서 좀 개선해보고자 합니다.

not contain filter operator

not contain filter operator

logger

logger 도입

윈도우 테스트 통과

#24

Change to use packages modified promql instead of VictoriaSql

What would you like to be added?

Change to use packages modified promql instead of VictoriaSql

Why is this needed?

VictoriaSql has no update anymore and the modified promql is a more suitable form to apply the filter we want.