Comments (7)
Can you share your filebeat config?
from siglens.
this is filebeat config,
$ cat /data/soft/filebeat/filebeat.yml
filebeat.inputs:
- type: log
id: backend-api
enabled: true
paths:
- /data/logs/backend/api.log
tags: ["pd_java_log","backend"]
processors:
- add_locale:
format: abbreviation
timezone: UTC
- dissect:
tokenizer: "%{datetime} [%{thread}] %{logLevel} %{logger} %{pid} - log:[contentType:%{contentType}][request:%{request}][method:%{method}][status:%{status}][body:%{body}][response-contentType:%{response-type}][response:%{response}][duration:%{duration}]"
field: "message"
- timestamp:
field: dissect.datetime
timezone: UTC
layouts:
- '2006-01-02 15:04:05.999'
test:
- '2019-06-22 16:33:51.123'
- drop_event:
when:
not:
has_fields: ['dissect.body']
- drop_fields:
fields: ["message","dissect.thread","dissect.logger","dissect.response-type","dissect.pid"]
output.elasticsearch:
hosts: ['http://172.21.0.247:8081/elastic/']
index: 'filebeat-ind-0'
setup.template.enabled: false
setup.ilm.enabled: false
logs:
2024-04-11 10:15:10.729 [http-nio-8889-exec-27] INFO com.pxxedu.book.web.filter.LogFilter 177 - log:[contentType:application/json][request:/actuator/health][method:POST][status:200][body:{"progressTime":110,"progress":10,"resourceId":3620,"studentId":137150,"replyId":5649963,"reportId":1712801590418,"openId":"","appType":"1"}][response-contentType:application/json][response:{"msg":"请求成功","code":10000,"data":{"replayId":5649963},"success":true,"serverTime":1712801710830}][duration:15ms]
OS:centos 7.9
give it a try and see how it goes
from siglens.
I deleted data folder under filebeat dir before running this. Please try doing that.
I tried using the sample log you provided and the following config file and I was able to see the logline on siglens UI in real time.
filebeat.inputs:
- type: log
id: backend-api
enabled: true
paths:
- /data/logs/backend/api.log
tags: ["pd_java_log","backend"]
processors:
- add_locale:
format: abbreviation
timezone: UTC
- dissect:
tokenizer: "%{datetime} [%{thread}] %{logLevel} %{logger} %{pid} - log:[contentType:%{contentType}][request:%{request}][method:%{method}][status:%{status}][body:%{body}][response-contentType:%{response-type}][response:%{response}][duration:%{duration}]"
field: "message"
- timestamp:
field: dissect.datetime
timezone: UTC
layouts:
- '2006-01-02 15:04:05.999'
test:
- '2019-06-22 16:33:51.123'
- drop_event: # Drop events missing first_name
when:
not:
has_fields: ['dissect.body']
- drop_fields:
fields: ["message","dissect.thread","dissect.logger","dissect.response-type","dissect.pid"]
output.elasticsearch:
hosts: ['http://172.21.0.247:8081/elastic/']
index: 'filebeat-ind-0'
setup.template.enabled: false
setup.ilm.enabled: false
from siglens.
The configuration does not change, delete all files under /var/lib/filebeat/, still the same delay, need to test a longer time, such as half an hour
from siglens.
I deleted data folder under filebeat dir before running this. Please try doing that. I tried using the sample log you provided and the following config file and I was able to see the logline on siglens UI in real time.
filebeat.inputs: - type: log id: backend-api enabled: true paths: - /data/logs/backend/api.log tags: ["pd_java_log","backend"] processors: - add_locale: format: abbreviation timezone: UTC - dissect: tokenizer: "%{datetime} [%{thread}] %{logLevel} %{logger} %{pid} - log:[contentType:%{contentType}][request:%{request}][method:%{method}][status:%{status}][body:%{body}][response-contentType:%{response-type}][response:%{response}][duration:%{duration}]" field: "message" - timestamp: field: dissect.datetime timezone: UTC layouts: - '2006-01-02 15:04:05.999' test: - '2019-06-22 16:33:51.123' - drop_event: # Drop events missing first_name when: not: has_fields: ['dissect.body'] - drop_fields: fields: ["message","dissect.thread","dissect.logger","dissect.response-type","dissect.pid"] output.elasticsearch: hosts: ['http://172.21.0.247:8081/elastic/'] index: 'filebeat-ind-0' setup.template.enabled: false setup.ilm.enabled: false
did you test it? It should be reproducible
from siglens.
Is the log file /data/logs/backend/api.log getting updated continuously? Why do you suggest I should test this for half an hour? With one line of logdata you provided, I could see that line in siglens UI without any delay. I also tested this with another sample log file(siglens.log) and could see the logs in UI.
filebeat.inputs:
- type: log
id: backend-api
enabled: true
paths:
- /Users/github/siglens/logs/siglens.log
tags: ["pd_java_log","backend"]
processors:
- add_locale:
format: abbreviation
timezone: UTC
- dissect:
tokenizer: "%{logLevel}[%{datetime}]"
field: "message"
- timestamp:
field: dissect.datetime
timezone: UTC
layouts:
- '2006-01-02 15:04:05.999'
test:
- '2019-06-22 16:33:51.123'
output.elasticsearch:
hosts: ['http://localhost:8081/elastic/']
index: 'filebeat-ind-0'
setup.template.enabled: false
setup.ilm.enabled: false
from siglens.
I also noticed that the GUI has a delay of approximatly 2 minutes for displaying ingested data. I don't know exactly where this delay is coming from. Could it be some kind of caching of the front end? I also noticed that the delay could be shorter - but typically I have also this 2 minutes offset.
I'm using Vector.dev for log ingesting using the Elasticsearch API endpoint to ingest data.
from siglens.
Related Issues (20)
- [NEW] UI - Save the Dashboard on Panel Resizing
- UI - Update Default Dashboard Dimensions According to New Library
- [BUG] UI - Selected Index in Dashboard Panel Not Appearing in Other Panels
- [NEW] add query time lookup of additional data for enrichment
- [BUG] Refreshing UI logs page messes up Search Filter with spaces
- [BUG] Query Inconsistency on Scrolling in Logs UI
- [BUG] Empty metric name being sent to siglens server for an Internal metric
- [BUG] Exclude '_index' from columnsOrder when 'fields' command is entered
- [BUG] On a dashboard refresh queries for each panel are sent twice
- [BUG] When a dashboard page is loaded, it shows the pop-up of select panel type
- [BUG] Time Picker alignment on MyOrg Cluster stats screen
- [BUG] The Metrics Explorer is not sending correct queries when the TimePicker is Changed.
- [NEW] Add Unit Tests for Arithmetic(+ - * /) operators in metricsresults_test.go HOT 2
- [STORY] Add promql mad_over_time(range-vector) function
- [STORY] Add atan2 operator
- [STORY] Linearly interpolate percentile results
- [STORY] Add Unit Tests / E2E Tests for various Wildcard Search Patterns.
- [BUG] Duplicate Dashboard Names Allowed Between Default and User-Created Dashboards
- [STORY] Visualization Tab - Display Bar Graph for Stats Query on Logs Screen
- [NEW] UI - Implement Keyword Highlighting in Logs Results
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from siglens.