Problems configuring logstash using manticore

[haninerabahfawaz]

Problems configuring logstash using manticore

[haninerabahfawaz]

I removed all files related to (manticore, logstash, elasticsearch) and then installed proper version of logstash integration in files mentioned before and it works after creating files without change in elasticsearch.yml...

…

On Mon, 17 Jun 2024, 11:15 am Sergey Nikolaev, ***@***.***> wrote: I want to thank everyone that my code is finally working well. @haninerabahfawaz <https://github.com/haninerabahfawaz> what was the problem? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCMRG3VVJ63K3U2VKDZH2LKRAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGU4TKNZYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PavelShilin89]

Please let me know if the files have been uploaded and appear in manticore storage @sanikolaev

Your data has been received. We are working on the problem.

[haninerabahfawaz]

My goal is to parse the log files and send them to Manticore to insert the data into the index table.

[tomatolog]

manual about Logstash-configuration shows such output example

output {
  elasticsearch {
   index => " dpkg_log"
   hosts => ["http://localhost:9308"]
   ilm_enabled => false
   manage_template => false
  }
}

the is no any url => "http://localhost:9308/insert" properties

[haninerabahfawaz]

How can we use Elasticsearch in the same port as manticore (9308). I have already modified the default port of Elasticsearch to 9308 (default port of eslasticsearch: 9200), and I am getting an error.

…

On Thu, 13 Jun 2024, 10:26 pm Stas, ***@***.***> wrote: manual about Logstash-configuration <https://manual.manticoresearch.com/dev/Integration/Logstash#Logstash-configuration> shows such output example output { elasticsearch { index => " dpkg_log" hosts => ["http://localhost:9308"] ilm_enabled => false manage_template => false } } the is no any url => "http://localhost:9308/insert" properties — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDEZWGFTLIYULN42OI3ZHHXABAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGYYTCOBTG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tomatolog]

I sure you can not use multiple applications on the same ports. you need to assign different ports for each application

[haninerabahfawaz]

The port of elasticsearch is 9200 and the port of manticore is 9308. How we can analyse the data and send them to manticore?

…

On Thu, 13 Jun 2024, 11:17 pm Stas, ***@***.***> wrote: I sure you can not use multiple applications on the same ports. you need to assign different ports for each application — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDB26CNPVHYWEEHYEL3ZHH433AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGY4DSOBYG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[haninerabahfawaz]

I run this code and I get this error:

2024-06-13 23:54:57.294 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9308/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9308/][Manticore::ClientProtocolException] localhost:9308 failed to respond"}

input {
file {
path => ["/var/log/dpkg.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
mode => "read"
exit_after_read => "true"
file_completed_action => "log"
file_completed_log_path => "/dev/null"
}
}

output {
elasticsearch {
index => "logsnew"
hosts => ["http://localhost:9308"]
ilm_enabled => false
manage_template => false
}
}

[haninerabahfawaz]

This is my elasticsearch.yml:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.108
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: 192.168.0.108
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 13-06-2024 11:32:04
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled:  false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["haninefawaz-VirtualBox"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

[haninerabahfawaz]

@tomatolog any idea?

[sanikolaev]

@haninerabahfawaz The task is assigned to @Nick-S-2018 and we'll get to it eventually. In the meantime, what you can do to help yourself is try to understand why this https://play.manticoresearch.com/logstash/ works, but it doesn't work in your case.

[haninerabahfawaz]

Hi, I want to thank everyone that my code is finally working well. I want to ask if we have something or command that updates data in manticore daily because we have daily logs.

…

On Fri, 14 Jun 2024, 6:30 pm Sergey Nikolaev, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> The task is assigned to @Nick-S-2018 <https://github.com/Nick-S-2018> and we'll get to it eventually. In the meantime, what you can do to help yourself is try to understand why this https://play.manticoresearch.com/logstash/ works, but it doesn't work in your case. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDGGXZF32X4QNGTWUTDZHMEBVAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYGI3TMNZUG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Nick-S-2018]

Do you mean you need to add new log data from Logstash to Manticore regularly? I suppose, if you use the input file plugin, this plugin option should help: https://www.elastic.co/guide/en/logstash/7.10/plugins-inputs-file.html#plugins-inputs-file-stat_interval

[sanikolaev]

I want to thank everyone that my code is finally working well.

@haninerabahfawaz what was the problem?

[haninerabahfawaz]

Yes, I want the Manticore table to be updated automatically when new data is added to the log data.

…

On Mon, 17 Jun 2024, 10:33 am Nick Sergeev, ***@***.***> wrote: Do you mean you need to add new log data from Logstash to Manticore regularly? I suppose, if you use the input file plugin, this plugin option should help: https://www.elastic.co/guide/en/logstash/7.10/plugins-inputs-file.html#plugins-inputs-file-stat_interval — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDERQINCDTUES4BIGN3ZH2GNRAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGUYTGNRRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

Hi, I got this error after I added stat_interval input { file { path => [ "/var/log/dpkg.log", "/var/log/syslog", "/var/log/kern.log", "/var/log/auth.log", "/opt/lampp/logs/access_log", "/opt/lampp/logs/error_log" ] start_position => "beginning" sincedb_path => "/dev/null" mode => "read" exit_after_read => "true" file_completed_action => "log" file_completed_log_path => "/dev/null" stat_interval => "5 second" } } output { elasticsearch { index => "logsinfo" hosts => ["http://localhost:9308"] ilm_enabled => false manage_template => false } } *[ERROR] 2024-06-18 15:45:50.330 [[main]>worker1] elasticsearch - Encountered a retryable error (will retry with exponential backoff) {:code=>409, :url=>"http://localhost:9308/_bulk <http://localhost:9308/_bulk>", :content_length=>31390} *

…

On Mon, 17 Jun 2024 at 10:33, Nick Sergeev ***@***.***> wrote: Do you mean you need to add new log data from Logstash to Manticore regularly? I suppose, if you use the input file plugin, this plugin option should help: https://www.elastic.co/guide/en/logstash/7.10/plugins-inputs-file.html#plugins-inputs-file-stat_interval — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDERQINCDTUES4BIGN3ZH2GNRAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGUYTGNRRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

@Nick-S-2018

Hi, I got this error after I added stat_interval input { file { path => [ "/var/log/dpkg.log", "/var/log/syslog", "/var/log/kern.log", "/var/log/auth.log", "/opt/lampp/logs/access_log", "/opt/lampp/logs/error_log" ] start_position => "beginning" sincedb_path => "/dev/null" mode => "read" exit_after_read => "true" file_completed_action => "log" file_completed_log_path => "/dev/null" stat_interval => "5 second" } } output { elasticsearch { index => "logsinfo" hosts => ["http://localhost:9308"] ilm_enabled => false manage_template => false } } [ERROR] 2024-06-18 15:45:50.330 [[main]>worker1] elasticsearch - Encountered a retryable error (will retry with exponential backoff) {:code=>409, :url=>"http://localhost:9308/_bulk http://localhost:9308/_bulk", :content_length=>31390} *
…
On Mon, 17 Jun 2024 at 10:33, Nick Sergeev @.> wrote: Do you mean you need to add new log data from Logstash to Manticore regularly? I suppose, if you use the input file plugin, this plugin option should help: https://www.elastic.co/guide/en/logstash/7.10/plugins-inputs-file.html#plugins-inputs-file-stat_interval — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A33WFDERQINCDTUES4BIGN3ZH2GNRAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGUYTGNRRG4 . You are receiving this because you were mentioned.Message ID: @.*>

[sanikolaev]

Try adding:

max_packet_size = 128m

to section "searchd" of your Manticore configuration file and restarting it.

[haninerabahfawaz]

@sanikolaev *SAME ERROR:* [ERROR] 2024-06-20 14:19:13.638 [[main]>worker1] elasticsearch - Encountered a retryable error (will retry with exponential backoff) {:code=>409, :url=>"http://localhost:9308/_bulk", :content_length=>31128}

…

On Wed, 19 Jun 2024 at 21:00, Sergey Nikolaev ***@***.***> wrote: Try adding: max_packet_size = 128m to section "searchd" of your Manticore configuration file and restarting it. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDGVT6SUXCTKOLWK7MTZIHBKBAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZGI2TEOBTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sanikolaev]

Can you please provide an instruction how to reproduce it from scratch?

[haninerabahfawaz]

@sanikolaev

manticore.conf:

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    max_packet_size = 128m
}

index loggsdata
    {
        type = rt
	path = /var/lib/manticore/loggsdata
        rt_attr_bigint = id
        rt_field = host
        rt_field = @timestamp
        rt_field = message
        rt_field = @version
        rt_field = path
    }

index logsdata
    {
        type = rt
        path = /var/lib/manticore/logsdata
        rt_attr_bigint = id
        rt_field = host
        rt_field = @timestamp
        rt_field = message
        rt_field = @version
        rt_field = path
    }
    
index logsinfo
    {
        type = rt
        path = /var/lib/manticore/logsinfo
        rt_attr_bigint = id
        rt_field = host
        rt_field = @timestamp
        rt_field = message
        rt_field = @version
        rt_field = path
    }
    
index loggs
    {
        type = rt
        path = /var/lib/manticore/loggs
        rt_attr_bigint = id
        rt_field = host
        rt_field = @timestamp
        rt_field = message
        rt_field = @version
        rt_field = path
    }

logstash:

input {
  file {
    path => [
      "/var/log/dpkg.log",
      "/var/log/syslog",
      "/var/log/kern.log",
      "/var/log/auth.log",
      "/opt/lampp/logs/access_log",
      "/opt/lampp/logs/error_log"
    ]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
    stat_interval => "5 second"
  }
}

output {
  elasticsearch {
    index => "loggs"
    hosts => ["http://localhost:9308"]
    ilm_enabled => false
    manage_template => false
  }
}

COMMAND:

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstashloggs.conf

[sanikolaev]

Please provide these files:

"/var/log/dpkg.log",
"/var/log/syslog",
"/var/log/kern.log",
"/var/log/auth.log",
"/opt/lampp/logs/access_log",
"/opt/lampp/logs/error_log"

[haninerabahfawaz]

Hi, I want to ask if the log files are sent correctly?

…

On Fri, 21 Jun 2024 at 07:23, Sergey Nikolaev ***@***.***> wrote: Please provide these files: "/var/log/dpkg.log", "/var/log/syslog", "/var/log/kern.log", "/var/log/auth.log", "/opt/lampp/logs/access_log", "/opt/lampp/logs/error_log" — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCXCILBJS2ZCYQIBP3ZIOTEDAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRHE3TIMZZGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sanikolaev]

I want to ask if the log files are sent correctly?

Where did you send them? I can't find them on our s3 storage.

[haninerabahfawaz]

By mail, I will resend them by using we transfer.

…

On Mon, 24 Jun 2024, 11:39 am Sergey Nikolaev, ***@***.***> wrote: I want to ask if the log files are sent correctly? Where did you send them? I can't find them on our s3 storage. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDAWHG2YDEPVTKRYLV3ZI7LLLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHEZTGMJUGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[haninerabahfawaz]

Please, tell me if the files have been sent.

…

On Mon, 24 Jun 2024 at 11:53, Hanine Fawaz ***@***.***> wrote: On Mon, 24 Jun 2024 at 11:49, Hanine Fawaz ***@***.***> wrote: > By mail, I will resend them by using we transfer. > > On Mon, 24 Jun 2024, 11:39 am Sergey Nikolaev, ***@***.***> > wrote: > >> I want to ask if the log files are sent correctly? >> >> Where did you send them? I can't find them on our s3 storage. >> >> — >> Reply to this email directly, view it on GitHub >> <#2307 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/A33WFDAWHG2YDEPVTKRYLV3ZI7LLLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHEZTGMJUGI> >> . >> You are receiving this because you are subscribed to this thread.Message >> ID: ***@***.*** >> com> >> >

[sanikolaev]

I've received your files, thanks.

@PavelShilin89 pls try to reproduce the issue. I'm forwarding the files to you.

[haninerabahfawaz]

Any news? I want the index to always be up to date.

…

On Mon, 24 Jun 2024, 7:27 pm Sergey Nikolaev, ***@***.***> wrote: I've received your files, thanks. @PavelShilin89 <https://github.com/PavelShilin89> pls try to reproduce the issue. I'm forwarding the files to you. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDE6W37KQZDE67IQZQ3ZJBCH7AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWHE3DGNRWGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sanikolaev]

@haninerabahfawaz, sorry, we have no updates at the moment. Please remain patient. If you require a quicker resolution, consider exploring our professional services with an SLA. You can find more information here: https://manticoresearch.com/services

[PavelShilin89]

@haninerabahfawaz please specify which version of logstash, manticore you are using, and which os you have?

[haninerabahfawaz]

manticore:6.3.0 logstash: 7.14.0 os:linux ubuntu.

…

On Wed, 26 Jun 2024 at 12:55, Pavel Shilin ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> please specify which version of logstash, manticore you are using, and which os you have? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDF6KJMSOPFQMRXYOHTZJKFYLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGI4DOOJVHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PavelShilin89]

manticore:6.3.0 logstash: 7.14.0 os:linux ubuntu.
…
On Wed, 26 Jun 2024 at 12:55, Pavel Shilin @.> wrote: @haninerabahfawaz https://github.com/haninerabahfawaz please specify which version of logstash, manticore you are using, and which os you have? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A33WFDF6KJMSOPFQMRXYOHTZJKFYLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGI4DOOJVHA . You are receiving this because you were mentioned.Message ID: @.>

We currently support Logstash version 7.10.0, and everything works correctly on this version.

[haninerabahfawaz]

Okay,I used stat_interval to update the index if any records are added. correct?

…

On Wed, 26 Jun 2024, 1:37 pm Pavel Shilin, ***@***.***> wrote: manticore:6.3.0 logstash: 7.14.0 os:linux ubuntu. … <#m_-7273637130236502109_> On Wed, 26 Jun 2024 at 12:55, Pavel Shilin *@*.*> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> https://github.com/haninerabahfawaz <https://github.com/haninerabahfawaz> please specify which version of logstash, manticore you are using, and which os you have? — Reply to this email directly, view it on GitHub <#2307 (comment) <#2307 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A33WFDF6KJMSOPFQMRXYOHTZJKFYLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGI4DOOJVHA <https://github.com/notifications/unsubscribe-auth/A33WFDF6KJMSOPFQMRXYOHTZJKFYLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGI4DOOJVHA> . You are receiving this because you were mentioned.Message ID: @.*> We currently support Logstash version 7.10.0, and everything works correctly on this version. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDE5WA2HIFRJHS4MLOLZJKKX3AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGM3DSNRZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sanikolaev]

Okay,I used stat_interval to update the index if any records are added.

Does it have anything to do with the versions?

Is there any problem left? If stat_interval works for you - good. If it doesn't, what's the problem?

[haninerabahfawaz]

I want to remove my logstash version and try to install version 7.10.0, then I will add stat_interval.

…

On Wed, 26 Jun 2024 at 14:12, Sergey Nikolaev ***@***.***> wrote: Okay,I used stat_interval to update the index if any records are added. Does it have anything to do with the versions? Is there any problem left? If stat_interval works for you - good. If it doesn't, what's the problem? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDDF7EIEXM5JLZKC3JLZJKOYBAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGQZTGOJQGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

I want to share something with you, I've noticed this when I'm typing on my device. select count(*) from logggs; The number has been increased which means it is ok but the number has increased more than 10000 which I think is not logical. And also when I write another query select count(*) from logggs where match('syslog') The number is always fixed, which means something illogical.

…

On Wed, 26 Jun 2024, 2:12 pm Sergey Nikolaev, ***@***.***> wrote: Okay,I used stat_interval to update the index if any records are added. Does it have anything to do with the versions? Is there any problem left? If stat_interval works for you - good. If it doesn't, what's the problem? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDDF7EIEXM5JLZKC3JLZJKOYBAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGQZTGOJQGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PavelShilin89]

@haninerabahfawaz I'm working with the files you sent, but I don't see in the logggs table.
I got this:

mysql> SHOW TABLES;
+-----------+------+
| Index     | Type |
+-----------+------+
| loggs     | rt   |
| loggsdata | rt   |
| logsdata  | rt   |
| logsinfo  | rt   |
+-----------+------+
mysql> SELECT COUNT(*) FROM loggs;
+----------+
| count(*) |
+----------+
|   140846 |
+----------+
mysql> select count(*) from loggs where match('syslog');
+----------+
| count(*) |
+----------+
|   110650 |
+----------+

The number of lines remains unchanged.

[haninerabahfawaz]

I have created a new table like the other tables...

…

On Wed, 26 Jun 2024, 8:46 pm Pavel Shilin, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> I'm working with the files you sent, but I don't see in the logggs table. I got this: mysql> SHOW TABLES; +-----------+------+ | Index | Type | +-----------+------+ | loggs | rt | | loggsdata | rt | | logsdata | rt | | logsinfo | rt | +-----------+------+ mysql> SELECT COUNT(*) FROM loggs; +----------+ | count(*) | +----------+ | 140846 | +----------+ mysql> select count(*) from loggs where match('syslog'); +----------+ | count(*) | +----------+ | 110650 | +----------+ The number of lines remains unchanged. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCGXZKLTTRH6JM6OVLZJL455AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGI4TCNBVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

As I mentioned, the number of records increased after a period of more than 10,000, but it does not make sense for me to write a query and the number increases in an illogical way. Which made me surprised. Assuming this is true, why is the number of matches always constant?

…

On Wed, 26 Jun 2024, 8:58 pm Hanine Fawaz, ***@***.***> wrote: I have created a new table like the other tables... On Wed, 26 Jun 2024, 8:46 pm Pavel Shilin, ***@***.***> wrote: > @haninerabahfawaz <https://github.com/haninerabahfawaz> I'm working with > the files you sent, but I don't see in the logggs table. > I got this: > > mysql> SHOW TABLES; > +-----------+------+ > | Index | Type | > +-----------+------+ > | loggs | rt | > | loggsdata | rt | > | logsdata | rt | > | logsinfo | rt | > +-----------+------+ > mysql> SELECT COUNT(*) FROM loggs; > +----------+ > | count(*) | > +----------+ > | 140846 | > +----------+ > mysql> select count(*) from loggs where match('syslog'); > +----------+ > | count(*) | > +----------+ > | 110650 | > +----------+ > > The number of lines remains unchanged. > > — > Reply to this email directly, view it on GitHub > <#2307 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A33WFDCGXZKLTTRH6JM6OVLZJL455AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGI4TCNBVGM> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[djklim87]

And also when I write another query
select count(*) from logggs where match('syslog')
The number is always fixed, which means something illogical.

You should probably check what duplicates you have, and definitely, it doesn't contain the word 'syslog'

[haninerabahfawaz]

Yes, I actually used another term to match...same thing nothing has changed...

…

On Wed, 26 Jun 2024 at 23:00, Klim Todrik ***@***.***> wrote: And also when I write another query select count(*) from logggs where match('syslog') The number is always fixed, which means something illogical. You should probably check what duplicates you have, and definitely, it doesn't contain the word 'syslog' — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDHXOAIYJOAW42NYSTTZJMMU3AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGUZDKNJYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

I would like to share something please, I noticed when I rebooted the device and logged in that the number of records is fixed, no changes... I check there is an error because when the number of records increases the timestamp is the same as the execution time....

…

On Wed, 26 Jun 2024 at 20:46, Pavel Shilin ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> I'm working with the files you sent, but I don't see in the logggs table. I got this: mysql> SHOW TABLES; +-----------+------+ | Index | Type | +-----------+------+ | loggs | rt | | loggsdata | rt | | logsdata | rt | | logsinfo | rt | +-----------+------+ mysql> SELECT COUNT(*) FROM loggs; +----------+ | count(*) | +----------+ | 140846 | +----------+ mysql> select count(*) from loggs where match('syslog'); +----------+ | count(*) | +----------+ | 110650 | +----------+ The number of lines remains unchanged. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCGXZKLTTRH6JM6OVLZJL455AVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGI4TCNBVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PavelShilin89]

@haninerabahfawaz hi! please check the path you specify to the dpkg.log file when indexing, maybe you are using a base dpkg.log that is created in a different path and it is causing more data than you expect. Maybe you need to specify the full path to the file everywhere and respect line breaks. If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem.

[haninerabahfawaz]

Hello, As for the dpkg.log file, I don't need it. I'm interested in the files (syslog/kern.log/auth.log/access_log/error_log).

…

On Fri, 28 Jun 2024, 1:02 am Pavel Shilin, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> hi! please check the path you specify to the dpkg.log file when indexing, maybe you are using a base dpkg.log that is created in a different path and it is causing more data than you expect. Maybe you need to specify the full path to the file everywhere and respect line breaks. If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCQWKEG4SRPONDZBUTZJSDYDAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVG4ZTGMRYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

start_interval was working yesterday, at first the number of lines was increasing, then it became stable, and today it has not changed at all. I think I need another way to try it....

…

On Fri, 28 Jun 2024, 1:02 am Pavel Shilin, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> hi! please check the path you specify to the dpkg.log file when indexing, maybe you are using a base dpkg.log that is created in a different path and it is causing more data than you expect. Maybe you need to specify the full path to the file everywhere and respect line breaks. If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCQWKEG4SRPONDZBUTZJSDYDAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVG4ZTGMRYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

Hi, Please guys if there is any updates and if you want any files from me let me know. I just want to update the index if any record is added to any log file then the index should be updated automatically... thank you all.

…

On Fri, 28 Jun 2024, 1:02 am Pavel Shilin, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> hi! please check the path you specify to the dpkg.log file when indexing, maybe you are using a base dpkg.log that is created in a different path and it is causing more data than you expect. Maybe you need to specify the full path to the file everywhere and respect line breaks. If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCQWKEG4SRPONDZBUTZJSDYDAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVG4ZTGMRYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tomatolog]

as said in the last comment

If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem.

it could be better if you provide all files these reproduces this case locally (configs, data source \ logs and query that shows increasing documents)

[haninerabahfawaz]

OK, I will send them tomorrow early morning.

…

On Sat, 29 Jun 2024, 1:32 am Stas, ***@***.***> wrote: as said in the last comment <#2307 (comment)> If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem. it could be better if you provide all files these reproduces this case locally (configs, data source \ logs and query that shows increasing documents) — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCNZERUUOQYUQJA2LLZJXP6HAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXG4ZTINZTG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

Hi, *Logstash.conf:* input { file { path => [ "/var/log/syslog", "/var/log/kern.log", "/var/log/auth.log", "/opt/lampp/logs/access_log", "/opt/lampp/logs/error_log" ] start_position => "beginning" sincedb_path => "/dev/null" mode => "read" exit_after_read => "true" file_completed_action => "log" file_completed_log_path => "/dev/null" stat_interval => "5 second" } } filter { if [path] == "/var/log/syslog" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/var/log/kern.log" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/var/log/auth.log" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/opt/lampp/logs/access_log" { grok { match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } remove_field => ["clientip", "ident", "auth", "method", "request", "httpversion", "response", "bytes"] } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] timezone => "Asia/Beirut" target => ***@***.***" } mutate { remove_field => ["timestamp"] } } if [path] == "/opt/lampp/logs/error_log" { grok { match => { "message" => "\[%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}\] \[%{DATA:loglevel}\] \[pid %{NUMBER:pid}\] %{GREEDYDATA:rest}" } } mutate { add_field => { "timestamp" => "%{day} %{month} %{monthday} %{time} %{year}" } } date { match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] timezone => "Asia/Beirut" target => ***@***.***" } mutate { remove_field => ["day", "month", "monthday", "time", "year", "timestamp"] } } } output { elasticsearch { index => "loginf" hosts => ["http://localhost:9308"] ilm_enabled => false manage_template => false } stdout { codec => rubydebug } } *Table:* mysql> DESCRIBE loginf; +------------+-----------+----------------+ | Field | Type | Properties | +------------+-----------+----------------+ | id | bigint | | | host | text | indexed stored | | message | text | indexed stored | | @Version | text | indexed stored | | pid | text | indexed stored | | rest | text | indexed stored | | loglevel | text | indexed stored | | path | text | indexed stored | | @timestamp | timestamp | | | tags | json | | +------------+-----------+----------------+ 10 rows in set (0.00 sec) *COMMAND:* sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/loginf.conf *manticore.conf:* searchd { listen = 127.0.0.1:9312 listen = 127.0.0.1:9306:mysql listen = 127.0.0.1:9308:http log = /var/log/manticore/searchd.log query_log = /var/log/manticore/query.log pid_file = /var/run/manticore/searchd.pid data_dir = /var/lib/manticore } The log files will be sent after this message.

…

On Sat, 29 Jun 2024 at 01:32, Stas ***@***.***> wrote: as said in the last comment <#2307 (comment)> If the problem persists, it would be great if you could send us all the files, configurations, logs and commands you are using. So that we can correctly reproduce the problem. it could be better if you provide all files these reproduces this case locally (configs, data source \ logs and query that shows increasing documents) — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCNZERUUOQYUQJA2LLZJXP6HAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXG4ZTINZTG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

syslog <https://drive.google.com/file/d/1BnHATaCJjVut7dHrQRHkKo7wnxv-cIfN/view?usp=drive_web>

…

On Sat, 29 Jun 2024 at 16:32, Hanine Fawaz ***@***.***> wrote: *Log Files:* On Sat, 29 Jun 2024 at 16:29, Hanine Fawaz ***@***.***> wrote: > Hi, > > *Logstash.conf:* > input { > file { > path => [ > "/var/log/syslog", > "/var/log/kern.log", > "/var/log/auth.log", > "/opt/lampp/logs/access_log", > "/opt/lampp/logs/error_log" > ] > start_position => "beginning" > sincedb_path => "/dev/null" > mode => "read" > exit_after_read => "true" > file_completed_action => "log" > file_completed_log_path => "/dev/null" > stat_interval => "5 second" > } > } > > filter { > if [path] == "/var/log/syslog" { > grok { > match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} > %{GREEDYDATA}" } > } > date { > match => [ "log_timestamp", "ISO8601" ] > target => ***@***.***" > remove_field => ["log_timestamp"] > } > } > > if [path] == "/var/log/kern.log" { > grok { > match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} > %{GREEDYDATA}" } > } > date { > match => [ "log_timestamp", "ISO8601" ] > target => ***@***.***" > remove_field => ["log_timestamp"] > } > } > > if [path] == "/var/log/auth.log" { > grok { > match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} > %{GREEDYDATA}" } > } > date { > match => [ "log_timestamp", "ISO8601" ] > target => ***@***.***" > remove_field => ["log_timestamp"] > } > } > > if [path] == "/opt/lampp/logs/access_log" { > grok { > match => { "message" => "%{IPORHOST:clientip} %{USER:ident} > %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{DATA:request} > HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } > remove_field => ["clientip", "ident", "auth", "method", "request", > "httpversion", "response", "bytes"] > } > > date { > match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] > timezone => "Asia/Beirut" > target => ***@***.***" > } > > mutate { > remove_field => ["timestamp"] > } > } > > if [path] == "/opt/lampp/logs/error_log" { > grok { > match => { "message" => "\[%{DAY:day} %{MONTH:month} > %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}\] \[%{DATA:loglevel}\] \[pid > %{NUMBER:pid}\] %{GREEDYDATA:rest}" } > } > > mutate { > add_field => { "timestamp" => "%{day} %{month} %{monthday} %{time} > %{year}" } > } > > date { > match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] > timezone => "Asia/Beirut" > target => ***@***.***" > } > > mutate { > remove_field => ["day", "month", "monthday", "time", "year", > "timestamp"] > } > } > } > > output { > elasticsearch { > index => "loginf" > hosts => ["http://localhost:9308"] > ilm_enabled => false > manage_template => false > } > stdout { codec => rubydebug } > } > > *Table:* > mysql> DESCRIBE loginf; > +------------+-----------+----------------+ > | Field | Type | Properties | > +------------+-----------+----------------+ > | id | bigint | | > | host | text | indexed stored | > | message | text | indexed stored | > | @Version | text | indexed stored | > | pid | text | indexed stored | > | rest | text | indexed stored | > | loglevel | text | indexed stored | > | path | text | indexed stored | > | @timestamp | timestamp | | > | tags | json | | > +------------+-----------+----------------+ > 10 rows in set (0.00 sec) > > *COMMAND:* > sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/loginf.conf > > *manticore.conf:* > searchd { > listen = 127.0.0.1:9312 > listen = 127.0.0.1:9306:mysql > listen = 127.0.0.1:9308:http > log = /var/log/manticore/searchd.log > query_log = /var/log/manticore/query.log > pid_file = /var/run/manticore/searchd.pid > data_dir = /var/lib/manticore > } > > The log files will be sent after this message. > > On Sat, 29 Jun 2024 at 01:32, Stas ***@***.***> wrote: > >> as said in the last comment >> <#2307 (comment)> >> >> If the problem persists, it would be great if you could send us all the >> files, configurations, logs and commands you are using. So that we can >> correctly reproduce the problem. >> >> it could be better if you provide all files these reproduces this case >> locally (configs, data source \ logs and query that shows increasing >> documents) >> >> — >> Reply to this email directly, view it on GitHub >> <#2307 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/A33WFDCNZERUUOQYUQJA2LLZJXP6HAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXG4ZTINZTG4> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

[haninerabahfawaz]

I will send the files to @sanikolaev. Please send the files to everyone, because I can't send them here.

[tomatolog]

you can upload your files into our s3 storage as described in the manual https://manual.manticoresearch.com/dev/Reporting_bugs#Uploading-your-data

[haninerabahfawaz]

Hello, I have uploaded the log files and you can find them at manticore-logs-2307. Thank you.

…

On Sat, 29 Jun 2024, 7:41 pm Stas, ***@***.***> wrote: you can upload your files into our s3 storage as deceived in the manual https://manual.manticoresearch.com/dev/Reporting_bugs#Uploading-your-data — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDB2XIERAWW5VF6ZVDLZJ3PVLAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGI2TMNRWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sanikolaev]

you can find them at manticore-logs-2307

I can't find them. Pls upload to issue-2307.

[haninerabahfawaz]

[image: image.png]

…

On Mon, 1 Jul 2024 at 12:34, Sergey Nikolaev ***@***.***> wrote: you can find them at manticore-logs-2307 I can't find them. Pls upload to issue-2307. — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDCLOPYC7PKNERFFILDZKEPDRAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJZGY4DCNJYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[haninerabahfawaz]

hanine@hanine-VirtualBox:/minio-binaries$ ./mc cp -r $HOME/issue-2307/ manticore/write-only/issue-2307/
...7/auth.log: 9.78 MiB / 9.78 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 285.13 MiB/s 0shanine@hanine-VirtualBox:/minio-binaries$ ls
manticore mc
hanine@hanine-VirtualBox:/minio-binaries$ cd manticore
hanine@hanine-VirtualBox:/minio-binaries/manticore$ ls
write-only
hanine@hanine-VirtualBox:/minio-binaries/manticore$ cd write-only
hanine@hanine-VirtualBox:/minio-binaries/manticore/write-only$ ls
issue-2307 manticore-logs-2307
hanine@hanine-VirtualBox:/minio-binaries/manticore/write-only$ cd issue-2307
hanine@hanine-VirtualBox:/minio-binaries/manticore/write-only/issue-2307$ ls
access_log auth.log error_log kern.log syslog

@sanikolaev I also uploaded it in issue-2307

[haninerabahfawaz]

Please let me know if the files have been uploaded and appear in manticore storage @sanikolaev

[haninerabahfawaz]

any news? @PavelShilin89

[PavelShilin89]

@haninerabahfawaz try adding max_packet_size = 128M to manticore.conf.

This way I was able to fully index all the data without loss:

docker network create t_network 

cat <<EOL > /tmp/manticore.conf
searchd {
    listen = 0.0.0.0:9312
    listen = 0.0.0.0:9306:mysql
    listen = 0.0.0.0:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    max_packet_size = 128M
}
EOL

cat <<EOL > /tmp/logstash.conf
input {
  file {
    path => [
      "/var/log/syslog",
      "/var/log/kern.log",
      "/var/log/auth.log",
      "/opt/lampp/logs/access_log",
      "/opt/lampp/logs/error_log"
    ]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    exit_after_read => "true"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
    stat_interval => "5 second"
  }
}

filter {
  if [path] == "/var/log/syslog" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" }
    }
    date {
      match => [ "log_timestamp", "ISO8601" ]
      target => "@timestamp"
      remove_field => ["log_timestamp"]
    }
  }

  if [path] == "/var/log/kern.log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" }
    }
    date {
      match => [ "log_timestamp", "ISO8601" ]
      target => "@timestamp"
      remove_field => ["log_timestamp"]
    }
  }

  if [path] == "/var/log/auth.log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" }
    }
    date {
      match => [ "log_timestamp", "ISO8601" ]
      target => "@timestamp"
      remove_field => ["log_timestamp"]
    }
  }

  if [path] == "/opt/lampp/logs/access_log" {
    grok {
      match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \\\"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
      remove_field => ["clientip", "ident", "auth", "method", "request", "httpversion", "response", "bytes"]
    }

    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "Asia/Beirut"
      target => "@timestamp"
    }

    mutate {
      remove_field => ["timestamp"]
    }
  }

  if [path] == "/opt/lampp/logs/error_log" {
    grok {
      match => { "message" => "\\[%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}\\] \\[%{DATA:loglevel}\\] \\[pid %{NUMBER:pid}\\] %{GREEDYDATA:rest}" }
    }

    mutate {
      add_field => { "timestamp" => "%{day} %{month} %{monthday} %{time} %{year}" }
    }

    date {
      match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ]
      timezone => "Asia/Beirut"
      target => "@timestamp"
    }

    mutate {
      remove_field => ["day", "month", "monthday", "time", "year", "timestamp"]
    }
  }
}

output {
  elasticsearch {
    index => "loginf"
    hosts => ["http://manticore:9308"]
    ilm_enabled => false
    manage_template => false
  }
  stdout { codec => rubydebug }
}
EOL

docker run --network=t_network --rm -it -e EXTRA=1 --platform linux/amd64 --name manticore manticoresearch/manticore:6.3.0

docker run -e PIPELINE_BATCH_SIZE=10000 \
-e XPACK_MONITORING_ENABLED=false \
-e PIPELINE_WORKERS=8 \
--network=t_network --rm -it \
-v /Users/pavelshilin/Downloads/Logs_New/var:/var/log/ \
-v /Users/pavelshilin/Downloads/Logs_New/opt:/opt/lampp/logs/ \
-v /tmp/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
docker.elastic.co/logstash/logstash:7.10.0

mysql> DESCRIBE loginf;
+------------+-----------+----------------+
| Field      | Type      | Properties     |
+------------+-----------+----------------+
| id         | bigint    |                |
| rest       | text      | indexed stored |
| loglevel   | text      | indexed stored |
| message    | text      | indexed stored |
| pid        | text      | indexed stored |
| path       | text      | indexed stored |
| host       | text      | indexed stored |
| @version   | text      | indexed stored |
| @timestamp | timestamp |                |
| tags       | json      |                |
+------------+-----------+----------------+
mysql> select count(*) from loginf;
+----------+
| count(*) |
+----------+
|   193225 |
+----------+

The data after indexing is identical to that contained in your files.

❯ cat /Users/pavelshilin/Downloads/Logs_New/var/auth.log | wc -l
    3252
❯ cat /Users/pavelshilin/Downloads/Logs_New/var/kern.log | wc -l
   29448
❯ cat /Users/pavelshilin/Downloads/Logs_New/var/syslog | wc -l
  156592
❯ cat /Users/pavelshilin/Downloads/Logs_New/opt/access_log | wc -l
    3821
❯ cat /Users/pavelshilin/Downloads/Logs_New/opt/error_log | wc -l
     112

Total 193225

[tomatolog]

if this helps it could be better to create another ticket to provide Elastic comparable reply in case incoming packet overflows the searchd limit

[haninerabahfawaz]

Thank you @PavelShilin89 Please, I want to ask if the Manticore index will be updated, with this configuration, and if any new records are added without any repetition in the records data.

…

On Tue, 9 Jul 2024 at 14:57, Pavel Shilin ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> try adding max_packet_size = 128M to manticore.conf. This way I was able to fully index all the data without loss: docker network create t_network cat <<EOL > /tmp/manticore.conf searchd { listen = 0.0.0.0:9312 listen = 0.0.0.0:9306:mysql listen = 0.0.0.0:9308:http log = /var/log/manticore/searchd.log query_log = /var/log/manticore/query.log pid_file = /var/run/manticore/searchd.pid data_dir = /var/lib/manticore max_packet_size = 128M } EOL cat <<EOL > /tmp/logstash.conf input { file { path => [ "/var/log/syslog", "/var/log/kern.log", "/var/log/auth.log", "/opt/lampp/logs/access_log", "/opt/lampp/logs/error_log" ] start_position => "beginning" sincedb_path => "/dev/null" mode => "read" exit_after_read => "true" file_completed_action => "log" file_completed_log_path => "/dev/null" stat_interval => "5 second" } } filter { if [path] == "/var/log/syslog" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/var/log/kern.log" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/var/log/auth.log" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA}" } } date { match => [ "log_timestamp", "ISO8601" ] target => ***@***.***" remove_field => ["log_timestamp"] } } if [path] == "/opt/lampp/logs/access_log" { grok { match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \\\"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } remove_field => ["clientip", "ident", "auth", "method", "request", "httpversion", "response", "bytes"] } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] timezone => "Asia/Beirut" target => ***@***.***" } mutate { remove_field => ["timestamp"] } } if [path] == "/opt/lampp/logs/error_log" { grok { match => { "message" => "\\[%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}\\] \\[%{DATA:loglevel}\\] \\[pid %{NUMBER:pid}\\] %{GREEDYDATA:rest}" } } mutate { add_field => { "timestamp" => "%{day} %{month} %{monthday} %{time} %{year}" } } date { match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] timezone => "Asia/Beirut" target => ***@***.***" } mutate { remove_field => ["day", "month", "monthday", "time", "year", "timestamp"] } } } output { elasticsearch { index => "loginf" hosts => ["http://manticore:9308"] ilm_enabled => false manage_template => false } stdout { codec => rubydebug } } EOL docker run --network=t_network --rm -it -e EXTRA=1 --platform linux/amd64 --name manticore manticoresearch/manticore:6.3.0 docker run -e PIPELINE_BATCH_SIZE=10000 \ -e XPACK_MONITORING_ENABLED=false \ -e PIPELINE_WORKERS=8 \ --network=t_network --rm -it \ -v /Users/pavelshilin/Downloads/Logs_New/var:/var/log/ \ -v /Users/pavelshilin/Downloads/Logs_New/opt:/opt/lampp/logs/ \ -v /tmp/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \docker.elastic.co/logstash/logstash:7.10.0 mysql> DESCRIBE loginf; +------------+-----------+----------------+ | Field | Type | Properties | +------------+-----------+----------------+ | id | bigint | | | rest | text | indexed stored | | loglevel | text | indexed stored | | message | text | indexed stored | | pid | text | indexed stored | | path | text | indexed stored | | host | text | indexed stored | | @Version | text | indexed stored | | @timestamp | timestamp | | | tags | json | | +------------+-----------+----------------+ mysql> select count(*) from loginf; +----------+ | count(*) | +----------+ | 193225 | +----------+ The data after indexing is identical to that contained in your files. ❯ cat /Users/pavelshilin/Downloads/Logs_New/var/auth.log | wc -l 3252 ❯ cat /Users/pavelshilin/Downloads/Logs_New/var/kern.log | wc -l 29448 ❯ cat /Users/pavelshilin/Downloads/Logs_New/var/syslog | wc -l 156592 ❯ cat /Users/pavelshilin/Downloads/Logs_New/opt/access_log | wc -l 3821 ❯ cat /Users/pavelshilin/Downloads/Logs_New/opt/error_log | wc -l 112 Total 193225 — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDA4SUBKD6VJR6MHBYLZLPFZNAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJXGQ2TAOJYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tomatolog]

it is still not clear how do you count for the repetition - it could be better to provide complete case your configuration (we got logstash config but not the daemon config) And initial guess with daemon config shows that searchd.max_packet_size = 128M should be set for logstash to finish well.

Now you said about repetition but that output is matched to the lines in the log files you provided

mysql> select count(*) from loginf;
+----------+
| count(*) |
+----------+
|   193225 |
+----------+

that is why it is not clear what you mean while you said about repetition in the records data.

It could be better to provide complete example, ie

• logstash configs
• all log files these posted into logstash
• daemon config
• queries you issue to get the data statistics along with queries replies

that allow us to run the case on the empty instance and compare the output of the query you posted with replies after logstash posts data and check the differences,

[haninerabahfawaz]

@tomatolog

That's not what I mean.

If you always know that we have new data in the log file.

My task or goal is after running logstash configuration then if any new data is added in any input file then this new data should also be added in the table to create manticore the index is always updated.

[tomatolog]

I do not use logstash but all your examples use sincedb_path => "/dev/null" and documentation said there logstash keeps track of the current position of monitored log files. Maybe better to consult the logstash forum to get the configuration that works as you need?

[sanikolaev]

@haninerabahfawaz I agree sincedb_path => "/dev/null" may be not good if you want to put new data to Manticore (or Elasticsearch) as soon as it's updated in the file. Pls experiment with it.

@PavelShilin89 pls try reproducing the same issue with Elasticsearch (by lowering it's http.max_content_length setting). It's interesting what Elasticsearch and Logstash return in this case and if we can improve our error message so it makes more sense (so it's clear that the user needs to increase max_packet_size)

[PavelShilin89]

@haninerabahfawazСогласен, sincedb_path => "/dev/null"это может быть не очень хорошо, если вы хотите помещать новые данные в Manticore (или Elasticsearch) сразу после их обновления в файле. Пожалуйста, поэкспериментируйте с этим.

@PavelShilin89Пожалуйста, попробуйте воспроизвести ту же проблему с Elasticsearch (уменьшив его http.max_content_length настройку ). Интересно, что Elasticsearch и Logstash возвращают в этом случае, и можем ли мы улучшить наше сообщение об ошибке, чтобы оно имело больше смысла (чтобы было ясно, что пользователю нужно увеличить max_packet_size)

When reproducing the same problem with Elasticsearch by reducing its http.max_content_length we also don't get a clearer error or notification.

The Elasticsearch logs only ara warnings about low disk level and JVM rubbish collection, which may indicate possible performance issues.
- Deprecation warning that using types in bulk queries is deprecated.

There is an error in Logstash logs:

[2024-07-10T09:41:39,215][ERROR][logstash.outputs.elasticsearch][main][7d2347ba5a0b22293324ccaa54fff4d7d4657d43825b29f6901128b6a93a70ac] Encountered a retryable error. Will Retry with exponential backoff {:code=>413, :url=>"http://es1:9200/_bulk"}

[haninerabahfawaz]

@sanikolaev @tomatolog @PavelShilin89
yes, we can use sincedb_path => "/var/lib/logstash/since_db" .

sincedb_path is used to keep the database in memory across reboots. This means that if you set cedb_path to NUL (on Windows, or /dev/null on UNIX), when you restart logstash, it will think about each new file and reread it.

My question:
After modifying sincedb_path , if any records are added in any input log file, they should be added to the index automatically...

Example:
I run the command and let's say the number of records is 100000. Then new records are added in the input files, let's say 3000 new records, they should be added automatically….

[sanikolaev]

After modifying sincedb_path , if any records are added in any input log file, they should be added to the index automatically...

I think so. What's the question? Does it happen with Elasticsearch, but not with Manticore?

[haninerabahfawaz]

@sanikolaev
No, I don't mean that...

You know that logs are dynamic files, not static, which means that everything on your computer is added to the log files.
This means that the index table must always be updated.

We need to add something in the logstash file that always updates the index file if any log is added....

[sanikolaev]

We need to add something in the logstash file that always updates the index file if any log is added

As I understand sincedb_path does that, doesn't it? Anyway it looks like you are currently struggling with a logstash issue, not Manticore's, right? In this case it's better to discuss it with Logstash experts.

[haninerabahfawaz]

@sanikolaev
sincedb_path updates the read position of log files only.
The goal is to always update the index table in manticore.
Why ?
Because the input files are always modified.

[haninerabahfawaz]

@sanikolaev @PavelShilin89

I always get this problem in my system log:

2024-07-15T17:51:37.423966+03:00 hanine-VirtualBox logstash[3445]: [2024-07-15T17:51:37,423][ERROR][logstash.outputs.elasticsearch][main][4a1440c56bbde552634034a0198050029a19e1012f1897d5b1d8681720180424] Encountered a retryable error. Will Retry with exponential backoff {:code=>409, :url=>"http://localhost:9308/_bulk"}

[PavelShilin89]

@haninerabahfawaz Is your goal to update the old value in the log file and update it in Manticore Search?

[haninerabahfawaz]

@PavelShilin89 No, new ones only. The first time, it will read all files. And the second time, because I want to keep the manticore always up to date, new ones will be read….

…

On Tue, 16 Jul 2024, 12:14 pm Pavel Shilin, ***@***.***> wrote: @haninerabahfawaz <https://github.com/haninerabahfawaz> Is your goal to update the old value in the log file and update it in Manticore Search? — Reply to this email directly, view it on GitHub <#2307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A33WFDGLFXVBOTCCZWS5SODZMTP5RAVCNFSM6AAAAABJI7H24KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZQGQYDSMJWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sanikolaev]

Here's an example of integrating Logstash with Manticore, so as soon as a new line is added to the log file, it's immediately inserted into Manticore by Logstash:

Execute this in Ubuntu Jammy:

mkdir /usr/share/logstash && cd $_

wget -q https://artifacts.elastic.co/downloads/logstash/logstash-7.10.0-linux-x86_64.tar.gz

tar -xvzf logstash-7.10.0-linux-x86_64.tar.gz

cd /tmp/
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt -y update
sudo apt -y install manticore manticore-extra

cd /usr/local/src

ln -s /usr/share/logstash/logstash-7.10.0/bin/logstash /usr/bin/logstash

echo 'input { file { path => ["/var/log/dpkg.log"] start_position => "end" sincedb_path => "/tmp/sincedb_path" mode => "tail" } } output { elasticsearch { index => "dpkg_log" hosts => ["http://localhost:9308"] ilm_enabled => false manage_template => false } }' > logstash.conf

systemctl start manticore

logstash -f logstash.conf &

# Wait until Logstash starts

echo -e "2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1" >> /var/log/dpkg.log

mysql -NB -P9306 -h0 -e "select count(*) from dpkg_log"

# You'll get:
# 1

echo -e "2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n" >> /var/log/dpkg.log

mysql -NB -P9306 -h0 -e "select count(*) from dpkg_log"

# You'll get:
# 2

# and so on

I see no problem. If you see any issue please modify this example to demonstrate it.