sematext/logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint

- read more

What is Logagent

Logagent is a modern, open-source, light-weight log shipper. It is like Filebeat and Logstash in one, without the JVM memory footprint. It comes with out of the box and extensible log parsing, on-disk buffering, secure transport, and bulk indexing to Elasticsearch, Sematext Logs, and other destinations. Its low memory footprint and low CPU overhead make it suitable for deploying on edge nodes and devices, while its ability to parse and structure logs makes it a great Logstash alternative.

Docker

Details about the the Logagent Docker image are described in the Docker Hub Readme

Installation

1) Install Node.js

Official Node.js downloads and instructions. E.g. for Debian/Ubuntu:

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt-get install -y nodejs
Install Logagent with npm
sudo npm i -g @sematext/logagent

2) Run logagent command line tool

logagent --help

3) Example: Index your log files in Elasticsearch

logagent -e http://localhost:9200 -i logs -g ‘/var/log/**/*.log’

4) Optional: Install service & config

Install service for Logagent using systemd, upstart, launchd To quickly create a config file for indexing into Elasticsearch without having to edit it run something like this:

sudo logagent-setup -u http://localhost:9200 -i INDEX_NAME -g '/var/log/**/*.log'
# Logsene US: use -u https://logsene-receiver.sematext.com and your Logsene App Token as index name.
# Logsene EU: use -u https://logsene-receiver.eu.sematext.com and your Logsene App Token as index name.

Configuration

To configure different inputs, different event processing, or different outputs (e.g. your own Elasticsearch) edit /etc/sematext/logagent.conf, e.g.:

output:
  logsene:
    module: elasticsearch
    url: http://elasticsearch-server:9200
    index: logs

Then restart the service with sudo service logagent restart. Troubleshooting & Logs Logagent’s own logs:

• Upstart: /var/log/upstart/logagent.log
• Systemd: journalctl -u logagent
• Launchd: /Library/Logs/logagent.log

Location of service scripts:

• Upstart: /etc/init/logagent.conf
• Systemd: /etc/systemd/system/logagent.service
• Launchd: /Library/LaunchDaemons/com.sematext.logagent.plist

Start/stop service:

• Upstart: service logagent stop/start
• Systemd: systemctl stop/start logagent
• Launchd: launchctl start/stop com.sematext.logagent

Documentation & Support

• Full documentation
• Logagent main page
• Logagent on Github
• Forum: https://groups.google.com/forum/#!forum/logagent
• Twitter: @sematext
• Blog: sematext.com/blog

Development

• Update to the last node version
• From root folder node type: node ./bin/logagent -h
• To test from root folder type: node test

logagent-js's Issues

Implement Dead Letter Queue

A dead letter queue (DLQ) could be like the existing disk buffer, but it would really be dead -- write+delete only, and never re-read.

To re-process DLQ one would:

• stop Logagent
• move DLQ to the buffer location/dir
• optionally modify DLQ content if that's needed
• start Logagent

Like the disk buffer, the DLQ should have a configurable max size and should start permanently removing the oldest entries once that max size is reached.

Specify document _id

I like to define document ids so that I can reprocess logs without adding duplicates. When I define an _id field, it gets renamed to id. Not sure yet if logagent-js is doing this or logsene-js.

Implement JMX input

Maybe via https://www.npmjs.com/package/jmx (but didn't look into options deeply), though I see this depends on some node-java package and that node-java README mentions Python...

Add JSLint

... or something else to help us keep the code style and formatting consistent.

License clarity

Not sure if this is cruft from a copy & paste or if intended for sematext customers only:

 * This source code may not be copied, reverse engineered, or altered for any purpose.
 * This source code is to be used exclusively by users and customers of Sematext.
 * Please see the full license (found in LICENSE in this distribution) for details on its license and the licenses of its dependencies.

Are non-sematext customers licensed to use this? Thanks!

Disk buffer should be a ring-buffer

Logagent buffers logs (by default 10000 bulk requests with a max. size of 1000 logs) on disk when shipping via HTTPS/HTTP fails. If such a problem remains longer, logagent would stop to buffer logs on disk. I think most users would prefer to drop the oldest file to keep most recent logs in the (always limited) disk buffer.

Current workarounds:

1. If disk space is not an issue, one could increase the limit to a higher value e.g. 50k bulk files.
   export LOGSENE_MAX_STORED_REQUESTS=50000
2. Run a chron job to delete oldest files form LOGSENE_TMP_DIR. Some potential solutions are here https://stackoverflow.com/questions/2960022/shell-script-to-count-files-then-remove-oldest-files

See https://github.com/sematext/logsene-js#environment-variables for further details of logsene-js settings.

Add support for multi-line logs

e.g. Java exception stack traces

Inspiration for how (not to) handle this could be taken from Logstash or rsyslog.

Refactoring / directory structure

New directory layout:

- bin # executables
- lib/core # framework for logagent service 
- lib/plugins # plugins for input/output
- lib/plugins/input
- lib/plugin/output
- lib/util # helper functions
- lib/parser # parser / regex matching

stdin duplicated

While testing, it took me a while to figure out why I was seeing duplicates. Not sure if user error on my part, poor usage text, or a bug. Here is the part of the code that causes duplicates. Leaving the --stdin flag off produced expected behavior. Maybe best to remove the --stdin option?

Consider hot-reloading pattern files

Maybe check if the file was updated every 60 seconds and reload it only if it was indeed updated.

I was going to say this will help during pattern creation time, because one won't have to restart logagent after changing the patterns, but in that case 60 seconds would be too long to wait, so maybe that frequency should be configurable, so one can set it to e.g. 60s or even 300s for production, and 2s for dev.

Feature: Make logagent work with Solr v4 log

I tried using it on Solr v4.10.4 log and it freezes logagent.

I used the following command:

logagent -t $MBO_CLI_TOKEN -y short-sample-solr.log

Is that correct syntax?

Here's a sample of the file I used:

WARN  - 2015-03-26 15:14:25.096; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:25.268; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:25.556; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:25.954; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.046; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.287; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.366; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.574; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.680; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.741; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.819; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:26.965; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:27.238; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:27.271; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:27.576; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.141; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.214; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.275; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.595; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.658; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.703; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:29.728; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:30.046; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:30.232; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:30.502; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:30.749; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:31.138; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:31.701; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:31.995; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:32.556; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:33.027; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:33.140; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:33.618; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:35.215; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:35.559; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:35.948; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:36.358; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:37.619; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:37.944; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:38.635; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:38.904; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:39.054; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:39.243; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:39.709; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:42.110; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:43.500; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:43.885; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:44.735; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.052; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.321; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.383; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.547; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.728; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:45.965; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:46.835; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:47.058; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:47.579; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:47.652; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:47.763; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:47.973; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:48.215; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:48.397; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:48.469; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:48.777; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:49.295; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:49.336; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:49.379; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:49.425; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:49.960; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:50.467; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:50.509; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:50.592; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:50.633; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:51.430; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:53.090; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:53.518; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:53.551; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:53.613; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:14:53.716; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
ERROR - 2015-03-26 15:15:58.807; org.apache.solr.common.SolrException; java.lang.IllegalStateException: field "pagetext" was indexed without position data; cannot run PhraseQuery (term=inov)
    at org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277)
    at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
    at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
    at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
    at org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
    at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:209)
    at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1712)
    at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1429)
    at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:514)
    at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:484)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)

ERROR - 2015-03-26 15:15:58.808; org.apache.solr.common.SolrException; null:java.lang.IllegalStateException: field "pagetext" was indexed without position data; cannot run PhraseQuery (term=inov)
    at org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277)
    at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
    at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
    at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
    at org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
    at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:209)
    at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1712)
    at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1429)
    at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:514)
    at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:484)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)

ERROR - 2015-03-26 15:15:58.813; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://172.16.0.144:8081/solr/forum, http://172.16.0.143:8081/solr/forum, http://172.16.0.141:8081/solr/forum, http://172.16.0.141:8080/solr/forum]
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://172.16.0.144:8081/solr/forum, http://172.16.0.143:8081/solr/forum, http://172.16.0.141:8081/solr/forum, http://172.16.0.141:8080/solr/forum]
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324)
    at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:223)
    at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
    at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: field "pagetext" was indexed without position data; cannot run PhraseQuery (term=inov)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:345)
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:306)
    ... 9 more

ERROR - 2015-03-26 15:15:58.815; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://172.16.0.144:8081/solr/forum, http://172.16.0.143:8081/solr/forum, http://172.16.0.141:8081/solr/forum, http://172.16.0.141:8080/solr/forum]
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://172.16.0.144:8081/solr/forum, http://172.16.0.143:8081/solr/forum, http://172.16.0.141:8081/solr/forum, http://172.16.0.141:8080/solr/forum]
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324)
    at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:223)
    at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
    at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: field "pagetext" was indexed without position data; cannot run PhraseQuery (term=inov)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
    at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:345)
    at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:306)
    ... 9 more

WARN  - 2015-03-26 15:16:53.647; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:53.806; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.207; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.380; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.398; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.643; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.724; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:54.834; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:16:57.058; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:10.931; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:11.245; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:11.377; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:12.282; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:12.572; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:12.990; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:14.237; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:14.523; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:14.593; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
ERROR - 2015-03-26 15:18:14.742; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:204)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:155)
    at org.apache.solr.search.DisMaxQParser.getUserQuery(DisMaxQParser.java:239)
    at org.apache.solr.search.DisMaxQParser.addMainQuery(DisMaxQParser.java:199)
    at org.apache.solr.search.DisMaxQParser.parse(DisMaxQParser.java:110)
    at org.apache.solr.search.QParser.getQuery(QParser.java:141)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
    ... 31 more
Caused by: org.apache.solr.parser.ParseException: Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.QueryParser.generateParseException(QueryParser.java:649)
    at org.apache.solr.parser.QueryParser.jj_consume_token(QueryParser.java:531)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:216)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
    ... 36 more

WARN  - 2015-03-26 15:18:15.344; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:15.390; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:15.515; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:15.735; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:15.829; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:16.080; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:16.292; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:18.098; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:18.412; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:19.810; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:21.225; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:22.049; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:22.793; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:23.073; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.233; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.262; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.449; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.565; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.615; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.798; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:24.973; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:25.678; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:26.481; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:27.567; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:28.666; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:29.124; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:29.533; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
ERROR - 2015-03-26 15:18:29.908; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:204)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:155)
    at org.apache.solr.search.DisMaxQParser.getUserQuery(DisMaxQParser.java:239)
    at org.apache.solr.search.DisMaxQParser.addMainQuery(DisMaxQParser.java:199)
    at org.apache.solr.search.DisMaxQParser.parse(DisMaxQParser.java:110)
    at org.apache.solr.search.QParser.getQuery(QParser.java:141)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
    ... 31 more
Caused by: org.apache.solr.parser.ParseException: Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.QueryParser.generateParseException(QueryParser.java:649)
    at org.apache.solr.parser.QueryParser.jj_consume_token(QueryParser.java:531)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:216)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
    ... 36 more

WARN  - 2015-03-26 15:18:30.037; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:30.728; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:30.853; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:30.896; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:31.017; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:31.166; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:31.223; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:31.294; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:31.619; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
ERROR - 2015-03-26 15:18:31.784; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:204)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:155)
    at org.apache.solr.search.DisMaxQParser.getUserQuery(DisMaxQParser.java:239)
    at org.apache.solr.search.DisMaxQParser.addMainQuery(DisMaxQParser.java:199)
    at org.apache.solr.search.DisMaxQParser.parse(DisMaxQParser.java:110)
    at org.apache.solr.search.QParser.getQuery(QParser.java:141)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
    ... 31 more
Caused by: org.apache.solr.parser.ParseException: Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.QueryParser.generateParseException(QueryParser.java:649)
    at org.apache.solr.parser.QueryParser.jj_consume_token(QueryParser.java:531)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:216)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
    ... 36 more

WARN  - 2015-03-26 15:18:31.850; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:32.416; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:32.547; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:33.227; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:33.592; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:35.648; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:35.693; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:35.742; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:35.896; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:36.129; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
ERROR - 2015-03-26 15:18:36.207; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:204)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.search.SyntaxError: Cannot parse '+': Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:155)
    at org.apache.solr.search.DisMaxQParser.getUserQuery(DisMaxQParser.java:239)
    at org.apache.solr.search.DisMaxQParser.addMainQuery(DisMaxQParser.java:199)
    at org.apache.solr.search.DisMaxQParser.parse(DisMaxQParser.java:110)
    at org.apache.solr.search.QParser.getQuery(QParser.java:141)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
    ... 31 more
Caused by: org.apache.solr.parser.ParseException: Encountered "<EOF>" at line 1, column 1.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...
    <TERM> ...
    "*" ...

    at org.apache.solr.parser.QueryParser.generateParseException(QueryParser.java:649)
    at org.apache.solr.parser.QueryParser.jj_consume_token(QueryParser.java:531)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:216)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
    ... 36 more

WARN  - 2015-03-26 15:18:36.602; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups
WARN  - 2015-03-26 15:18:36.640; org.apache.solr.handler.loader.XMLLoader; XML element <add> has invalid XML attr: allowDups

API for filtering / dropping?

I can't find anything in the docs or code to show how to drop a message. Is that supported? If not and something you'd like to support, any thoughts on how you'd like to support that functionality? Some options that come to mind:

• Allow transform functions to declare the message as filtered (eg parsed.filter(bool))?
• Support something similar to transform (eg filtered) that accepts a function that should return true if message should be dropped?

Elasticsearch support

Thanks at first.

Ok, elasticsearch is supported, but any example? who i do output to my private elascticsearch sever?

TIA

Make it possible to set max log size

Maybe we should have --max-log-size

Maybe we should also have an option that lets one control what happens with logs that are bigger than --max-log-size.
Such logs could:

• just get dropped, or
• get trimmed to --max-log-size

I assume we should log such situations, so one can see when they have this problem.

Logagent-js should have on-disk buffering

spm-agent has:

• Buffering metrics to disk in case of network outages (using NeDB)
• Limit the file size of buffers

logagent-js should have that, too.

Pull disk buffer into Logagent from logsene-js

The disk buffer should be in core Logagent, so all outputs can use it, not just Elasticsearch one.

Plugin to query systemd/journald logs

The plugin should store the timestamp of the last query to make sure it does not loose logs during restarts.

Bunyan format is not recognized, when version = 0 (v:0)

Rework logagent.conf to be a real config file

The logagent.conf is not really a config currently. It's just a place where all command-line parameters can be stored. This should probably be changed, so logagent.conf looks like a typical config file.

Implement (Exponential) Backoff

Logagent outputs should have (or let one enable) exponential/incremental backoff in order to avoid hammering the output target too much.

See https://en.wikipedia.org/wiki/Exponential_backoff
https://www.npmjs.com/package/backoff
https://www.npmjs.com/package/oibackoff

Implement bandwidth throttling

If input (e.g. log file) volume goes crazy (e.g. due to a bug) Logagent will of course try to read the input and ship the data as fast as it can. That could have a negative impact on network, CPU, etc.
Maybe now, when working on #28 it makes sense to think about throttling because it probably makes the most sense for throttling to happens as close to input as possible (e.g., don't even read data from input is specified max input bytes per second limit has been reached).

Add pluggable processors, inputFilters, and outputFilters

Now that Logagent has input and output plugins it should also have the notion of pluggable "processors" to create this sort of a pipeline:

input | inputFilter | processors | outputFilter | output

...where inputFilter and outputFilter are just types of a processors that filter data (e.g. drop it instead of doing something with it, like #40 might do, for example)

Implement comma/tab delimited input

Things that should be configurable:

• the delimiter character
• skipping of the very first line in a file, to avoid reading the headings that such files sometimes have and using it to map values to field names
• default value to use for an empty field

multi-line logs without lines being indented by spaces?

I have a log that looks like this (simplified):

2016-12-08 10:09:16 - [ERROR] - from application - 
! @72aki4439 - Internal server error, for (POST) [/rest/v1/item] ->
play.api.Application$$anon$1: Execution exception[[NullPointerException: null]]
		at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.11-2.3.8.jar:2.3.8]
		at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.11-2.3.8.jar:2.3.8]
Caused by: java.lang.NullPointerException: null
		at controllers.RestItem$2.apply(RestItem.java:577) ~[bla-1.0.jar:1.0]

This is one single log entry, but the lines with !, play.api.Application, and Caused by are not indented with spaces. I tried all values for regex and blockStart that I could think of, but I couldn't get this recognized as a single entry. Is there a way to do this or will I need to change the log format (which isn't trivial, as far as I understand)?

Implement RDBMS input

Allow one to provide one or more SQL statements and their execution schedule.

Parsing a date error

I’m trying to parse Fastly’s web logs. Here is an example log line:
107.77.111.34 "-" "-" Fri, 08 Jan 2016 08:59:07 GMT "POST /test" 200 25 "http://www.example.com/articles/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75 [FBAN/FBIOS;FBAV/46.0.0.54.156;FBBV/18972819;FBDV/iPhone7,2;FBMD/iPhone;FBSN/iPhone OS;FBSV/9.2;FBSS/2; FBCR/AT&T;FBID/phone;FBLC/en_US;FBOP/5]”

Here is my pattern:

• Fastly

  sourceName: !!js/regexp /fastly/
  match:
  • type: fastly
    regex: !!js/regexp /(^[0-9a-f.:]+)\s+"(-|.+?)"\s+"(-|.+?)"\s+([a-z]{3}, [0-9]{2} [a-z]{3} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}[^]]*) "(\S+?)\s{0,1}(\S+?)" ([0-9|-]+) ([0-9|-]+)/i
    fields: [client_ip,remote_id,user,ts,method,path,status_code,size]
    dateFormat: ddd, dd MMM YYYY HH:mm:ss zzz
    transform: !!js/function >
    function (p) {
    p.message = p.method + ' ' + p.path
    }

This seems to parse correctly, except for the date:
originalLine: 107.77.111.34 "-" "-" Fri, 08 Jan 2016 08:59:07 GMT "POST /test" 200 25 "http://www.example.com/articles/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75 [FBAN/FBIOS;FBAV/46.0.0.54.156;FBBV/18972819;FBDV/iPhone7,2;FBMD/iPhone;FBSN/iPhone OS;FBSV/9.2;FBSS/2; FBCR/AT&T;FBID/phone;FBLC/en_US;FBOP/5]"
_type: fastly
client_ip: 107.77.111.34
remote_id: -
user: -
ts: Fri, 08 Jan 2016 08:59:07 GMT
method: POST
path: /test
status_code: 200
size: 25
@timestamp: Invalid Date
message: POST /test

Is "ddd, dd MMM YYYY HH:mm:ss zzz" the right syntax for a date in this format: "Fri, 08 Jan 2016 08:59:07 GMT"

Support multiple tokens for tail-file mode

Currently the logagent-js setup running as Linux service supports only one Logsene token/elasticserach index. Logs are stored with differten _type in Logsne/Elasticsearch, but it would be helpful to map file names and tokens somehow to send different logs to separate indices.

Logagent CPU usage growing

$ npm view logagent-js version
1.2.30

If you hide everything except "nice", I think that is mostly because of Logagent:
https://apps.sematext.com/spm-reports/s/05RV4w2xMG
I do have a few other processes that use nice 19, like Logagent process, but top tells me Logagent is spending more time using the CPU than these other processes...

Move recent bugfixes and enhancements from 1.x branch to 2x (master)

This applies to:

• #26
• #25
• #22 (TBC)
• #20 (TBC)
• sematext/sematext-agent-docker#12 commit: 3361aa0
• Add uninstall script: 1c84dba
• GeoIP fix #34

Add tests for GeoIP functionality

This functionality depends on MaxMind DB and they seem to like changing what their functions return, so tests are needed to catch that.

Implement Elasticsearch Index API

So one can easily feed Logagent from other data sources that can output to Elasticsearch.

Support Kafka input and output

Plugins for writing to and reading from Kafka should be useful for many setups that have Kafka as a central queue.

One caveat to mention here: reading from Kafka should be synchronous, IMO. That is, ACK the message after it's being successfully sent to the destination. Otherwise, if we ACK on successful read (enqueue), then we simply transfer the messages from Kafka to Logagent's internal queue, which is probably not what we want. One should probably use multiple consumers (Logagent instances? Pipelines within Logagent?) to send data from Kafka to multiple destinations, if we want multiple subscribers.

Add "recipes" for masking common sensitive data

Things like:

• date of birth
• social security numbers
• phone numbers
• credit card numbers

... so it's easy for new users to enable this, get it out of the box, and also add their own patterns, and so that it's easy to point to this from documentation.

Stored file position contains "NaN" / invalid position.

Spawning in the background and doing a kill -15 PID causes null to be stored as the position:

2016-07-20T00:01:30.603Z Stop watching /path/to/some-file inode: 147307 pos:NaN

Also, would be nice to have positions stored on SIGINT as well.

replace tail-forever module

Tail-forever does not provide streams to limit input rate.

No output to rTail

using the following command and a rtail server at 9091 I'm seeing no output to the server.

logagent -sv --rtailHost 127.0.0.1 --rtailPort 9091 /var/log/*.log

logagent output:

2016-10-17T08:17:28.156Z pid[27748] init plugins 2016-10-17T08:17:28.166Z pid[27748] init plugins 2016-10-17T08:17:28.168Z pid[27748] init plugins 2016-10-17T08:17:28.168Z pid[27748] ../lib/plugins/input/stdin 2016-10-17T08:17:28.205Z pid[27748] ../lib/plugins/output/stdout 2016-10-17T08:17:28.236Z pid[27748] ../lib/plugins/input/files 2016-10-17T08:17:28.454Z pid[27748] Error reading file pointers: Error: ENOENT: no such file or directory, open '/var/folders/lp/tk08lrbj2nd3bg9m5ys7jfwc0000gn/T/logagentTailPointers.json' at Error (native) at Object.fs.openSync (fs.js:549:18) at Object.fs.readFileSync (fs.js:397:15) at InputFile.readFilePointers (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/input/files.js:195:17) at InputFile.start (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/input/files.js:55:30) at LaCli.<anonymous> (/usr/local/lib/node_modules/@sematext/logagent/bin/logagent.js:78:9) at Array.forEach (native) at LaCli.initPugins (/usr/local/lib/node_modules/@sematext/logagent/bin/logagent.js:72:11) at LaCli.initState (/usr/local/lib/node_modules/@sematext/logagent/bin/logagent.js:162:8) at new LaCli (/usr/local/lib/node_modules/@sematext/logagent/bin/logagent.js:41:8) 2016-10-17T08:17:28.492Z pid[27748] Watching file:/var/log/accountpolicy.log from position: 96732 2016-10-17T08:17:28.501Z pid[27748] Watching file:/var/log/authd.log from position: 9526 2016-10-17T08:17:28.502Z pid[27748] Watching file:/var/log/commerce.log from position: 22759235 2016-10-17T08:17:28.502Z pid[27748] Watching file:/var/log/corecaptured.log from position: 155289 2016-10-17T08:17:28.503Z pid[27748] Watching file:/var/log/coreduetd.log from position: 0 2016-10-17T08:17:28.503Z pid[27748] Watching file:/var/log/displaypolicyd.log from position: 21377 2016-10-17T08:17:28.503Z pid[27748] Watching file:/var/log/displaypolicyd.stdout.log from position: 0 2016-10-17T08:17:28.504Z pid[27748] Watching file:/var/log/fsck_hfs.log from position: 647052 2016-10-17T08:17:28.504Z pid[27748] Watching file:/var/log/install.log from position: 5018483 2016-10-17T08:17:28.504Z pid[27748] Watching file:/var/log/opendirectoryd.log from position: 3638 2016-10-17T08:17:28.504Z pid[27748] Watching file:/var/log/secinitd.log from position: 408 2016-10-17T08:17:28.505Z pid[27748] Watching file:/var/log/system.log from position: 3382822 2016-10-17T08:17:28.505Z pid[27748] Watching file:/var/log/wifi.log from position: 1677 2016-10-17T08:17:28.684Z pid[27748] Logagent report: pid[27748] 722 ms 0 lines parsed. 0 lines/s 0.000 MB/s - empty lines: 0 2016-10-17T08:17:28.684Z pid[27748] Logagent stats: usedTokens=0 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0 2016-10-17T08:17:28.684Z pid[27748] Memory stats: heapUsedMB=14 heapTotalMB=30 memoryRssMB=40

This is running on OSX 10.11

Read file from beginning

I'd like to be able to start from beginning of file when there is no stored position.

Make Logagent work with Node 7.4

Tried upgrading node to 7.4 yesterday and Logagent failed to start because of missing deps...

Json file input (bunyan) and transform

How i transform or filter json log file (v.g. bunyan)?

Thanks.

Come up with the framework for integration tests

So correctness can be tested end to end.

Add verbose mode/flag

Maybe it can be just an alias for print_stats if there is nothing other than stats a verbose mode could print.

Support plugins. Decouple functionalities by an event driven architecture

In logagent 1.x the CLI/server functionalities are tied together in a large script. In 2.x we started to split it into multiple files. The goal is to have plugins, like Etsy statsd backends, which could be installed as separate npm modules to extend the functionality.

How to do this?

• Introduce a central event queue with topics and pub/sub (a bit like Kafka, but in-process).
  eventemitter2 seems to be a good package for this approach. So plugins could work together without having dependencies to each other. This should be wrapped in a separate "class", so we could support in the future multi-process/IPC plugins by exchanging the central event router (e.g. using e.g. MQTT for distributed workloads).
• each event must include metadata as context (e.g. source of logs, or Elasticsearch index to use, ...)
• A simple plugin interface giving config to constructor and having / start(), stop() functions to register for events. Plugins should emit events. In addition plugins should be able to expose CLI or at least config file options.

What to do?

• rewrite exiting inputs/outputs for the event driven architecture (POC done locally for stdin/stdout/elasticsearch)
• implement plugin loader mechanism (and unload on SIGTERM/SIGINT/SIGQUIT)...
• implement config loader interface, so plugins could expose available options to the config loader

Add --help

And show usage with all options

Aggregate or down-sample events

In order to relieve pressure downstream and to avoid collation complexity, would be nice if I could aggregate logs within logagent-js. I'll try to explain more with an example. I would like logs such as:

{message: 'foo happened', field1: 'type1', metric1: 42, metric2: 1}
{message: 'foo happened', field1: 'type2', metric1: 24, metric2: 2}
{message: 'foo happened', field1: 'type1', metric1: 40, metric2: 3}

to be user-definably aggregated with something like the following before shipping them:

function aggregateFoo(pattern, buffer) {
  if (buffer.foo === undefined) {
    buffer.foo = {};
  }
  if (buffer.foo[pattern.field1] == undefined) {
    buffer.foo[pattern.field1] = {metric1: [], metric2: [], first: now()};
  }
  var agg = buffer.foo[pattern.key];
  agg.metric1.push(pattern.metric1);
  agg.metric2.push(pattern.metric2);
  agg.last = now();
  // Could aggregate in N-minute minimum buckets
  if (agg.last - agg.first > '5 minutes') {
    shipFoo(agg);
  }
  // Could aggregate once N samples collected
  if (agg.metric1.length > 100) {
    shipFoo(agg);
  }
  delete buffer.foo;
}

function shipFoo(agg) {
  // Demonstrating how users may want to define the aggregated event.
  var event = {};
  event.message = 'foo happened ' + agg.metric1.length + ' times';
  event.ts = agg.last;
  event.window = agg.last - first;
  event.metric1 = avg(agg.metric1);
  event.metric2 = max(agg.metric1);
  ship(event);
}

Is this already on the road map, and if not, is it in the scope of what you would like for logagent-js? I may have the green light to hack my way through this. If could be generally useful to others, let's discuss further here on how best to approach?

Process backlog more slowly to avoid sucking up resources and hurting the target

Related to #37

Instead of processing a potentially big backlog as fast as possible (and in the process negatively affecting other resources - CPU, network consumption, the output target, etc.) it may be better to start backlog processing slowly and then slowly increase the processing rate until the optimal speed has been achieved.

How "optical speed" is defined/determined is TBD. :)
Maybe Logagent should inspect responses it gets from the target looking for errors, and if certain errors/error codes are returned reduce the processing speed. Maybe it should keep track of response times and slow down if they are starting to take too long.

Maybe there should be a fixed maxOutputRate, too, that could be used as the absolute max limit.

common log format error

burger.letters.com - - [01/Jul/1995:00:00:12 -0400] "GET /shuttle/countdown/video/livevideo.gif HTTP/1.0" 200 0

the ip of type burger.letters.com is not getting parsed correctly

Access to pattern in transform()

At first, I didn't know why you'd pass pattern into the new filter() option. After looking at the example, it makes sense now and has me wondering if it makes sense to expose that in transform() as well. I'm looking for something similar to logstash's translate filter. For small to medium sized dictionaries, I think makes sense to be able to define those once in the pattern instead of having to define inside the transform function.

Before this grows legs, might want to consider a strategy for preventing name clashes between logagent and user's configs. Letting users define arbitrary fields in the pattern's namespace is likely to cause problems in the future. Maybe can simply be handled with documentation: "You should namespace anything yourself so we don't clobber it in the future."

Implement Filebeat input

So one can use Logagent in place of Logstash, for example.

Make it easy to exclude originalLine

Shipping originalLine plus the structured version of it means shipping 2x more bytes. It should be easy to include/exclude shipping of originalLine. As a matter of fact, maybe originalLine should not be shipped by default.

Add support for multiple pattern files

This way one wouldn't have to modify the original patterns.yml and count instead add custom ones.
e.g.
-f /path/to/patterns.yml,/path/to/my-custom-patterns.yml
or
-f /path/to/patterns.yml, -f /path/to/my-custom-patterns.yml

Order of files controls which patterns take priority if any definitions clash. The latter overwrites the previously defined one(s), like CSS.

(btw. maybe -p would have been a better thing to use than -f?)