Comments (8)
Updated to api 2 in 8d27e0f
from logstash-output-jdbc.
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/exponential-threadsafe/lib/logstash/outputs/elasticsearch/common.rb appears to be the implementation that logstash prefers.
Based on that, my TODO notes:
- Remove Stud::Buffer
- Consider what exceptions we want to make retry-able and what not to (question: just how feasible is this across all drivers?)
- Do we really want it to infinitely retry? Or do we want to keep the max_exceptions ring buffer
from logstash-output-jdbc.
e32b6e9 removes Stud::Buffer, keeps max_exceptions counter for a set of events, adds better support for recovering from batched insert failures. Missing tests.
from logstash-output-jdbc.
Notes for myself.
Currently discovering the pain from the JDBC batch behaviour. Includes, but not limited to:
- autocommit=true+batch behaviour is undefined and driver specific
- a driver may chose to continue processing after a failure
- getting the error for a particular entry in the batch in the event of the above doesn't seem possible
Since the plugin may be used with any JDBC-compatible engine, it may be just safer for each event to be processed as a single command (like unsafe goes through), and batches go in the bin π
from logstash-output-jdbc.
Batched SQL now in the bin.
TODO: Tests for new test cases.
from logstash-output-jdbc.
Shower thought: Do we actually want to retry any non-connection related JDBC exceptions. Most seem to be fatal errors that without human intervention won't resolve by retrying.
Retryable states by class:
- 08 β Connection Exception
- 24 β Invalid Cursor State (MAYBE)
- 25 β Invalid Transaction State (MAYBE)
- 40 β Transaction Rollback (MAYBE - specifically looking at deadlocks)
- 53 β Insufficient Resources
- 54 β Program Limit Exceeded (MAYBE)
- 55 β Object Not In Prerequisite State
- 57 β Operator Intervention
- 58 β System Error
Counter thought: JDBC drivers are inconsistent, and have their own SQLSTATE codes. Also May throw an SQL exception when they dont mean to (do we even care about this?). What's worse? Holding up the pipeline, or just losing a few events.
from logstash-output-jdbc.
Added custom retry_sql_states configuration. Currently this is the full sql state code, not the class.
from logstash-output-jdbc.
Closing for individual issues going forward.
from logstash-output-jdbc.
Related Issues (20)
- IllegalAccessError CompoundCacheKey HOT 3
- Insert emoji into mysql(utf8mb4) varchar field fail(Incorrect string value SQLException) HOT 4
- time zone information of date field lost, how to transfer time zone HOT 2
- How to fix jdbc output :connection has passed maxLifetime HOT 8
- Postgresql Class path contains multiple SLF4J bindings. HOT 1
- :error=>"missing class name (`com.zaxxer.hikari.HikariDataSource')" HOT 3
- Getting error while inserting records into oracle using Logstash HOT 5
- ERROR: Installation Aborted, message: incompatible encodings: Windows-1254 and UTF-8 HOT 2
- Error -1 (00000) : while preparing SQL HOT 1
- java.lang.VerifyError: Inconsistent stackmap frames at branch target 125 HOT 2
- Issue using JSON created field in statement. HOT 5
- Multi-table insert is not supported, please help meοΌAnd give examplesοΌ3Q HOT 10
- Issue when using the input and output jdbc plugin in the same conf file HOT 2
- how to insert binary object in statement? HOT 6
- call store procedure from oracle with output sys_refcursor HOT 4
- Connection not close when not have new data in a long time HOT 3
- Dynamically creation of insert statements HOT 6
- How to increase the output throughput for inserting the records HOT 7
- SSL off in postgresql HOT 1
- mysql INTEGER to long?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-output-jdbc.