Comments (10)
Hi
Thank you for your report, that would be a serious bug. Can you clarify a bit more on how this appeared - was the exception happening somewhere near the beginning of dataset or near the end?
Also, if you would be able to reproduce it - does the issue reproduce on previous versions, e.g. 6.8.1?
from clickhouse.client.
I have tried to reproduce the issue in a test and it doesn't seem to appear with a synthetic test - a serialization exception is always correctly propagated. Can you provide more context please?
from clickhouse.client.
from clickhouse.client.
Please try version 7.5.0 to see if it fixes the issue for you
from clickhouse.client.
Nop, still not working correctly, here is a demo to replicate this problem. Let me know, maybe I am doing something wrong.
https://github.com/marcinatpayholdingcom/clickhouseclient_bug_replication
from clickhouse.client.
Sorry - took me a while to get my hands on it. However, for me it works perfectly:
Hello, World!
O:\Projects\clickhouseclient_bug_replication\ClickHouse.ClientNotSerializing\bin\Debug\net8.0\ClickHouse.ClientNotSerializing.exe (process 21308) exited with code 0.
Press any key to close this window . . .
Any additional details I am missing? Do you use any special flags or parameters (e.g. UseSessions)? What OS was this reproed on?
Here are specific exceptions which are shown in the debugger:
from clickhouse.client.
I am running this on Windows
this part should throw the serialization exception but it does not:
I do not have any special setting, I just run it as it is in the project.
So the problem is, it does not throw, so only part of the data is inserted and chunks with broken data are skipped.
from clickhouse.client.
For me it is triggered quite perfectly every time
Just to double check, when you say "exception is not thrown", have you checked it in the debugger by putting a breakpoint on the logger._LogError
line, or did you check only by looking at the logs? Because your test project does not output any logs even in case of error
from clickhouse.client.
by setting the brekpoin at the log line and by seeing what data got inserted into Clickhouse.
When you get the serialization exception that means that no data got inserted into Clickhouse in the first attempt, am I right?
In my case there is some data inserted.
I think you can close this issue as not able to replicate. I have no clue what setup causes this situation on my machine.
First I thought that it was because of the original project I am working on, maybe I did something wrong. But when I created this demo and things are still failing incorrectly then I give up.
from clickhouse.client.
I think there may be just a misunderstanding here, which means I'll need to make docs more clear
When you get the serialization exception that means that no data got inserted into Clickhouse in the first attempt, am I right?
ClickHouse is not transactional (other than experimental features) - and hence BulkCopy adapter isn't as well. If you have a large set of data and the serialization (or insertion) has failed for one of the batches, it is expected that some of the batches may have made it through already, and it falls on the higher-level code to handles the situation. I.e. in this particular case, 20k of rows get into database before exception - and that's by design.
If you need to have some semblance of atomicity, the data needs to go as one 'batch' - so you can try setting BatchSize
to Int32.MaxValue
(reference). It would be slow and memory-consuming, but you would get a binary result.
from clickhouse.client.
Related Issues (20)
- Error when executing a bulk insert - Value was either too large or too small for a UInt16. HOT 2
- Understanding connection lifetime & pooling
- Incorrect conversion of ClickHouseDecimal to intger types HOT 2
- NestedType problem. HOT 2
- Add random structure tests
- Insert two fields with the common prefix HOT 1
- [bug] BulkInsert intermittently fails HOT 7
- handling of Nullable types. HOT 1
- Bulk Insertion not working with Geo Data Types
- Incorrect conversion to Decimal type HOT 1
- Request hangs until timeout if the previous request has thrown an exception HOT 2
- Aspire component
- Query with id ... is already running. (QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING)
- Please remove dependency on System.Net.Http.Formatting HOT 1
- BulkCopy: Failing to convert DateOnly to DbType Date32 HOT 1
- Apache Arrow format HOT 1
- Ingest performance compression in multiple formats.
- Unable to create ClickHouse.Client.ADO.QueryStats (cannot convert to Int32) HOT 4
- ExecuteStatementAsync() not returning number of rows affected HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clickhouse.client.