String data, right truncation

[pacman82]

String data, right truncation

[pacman82]

@Htorne Closing this issue. I do not know the exact type of the column which caused you the warnings, but I know I fixed something ;-)

[pacman82]

@Htorne , we should find out why this is happening. For once, truncated strings could mean a loss of data and there is also the thing that to transfer that many diagnostic records from server to client takes a long time. How long did it take for you to execute this query?

[pacman82]

Since the messages generated by the driver do not contain a column number we could test which column it is, by only Selecting one column at a time.

[pacman82]

Let's see what types we have here:

• Numeric with scale 1. Wanted to map these properly to decimal anyway
• -9: These seem to be Wide Character Columns on the Database. Likely UCS2. Don't have a test for these yet. Maybe the driver does report the length to represent these in characters, rather than in bytes. A UTF-8 char might take up to 6Bytes of space so somthing might be coing on here. I can write a test for this though.
• -151: I'd have to look this up. Could you tell me what the type of the last column is?

[pacman82]

Column length seems to be reported in characters, rather than in bytes fo the target encoding.

[pacman82]

Your suggestion, that it is the geometry column is most likely right. As is visible in the log messages you posted, the driver reported 0, which is MS SQL Server specific to indicate that the user defined type is unlimited in size. The last version of odbc2parquet took the zero litterally however and allocated a buffer which could not fit a single char per row. There is now a new version 0.5.0 which will only log one warning about ignoring this column.

Other improvements which might be interssting to you. The decimal column you have should now be properly mapped to decimal, as support has been increased beyond decimals with scale 0. There is also extra space allocated for text columns, but I doubt you've hit any issues there.

If my local test setup is any indication, the new version might run significantly faster, as the server does no longer need to generate that many diagnostic messages.

If ignoring the "strange" column is not good enough for your use case please open a new issue stating the exact column type and use case.