Comments (2)
It is unclear whether that feature provides significant value to be implemented. Users today can load data into a staging table with all columns being of a string data type and then perform the data type conversion inside the database.
There is a valid argument that dates should be converted correctly when loaded into a table with the correct data types, hence requiring a way of passing on the date format (as NLS value for example with Oracle). However, the question is where to start and where to stop. For example, what to do with TIMESTAMP
columns, and what to do with NUMBER
columns in different regions in the world where the fraction symbol is ,
and not .
. What about character set conversion, etc.
By having date format available to be passed on as command line parameter, users could be lead to expect all other conversions to be provided as well, quickly moving the complexity of data type conversion out from the database and into csv2db
and further increase the already long list of parameters. Since the very inception of csv2db
it has been a core principle for it not to become an ETL tools with all the bells and whistles. There are already many, very good ETL tools out there and the world doesn't need yet another. Instead, csv2db
should aid users as a simple command line tool to get rows from a delimited file into a database table, and not more! According to these principles, the question of data type conversion can be answered by the user in three ways:
- Load the data into a staging table with all string columns, then perform the data type conversion with database provided functionality such as INSERT INTO SELECT and/or stored procedures. However, this does not solve the question of character set conversion.
- Have the database configured correctly to match the territory and incoming data of the files, hence allowing for implicit data type and avoiding unnecessary character set conversion
- Do not use
csv2db
but one of the many, good ETL tools instead.
from csv2db.
This ER will not be implemented based on the feedback above.
from csv2db.
Related Issues (20)
- Error handling should throw exception name for better diagnosis HOT 1
- Improve performance for Postgres data loads HOT 2
- Upsert support HOT 1
- -s input for TAB delimited files HOT 1
- Introduce flag for skipping errors in a file HOT 2
- Postgres fast load HOT 8
- Option: truncate table before load HOT 1
- Introduce logging of bad rows into error file HOT 1
- Postgres: execute_batch still significantly slower than copy command HOT 3
- Are tabs as separator with no quotes around strings possible HOT 1
- Error generating statement: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte HOT 2
- How to execute the unit tests? HOT 3
- Convert into a Python package HOT 3
- "Smart mode" for schema creation
- MySQL: Support for `LOAD DATA INFILE`
- Command `generate`: Fail, when header field is empty HOT 1
- Extra parameter for TNSNAMES.ORA use scenario (Oracle OCI instances connection)
- Do not implicitly uppercase identifiers HOT 1
- Provide mechanism for quoted identifiers HOT 2
- Offer csv2db as Docker image HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csv2db.