aarreedd/csv-to-elasticsearch

Simple tool to import CSV into ElasticSearch

csv-to-elasticsearch's Issues

Large files

I tried to use the script for large file content >500000 the script gives this error

 ----- CSV to ElasticSearch ----- 
Importing all rows into `bus` from 'mrget.csv'

Traceback (most recent call last):
  File "csv_to_elastic.py", line 241, in <module>
    elastic_address=parsed_args.elastic_address, ssl=parsed_args.ssl, username=parsed_args.username, password=parsed_args.password, id_column=parsed_args.id_column)
  File "csv_to_elastic.py", line 101, in main
    elif len(row[0]) == 0:    # Empty rows on the end of document
IndexError: list index out of range

JSONDecodeError

I am getting a JSON decode error:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The top of my CSV file looks like following:

"date","temperature"
"1981-01-01",20.7
"1981-01-02",17.9
"1981-01-03",18.8

I am trying to run your script using the following command:

python csv_to_elastic.py --elastic-address 'localhost:9200' --csv-file daily-minimum-temperatures-in-me.csv --elastic-index 'temperatures' --datetime-field=date --json-struct '{ "date" : "%date%", "temperature" : "%temperature%" }' --delimiter ','

What could be a reason for such issue and how can I solve it?

Disable automatic type inference

Fields are automatically inferred to strings or numbers. This causes problems in some cases; in particular some long digitnumbers were turned into E-notation, for example 1231231231234 -> '1.231231E12'. This format was then refused by logstash since it didn't match the type.

In the json-format, it's possible to double-quote ""%key%"" to make it a string in the output; however the type conversion that happens before can still cause problems.

So there should be a way to disable this automatic conversion for specific fields, or alltogether.

Specify python version/requirements

Hi, first off this is an incredibly helpful tool, thanks so much for putting it together!

I might suggeset that in your README, for noobs like me, you could specify that this is written for python3, and that you will need to pip3 install python-dateutil and http in order for this to work. It could save some folks a little time figuring it out if they're not fully versed in python.

You may also want to add documentation that the --delimiter and --max-rows field exists as well- I got stuck for a few minutes trying to figure out why it said it was "Importing None rows into...." to elasticsearch.

Once again- thanks!

Returned status code: 401

I feel so lucky to find this amazing tool for quick importing csv file to elastic and thanks a lot!
While I am new to python and came into a problem: it ran properly but returned 401 in the console.
I would appreciate so much if you can check and tell me what's wrong.

1 console
$ python3 csv_to_elastic.py --json-struct '{"answer" : "%answer%","question" : "%question%"}' --csv-file data.csv --elastic-index qa --elastic-address 'localhost:9200'

----- CSV to ElasticSearch -----
Importing all rows into qa from 'data.csv'

Reached end of CSV - sending to Elastic
Returned status code: 401

2 I put data.csv in the same direction of py script and it looks like this:

answer | question
A1 | Q1

3 my index name is qa

Thanks a lot :)

how to import 200million data to elasticsearch more quilckly

i have about 10000 csv files, 200million rows total. i want to import them to es once. so can anybody tell me how to solve this problems;

First column in CSV is not imported.

I'm having issues with the first column of the file not being imported into Elasticsearch. First discovered this issue when I tried to set the first column as the "id" and it couldn't be found.

After importing my CSV, I tried a search by items in first column and it returned empty.

Is this by design? Are we supposed to be inserting a blank column in position 1?

IndexError : list index out of range

Hello,

I imported my CSV file using this command :

python csv_to_elastic.py --elastic-address 'localhost:9200' --csv-file 'full_test.csv' --elastic-index 'stops_index' --datetime-field=dateField --json-struct '{"stop_id" : "%stop_id%", "stop_name" : "%stop_name%", "stop_lat" : "%stop_lat%", "stop_lon" : "%stop_lon%"}'

and I got this output :

 ----- CSV to ElasticSearch -----
Importing all rows into `stops_index` from 'full_test.csv'

Traceback (most recent call last):
  File "csv_to_elastic.py", line 130, in <module>
    elastic_address=parsed_args.elastic_address, id_column=parsed_args.id_column)
  File "csv_to_elastic.py", line 54, in main
    int(row[pos])
IndexError: list index out of range
Microsoft Windows [version 10.0.16299.371]
(c) 2017 Microsoft Corporation. Tous droits r▒serv▒s.

I'm running Python 3.6.5 on Windows 10.

Thank you in advance!