billy1380/javaepfimporter

Java version of apple's EPF importer

javaepfimporter's Issues

Multi-threaded

Add a command line flag that allows the user to specify that the tool should use more than 1 thread.

Not sure if it is a good idea to be reading and seeking on the same file but could we be reading more than one file and importing them at the same time?

Logging not rolling

The log does not seem to roll correctly.

Should probably be using something like this

http://stackoverflow.com/questions/13585921/log4j-roll-file-for-each-execution

Some data errors

Hi. Thanks for writing this tool - I couldn't get the python one working, so it's really appreciated.

I'm seeing these data issues. Is anyone else? Any tips on how to overcome them?

Thanks in advance.

2017-09-27 01:44:27,245 INFO  MySqlIngester - Beginning full ingest of epf_paid_application_popularity_per_genre (3169192 records)
2017-09-27 01:46:27,251 INFO  MySqlIngester - ...at record 923600...
2017-09-27 01:48:27,263 INFO  MySqlIngester - ...at record 1844200...
2017-09-27 01:50:27,266 INFO  MySqlIngester - ...at record 2766400...
2017-09-27 01:51:19,345 ERROR Program - An error occured while ingesting data.
java.util.IllegalFormatConversionException: d != java.lang.String

2017-09-27 01:51:19,347 WARN  Program - The following files encountered errors and were not imported:
        itunes/popularity20170913/[album_popularity_per_genre, paid_ipad_application_popularity_per_genre, free_ipad_application_popularity_per_genre, song_popularity_per_genre, free_application_popularity_per_genre, paid_application_popularity_per_genre]

and

ÏηÏεÏία i-Status είναι ÏÏÏÎ¯Ï ÎµÏιÏλέον ÏÏέÏÏη και μÏοÏεί να ενεÏγοÏοιηθεί για Ï
                                                                                                                       Î¹Ï online εγκαÏ
                                                                                                                                           αÏÏ
                                                                                                                                               άÏÎµÎ¹Ï Ï
                                                                                                                                                         Ïν εÏαÏμογÏν Status με Ï
          και username.', 'New stuff!!', NULL, 'http://www.mainsys.gr', 'http://is3.mzstatic.com/image/thumb/Purple1/v4/a8/f5/26/a8f526ab-d2bc-4a07-cad0-89017fd613b6/source/320x480bb.jpg', NULL, NULL, NULL, '320x480', NULL, NULL, NULL, 'http://is1.mzstatic.com/image/thumb/Purple3/v4/e2/e0/9b/e2e09b60-642a-980f-837f-240c7ec108ae/source/360x480bb.jpg', NULL, NULL, NULL, '360x480', NULL, NULL, NULL)
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: '\xE2?\xA2 In...' for column 'title' at row 18
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4206)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4140)
 at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2597)

Out of range value for column 'retail_price'

I see the error 'Out of range value for column 'retail_price' while importing. I think it's because DECIMAL(9,3) has not enough digits for some currencies. I think the solution would be to increase the digits (i.e. DECIMAL(16,3).
I see there is a DECIMAL definition in smalldata/application_price. Should I change it there?
Or should I do that in the database?

Thanks in advance!

Exponential backoff does not reset

Because of the way that exponential backoff was implemented there does not seem to be a good way to reset it, if a success is achieved after a failure, this will probably require changes to IngesterBase.java so that both importers get the changes.

How to compile?

Sorry if that is obvious, but how do I compile a jar? My java knowledge is rusty, I'm used to javac and ant.

resume after failure does not work

Cannot seem to resume after a failure, because status dictionary/file is not populated

The current resume functionality just resumes from the last attempted data file.