ben-strasser/fast-cpp-csv-parser

fast-cpp-csv-parser

fast-cpp-csv-parser's Issues

issues on creating installer file on VS2017

Your code works well while creating normal exe file. But I want to create a setup file for my project so I added Visual Studio Installer --> Setup Project. Upon trying to build a setup file it came up with these warnings.

---

You may refer my cpp program here.

How to parse less columns than in the line when using `set_header()`

I have a line with, say 15 columns, but I'm interested only in the first 5.
However, I don't have a header so I use set_header() instead of read_header(). But set_header() does not have an ignore_policy.

How can I parse just the first 5 columns, without a bunch of dummies?

test code

Hi,
I was wondering if you had any test code/benchmarking code ?

I add some CMake logic to easily add bunches of gtest tests...

I would like to add an istream interface to LineReader, but before starting any coding,
it would nice if there are some tests already I could add.

Cheers
Jiri

Partial lines

Hello,

I have a NFS file which is an append-only CSV. Another process keeps adding more lines to the end and my application keeps checking file size of it: when size increases it picks up new lines by calling pread() and uses fast-cpp-csv-parser to parse each line.

But sometimes I only got partial lines, and I believe it's because of this piece of code in next_line() method:"
if(buffer[line_end] == '\n'){
buffer[line_end] = '\0';
}else{
// some files are missing the newline at the end of the
// last line
++data_end;
buffer[line_end] = '\0';
}
"

Since the NFS file is a moving target, there might be partial lines when the other process each time writes/saves, but it doesn't mean it's the end of file missing the new line.

Thanks!

Fails to parse string with four consecutive quotes!

Fails to parse following input:

"Id","Title","Body","Tags"^M 
"1","How to check if an uploaded file is an image without mime type?","<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt=""""></p>
","php image-processing file-upload upload mime-types"^M 

The output for Body column should be

<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt=""></p>

But the ouput is:

<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt="></p>

Optional Values

I have a .csv file with data that looks like the following:

a,b,c
1,2,3
1,,3
1,2,3

I'd like to know that there's no b value for the second line (rather than interpreting it as 0). Is it possible to do this?

Passing a vector of columns to be populated instead of names

Hi,

Instead of explicitly typing out variables into which you'd parse the CSV file like this:

std::string id; std::string speed; std::string value; _reader->read_row(id, speed, value);

Is it possible to instead accept a vector containing columns with specific types to do this? Something like the following:

auto cols = new Columns{ ... three string cols ... };
_reader->read_row(cols);

will then populate cols[0], cols[1] and cols[2] as needed. Any suggestions on how this can be done?

too few colomns

I'm trying to parse a GTFS file, with a certain number of columns. Is this code conceptually able to parse columns in a random order as long as they are mentioned in the header? Using this code:

    io::CSVReader<3> in("/mnt/volatile/skinkie/calgary/stops.txt");
    in.read_header(io::ignore_extra_column, "stop_id", "stop_lat", "stop_lon");
    std::string quaycoderef; float latitude; double longitude;
    while(in.read_row(quaycoderef, latitude, longitude)){

I get the following error:

terminate called after throwing an instance of 'io::error::too_few_columns'
  what():  Too few columns in line 2 in file "/mnt/volatile/skinkie/calgary/stops.txt".

With the data:

stop_id,stop_code,stop_name,stop_desc,stop_lat,stop_lon,zone_id,stop_url,location_type
112112,112112,"Crescent Heights High School",51.060931,-114.065158,,,0

I have also tried:

    io::CSVReader<3, io::trim_chars<' '>, io::double_quote_escape<',','\"'>> in("/mnt/volatile/skinkie/calgary/stops.txt");

and...

    io::CSVReader<9, io::trim_chars<' '>, io::double_quote_escape<',','\"'>> in("/mnt/volatile/skinkie/calgary/stops.txt");

    in.read_header(io::ignore_extra_column, "stop_id", "stop_code", "stop_name", "stop_desc", "stop_lat", "stop_lon", "zone_id", "stop_url", "location_type");
    std::string quaycoderef; std::string stop_code; std::string stop_name; std::string stop_desc; float latitude; double longitude; std::string zone_id; std::string stop_url; std::string location_type;
    while(in.read_row(quaycoderef, stop_code, stop_name, stop_desc, latitude, longitude, zone_id, stop_url, location_type)){

limited column count?

I seem to have trouble compiling this code if the number of columns is greater than 10. For example here is code with 11 columns specified.

Invoking.. $ gcc -g -Wall -std=c++11 -I . -c test1.cpp where test1.cpp is...

#include "src/include/fast-cpp-csv-parser/csv.h"

int main(int argc, char* argv[] ) {
  io::CSVReader<11> * in = new io::CSVReader<11>("testdata.csv");
  in->set_header("a","b","c","d","e","f","g","h","i","j",'k');//,"l");

  typedef char * valtype;
  valtype sa=0,sb=0,sc=0,sd=0,se=0,sf=0,sg=0,sh=0,si=0,sj=0,sk=0;//,sl=0;

  in->read_row(sa,sb,sc,sd,se,sf,sg,sh,si,sj,sk);//,sl);
}

The error message include:

./src/include/fast-cpp-csv-parser/csv.h:1104:41: error: no matching function for call to ‘io::CSVReader<11u>::set_column_names(char)’
                         set_column_names(std::forward<ColNames>(cols)...);
                                         ^

The 3,6,9,10 column versions compile fine. I've tried this with gcc ver 4.9.2 and ver 5.4.0

"Too many columns" error bug

I get this error when there is a comma enclosed with quotes. Is there a way to ignore commas inside quotes?

How to read a cell that have many line?

When a cell in .csv file that has many lines, It throws exception "Too few columns in line ... ".
I can not read it as char *, please teach me how to solve it.
Thanks

LineReader buffer race condition with AsynchronousReader

When the LineReader destructor is called, it frees the data buffer. However, there can be an in-flight read call in the AsynchronousReader worker thread, which will try to put stuff into the freed data buffer. This leads to a segfault. To reproduce, read a large file and throw an exception while it is reading.

The solution is simple: don't deallocate the buffer until the reader is joined. I have a PR I can send over, which basically replaces the char* buffer with a std::vector<char> buffer and puts that before the AsynchronousReader reader in the class members so that it gets destructed after the reader is destroyed.

CSV library fails to link on Windows with Threads ON

I have an issue linking the library to our project on Windows,
you can see the original report and Appveyor output at: numenta/nupic.core-legacy#890 (comment)

In the followup we tried adding pthread as suggested, but didn't help, so we resolved to a workaround with disabled multithreading on Win (-DCSV_IO_NO_THREAD).

It's not critical but would be nice if the Win version worked the same as on UNIXes.

How to parse strings with commas in them

I get a "too many columns" error when trying to parse a string with comma in it. However, I can't escape this string in the CSV because it is a valid WKT string (and read by other tools reading this CSV file).

My CSV looks like this:
id, wkt
0,"POLYGON ((1 0,-1 0,1 0))"

Ignore values

Sometimes I don't want to read all values.
Is there a way to ignore columns e.g.:

int i;
in.read_row(i, io::ignore, io::ignore, io::ignore);

This would save me a lot of lines.

too many column names specified

when I use in.read_row(x,y,z,a,b,c),compiler said that "error C2338: too many columns specified"

Unused parameter warnings

I get a lot of warnings for unused parameters. Is there any way to fix or suppress them?

This is with gcc (i686-posix-dwarf-rev1, Built by MinGW-W64 project) 4.9.2.

I use it like this

CSVReader<2, trim_chars<>, no_quote_escape<'\t'>> reader(filename.toStdString());
reader.read_header(ignore_no_column, "time", "painlevel");

And the warnings look like this

In file included from ..\PainDetection\csvreader.h:7:0,
                 from ..\PainDetection\controller.cpp:2:
..\PainDetection\csv.h:519:51: warning: unused parameter 'line' [-Wunused-parameter]
                 static bool is_comment(const char*line){
                                                   ^
..\PainDetection\csv.h: In instantiation of 'static constexpr bool io::trim_chars<trim_char_list>::is_trim_char(char) [with char ...trim_char_list = {}]':
..\PainDetection\csv.h:509:73:   required from 'static void io::trim_chars<trim_char_list>::trim(char*&, char*&) [with char ...trim_char_list = {}]'
..\PainDetection\csv.h:707:69:   required from 'void io::detail::parse_header_line(char*, std::vector<int>&, const string*, io::ignore_column) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; std::string = std::basic_string<char>; io::ignore_column = unsigned int]'
..\PainDetection\csv.h:964:86:   required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_header(io::ignore_column, ColNames ...) [with ColNames = {const char*, const char*}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; io::ignore_column = unsigned int]'
..\PainDetection\controller.cpp:20:61:   required from here
..\PainDetection\csv.h:498:57: warning: unused parameter 'c' [-Wunused-parameter]
                 constexpr static bool is_trim_char(char c){
                                                         ^
..\PainDetection\csv.h: In instantiation of 'static void io::no_quote_escape<sep>::unescape(char*&, char*&) [with char sep = '\011']':
..\PainDetection\csv.h:708:74:   required from 'void io::detail::parse_header_line(char*, std::vector<int>&, const string*, io::ignore_column) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; std::string = std::basic_string<char>; io::ignore_column = unsigned int]'
..\PainDetection\csv.h:964:86:   required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_header(io::ignore_column, ColNames ...) [with ColNames = {const char*, const char*}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; io::ignore_column = unsigned int]'
..\PainDetection\controller.cpp:20:61:   required from here
..\PainDetection\csv.h:571:44: warning: unused parameter 'col_begin' [-Wunused-parameter]
                 static void unescape(char*&col_begin, char*&col_end){
                                            ^
..\PainDetection\csv.h:571:61: warning: unused parameter 'col_end' [-Wunused-parameter]
                 static void unescape(char*&col_begin, char*&col_end){
                                                             ^
..\PainDetection\csv.h: In instantiation of 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]':
..\PainDetection\csv.h:1029:50:   recursively required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t, T&, ColType& ...) [with T = int; ColType = {}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]'
..\PainDetection\csv.h:1029:50:   required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t, T&, ColType& ...) [with T = double; ColType = {int}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]'
..\PainDetection\csv.h:1053:64:   required from 'bool io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_row(ColType& ...) [with ColType = {double, int}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment]'
..\PainDetection\controller.cpp:24:39:   required from here
..\PainDetection\csv.h:1012:47: warning: unused parameter 'r' [-Wunused-parameter]
                 void parse_helper(std::size_t r){}
                                               ^

fatal error C1083: Cannot open include file: 'cerrno': No such file or directory

How can i compíle? I am using VC++ 2008

Bitwise vs. Logical And

https://github.com/ben-strasser/fast-cpp-csv-parser/blob/master/csv.h#L900
This is a bitwise and instead of a logical and. This causes parsing to be incorrect for some cases, for example when ignore_policy = 2 and ignore_missing_column = 3. The same thing is true for line 909.
To fix it, replace the & with an &&.

variable # of columns

could you change the code so that you allow CSVReader to be defined with the number of rows specified by a variable rather than a constant, and then allow read_header to use a vector array of strings for the header names?

3 buffers vs double buffer

Hi Ben,

This is not really an issue, just a question. I've read your code (async part) and I noticed that you used 3 buffers: every time you complete the first buffer you copy the second to the beginning, wait for the last third to complete and then you copy it as well . What's the reason for this design?
Why not using just double-buffer technique: parse from the first part while asynchronously read into the second half, wait for it to complete and then to parse the second part, asynchronously reading from the first part. It seems simpler so I am guessing there is a catch.

Dont throw too_many_columns() when using ignore_extra_column

Hi! Many thanks for a beautiful library.

I noticed that when we use in.read_header(io::ignore_extra_column, "vendor", "size", "speed");
if the csv file's last column is not "vendor", "size" or "speed" a too_many_columns error is thrown.

Would it make sense to not throw that error if we used the ignore_extra_column policy on the reader?

CSV Parser memory footprint for large file

I want to use this library on embedded device which has limited processing power and memory.
The size of CSV file to be imported is around 200KB - 500KB.

1. How does the library handles reading of this file? Does it read everything in one shot? Or does it read single row at a time?
2. Is it possible to specify number of row which should be read at a time? For example 20 rows at time, after processing them ask for another 50.
3. Do you have any memory benchmarks? If no, how much memory will the library allocate?

Regards

Compile errors under Embarcadero C++ compiler

Thanks for this library and it works great under macOS/LLVM. I do have a need to run this library under Win32 and for reasons I won't go into here, I'm using a free C++11 compiler from Embarcadero (Borland C++ in my youth). bcc32c reports the following errors:

include\fast-cpp-csv-parser/csv.h:236:52: error: use of overloaded operator '!=' is ambiguous
(with operand types 'const std::unique_ptr<ByteSourceBase>' and 'int')
                                return byte_source != 0;

include\fast-cpp-csv-parser/csv.h:262:48: error: use of overloaded operator '!=' is ambiguous
(with operand types 'std::unique_ptr<ByteSourceBase>' and 'int')
                                if(byte_source != 0){

include\fast-cpp-csv-parser/csv.h:296:52: error: use of overloaded operator '!=' is ambiguous 
(with operand types 'const std::unique_ptr<ByteSourceBase>' and 'int')
                                return byte_source != 0;

Any thoughts or insights?

encrypted csv files

How would you recommend reading/writing encrypted CSV files with this parser?
Assume the user has a encrypt/decrypt functions. string encrypt(string); and string decrypt(string)
I could extend your code.. but also any guidance would be great

Changeling separator

I have CSV with pipe (|) separator and I cant figure out howto to change separator character.

typo

in README.md
unsigned get_file_line(unsigned)const; -> unsigned get_file_line()const;

Can this read a CSV file that only has numbers, in one go?

So I have a csv file which contains 480 rows of 640 comma-separated numbers (so 640 columns of 480 values each) - representing a 480x640 matrix that was stored in the csv file. Is it possible to read this file into a cv::Mat or a standard float[][] matrix in C++ in one go? i.e. instead of copying 640 columns and figuring out the mapping to a Mat, can this copy all this data into a matrix? I don't have any headers.

Feature Request: Load gzipped csv files

Is there any plan on supporting loading some format of compressed csv files. It is very common people compress csv files to save space on disk and reduce I/O.

Unknown column numbers

Hello Ben Strasser,

I have CSV files which I want to import. Unfortunately I do not know numbers of columns before(either at compile time or before opening header in CSV file and counting header columns.)

1. Is it possible to include utility function which will count numbers of columns in CSV header(first row)? I can get value from this function to pass to CSVReader constructor?
2. Best choice is, CSVReader constructor readers CSV header and find outs number of columns.

Lots of code duplication in the line-reading exceptions

Example:

struct extra_column_in_header :
    base,
    with_file_name,
    with_column_name{
    void format_error_message()const{
                std::snprintf(error_message_buffer, sizeof(error_message_buffer),
                        "Extra column \"%s\" in header of file \"%s\"."
                        , column_name, file_name);
        }
};

struct missing_column_in_header :
        base,
        with_file_name,
        with_column_name{
        void format_error_message()const{
                std::snprintf(error_message_buffer, sizeof(error_message_buffer),
                        "Missing column \"%s\" in header of file \"%s\"."
                        , column_name, file_name);
        }
};

Essentially the same code but a different string. Considering how exceptions are, well, exceptional, I think it should not be impossible to get to something like:

using extra_column_in_header = exception<file_name, column_name>;
const char* exception_message_traits<extra_column_in_header>::format_string =
     "Extra column \"%s\" in header of file \"%s\".";
using missing_column_in_header  = exception<file_name, column_name>;
const char* exception_message_traits<missing_column_in_header >::format_string =
     "Missing column \"%s\" in header of file \"%s\".";

or whatever.

Doesn't work if a cell contains comma

a,b,c
1,"23,3",44
23,545,56

I get libc++abi.dylib: terminating with uncaught exception of type io::error::too_many_columns: Too many columns in line 2 in file "/Users/talha/Coding/ClionProjects/pricerightnlp/test.csv".

Handling files with optional header lines?

What is the best way to write the code to handle the situation where you get a file with the correct format but the header line is missing?

We get the same data from multiple sources and some sources provide a header and some do not.

forwarding to LineReader issue

Hi,

when trying to pass in a istream& in here, clang barfs because it tries to call a copy ctor.

The proper fix is to make the argument take a universal ref...

explicit CSVReader(Args&&...args):in(std::forward<Args>(args)...){

Cheers
Jiri

Variable number of columns per line

Can this class be used to parse a single CSV file without a header that may contain a variable number of columns per line?

For example, each line's first column may contain a type. Some types may be associated with more or less columns than others.

The goal would be to have a way of iterating through each line, peeking at the type in the first column, and parsing other fields as needed based on the type.

Example code does not run when compiled with CMake

I've tried compiling the example code from the readme using CMake but I keep getting the following error:

libc++abi.dylib: terminating with uncaught exception of type io::error::can_not_open_file: Can not open file "ram.csv" because "No such file or directory".

I've place a dummy CSV file named ram.csv in the same path as main.cpp and the included csv.h. The file also contains the appropriate columns and dummy data to match the example. My CMakeLists.txt is simply:

cmake_minimum_required(VERSION 3.3)
project(csv_example)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")

set(SOURCE_FILES main.cpp)
add_executable(csv_example ${SOURCE_FILES})

Any ideas on what I'm doing wrong? Thanks.

Support to skip N header rows

Hi,

is there a way to skip arbitrary N rows of the header? Our CSV files look like this:

colName1, colName2, ...
meta1,meta2
comment1, comment2
data1-1,data2-1
data1-2,data2-2
....

Is there a way to parse this? I'd call set_header("colName1", "colName2"); then something like: skipToRow(4), ...read data

If not, would you consider implementing that?
Thank you,

Work with stream

Hello,

My current application need to parse a CSV stream produced by a WebService.
Presently, i have to write it down to disk to use your parser, maybe it would be useful to create an interface for (string)stream ?

Handle different line endings

I ran into an issue attempting to parse the CSV file here: https://github.com/donohoe/nyt-crossword. The file has CR line endings, but this parser only supports LF and CR-LF. I had to run mac2unix on the file to get it to parse. It would be nice to have extra line ending support, or have it be configurable.

See https://en.wikipedia.org/wiki/Newline#Representations

<future> is not supported when compiling with /clr or /clr:pure

is not supported when compiling with /clr or /clr:pure

CSV files with LF as part of data

So I'm trying to parse a .CSV file that has a data member that is distinguished by quotes, but contains line feed in it.
Example line with 3 elements:
Colors,3,"Red
Blue
Green"
Is there a way to parse this with your tool? It seems to read: Colors,3,"Red, and then throw an escaped string not closed error, or a not enough columns error if i dont use escape strings. Any solutions?

why not use two block_len buffer instead of three?

reuse variable

I know it's probably a bad design but I'm trying to find a way to reuse an existing CSVReader instance for reading another file so that I can treat a lot of csv files as a single merged file, any ideas? (I can't iterate on a list of files either as there's no offset setting on the library to return to last read row when I have to stop the iteration midway)

Seems cumbersome interface in certain cases?

Say I have a CSV file with 30 columns, an unreliable header (but order is guaranteed), and I only care about 4 columns. So, I've got to call set_header with 30 names for the columns. Then in my read call I need to pass 30 variables. Is that correct? Seems cumbersome, even if I needed all 30 columns. It would be nice to have an option where the row elements are returned in std::vector and then I do the data type conversions myself for the columns I need.

Does not parse multiple lines in a column separated with '\n'

Consider following input:

"Id","Title","Body","Tags"^M 
"1","How to check if an uploaded file is an image without mime type?","<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem i    s that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>

<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>
","php image-processing file-upload upload mime-types"^M 

Here the Body column consists of HTML text with \n as a separator.The newline is denoted with \r\n or ^M. The attached file highlights occurance of all newline characters in the text. I can mail you the sample file in case there is any confusion.Unable to format it correctly using markdown. The data is actually stackoverflow tag prediction data from kaggle which is available openly.

The parser throws following error:

what():  Escaped string was not closed in line 2 in file "../data/pristine/Train_100.txt".
Aborted (core dumped)

I temporarily fixed this issue by a small hack in next_line() function line number 272.

    while(buffer[line_end] != '\r' && line_end != data _end){
        ++line_end;
    }
    ++line_end;//on assumption that \r is followed by \n

I am searching for \r instead of \n. This works for my case but can we have something more robust that handles such input data.

PS: I am new to github as well and this is my first issue

Number and names of CSV columns only known at run-time

Hi, firstly thanks for writing this CSV library!

I was wondering if it is possible to read a CSV file whose structure is only known at run-time. In particular, I have a vector of column headers (std::vector<std::string>) to read from the CSV which I would ideally like to pass to the parser directly – instead of a series of separate parameters which must be known at compile-time. The size of the vector is also only known at run-time.

I am sure I could use the LineReader class for this, however, if the changes required in the CSVReader class are only minor I might try and adapt it.

Do you think the class is apt for this purpose?

Separate csv.h into several header files

This is a nice little library, but "header-only" should not mean "one header only", and 1200 lines, while not terrible, is still a lot for just a CSV parser. There are clear cutoff points for separating out several header files; certainly LineReader and CSVReader can have files of their own.

... and then you tie it all together with a csv.h that includes the different header files.

Could I change the column_count from compile time to runtime?

Rather than

io::CSVReader<3> in("ram.csv");

change it to

io::CSVReader<> in(3, "ram.csv");

I did the change already, it work as expected.

what needs to be adjusted to make the code work without C++ 11

Could you please move the instructions from google code here local in github?
https://code.google.com/p/fast-cpp-csv-parser/issues/detail?id=6

For within some regions, we could not access services from Google =_=!

It seems OK after deleting constexpr and replacing std::snprintf() with fprintf()

does not parse part of a file

very nice. but I needed to read only some lines of a file, and only some parameters from these lines. this is why I decided unwillingly to do something worse myself... (https://github.com/pgallir/easy-cpp-subfile-parser).
by the way, this could be an useful features for many. as far as I am concerned, you should introduce this feature in your lib.