warrenweckesser/npreadtext

Read text files into a NumPy array.

npreadtext's Issues

Question: handling encoding parameter

One of the outstanding tasks in #1 is handling the encoding parameter.

I've looked at this a bit and thought that doing so may require some changes in the parsing code, particularly in tokenize. Currently, tokenize uses char32_t to represent elements in the word buffer which works as expected for utf encoded files, but is a tricky fit for buffers of bytes. I was thinking that the simplest solution would be to template the tokenize function to provide version that handled buffers of bytes also.

IYO does this sound sensible? Or is there maybe a simpler way of treating buffers of bytes without having to modify the tokenization function?

Create clone of numpy.loadtxt

Create the function loadtxt with an API that is compatible with numpy.loadtxt.

API completion tasks

• handle sequences, iterators, generators, etc.--whatever loadtxt can handle (currently read handles a filename or a file-like object)
• converters parameter
• encoding parameter
• parse complex values (e.g. 1.2-3.4j). (Make the complex unit character configurable.)
• unpack parameter
• ndmin parameter
• dtype parameter
• comments parameter
  • one comment string, max 2 characters.
  • multiple comment strings (low priority)
• delimiter parameter
• skiprows parameter
• usecols parameter
• max_rows parameter

Optimization

• Dynamically update data types in a single pass (currently, giving an explicit dtype results in a single pass, and specifying dtype=None results in two passes)