Identify data type and then push into array (in case of csv2array)

[evanplaice]

Identify data type and then push into array (in case of csv2array)

[evanplaice]

From [email protected] on October 01, 2012 14:54:15

Part of the problem with using CSV as a data type is, there's no way to specify whether the data should be cast to scalar values or if they should remain in string form.

There are two clear reasons why I don't want to make auto-casting a default action of the parser.

First, it's not a good idea to assume that all data needs to be cast to scalar values. To maximize performance, it's best to avoid any unnecessary processing by default. For instance, in cases where all of the data is strings then taking the time to type-check every single value is just a waste.

Second, I want to keep the API as slim and predictable as possible. Doing too much magic under the covers by default may introduce unwanted side-effects (ex casting automatically where the user wants the value as a string). When that happens the abstraction begins to leak and the user is pained with figuring out a way to disable the internal implementation.

I would like to avoid leaky abstractions and side-effects as much as physically possible while providing a stable and performant parser to the users.

To strike a healthy balance I want to leave this functionality out by default but make it possible to add it in through a different approach. Basically, by adding a hook (ie event callback) within the parser stage (ie just before output.push) the user will be able to insert an arbitrary function to enable further processing.

In your case you'd want to add auto-casting. In other cases the user may want to provide a string formatting function that outputs float values as strings with only two digits of precision with a dollar sign and parentheses around negative values (ie as is done in accounting). I could probably think of numerous use cases but the point is, it's more useful to empower the users to extend the parsing capabilities than to try and strike the perfect balance of functionality among a large diverse user base.

Fortunately, because javascript is a 'functional' language, the default function can be easily overridden by simply assigning a new function value to the hook parameter.

As soon as I have a working implementation I will incorporate your code to add auto-casting as the first use case...

If you have any more feedback, I'm all ears. If not, a working implementation will be made available as soon as I have it ready.

Thanks,
Evan

[evanplaice]

From [email protected] on October 02, 2012 10:55:10

Thanks. The hook is better as you said, leaving the parser ... be just the
parser :)

[evanplaice]

From [email protected] on October 06, 2012 22:21:25

OK, it's done. Released with 0.64.

I'm on a roll.

The feature isn't documented yet but there's a new parameter called onParseValue.

You set it with a function-as-a-value that works as a callback. It takes one parameter called 'value' the parser sets that you can manipulate. I also created $.csv.hooks to contain convenient hook functions which includes one called castToScalar (loosely based on your code.

Here's an example of how you can call it:

var output = $.csv2Array(testHooks, {
  onParseValue: $.csv.hooks.castToScalar
});

Here's the code for castToScalar:

$.csv.hooks.castToScalar: function(value) {
  var isNumber = /^[\d\.]+$/;
  var hasDot = /\./;
  if (value.length) {
    if (isNaN(value)) {
      return value;
    } else if (isNumber.test(value)){
      if (hasDot.test(value)) {
        return parseFloat(value);
      } else {
        return parseInt(value);
      }
    } else {
      return undefined;
    }
  }
}

I replaced your not_a_number function with the one built into JavaScript, and changed the naming conventions but it should function the same way.

There's also a test in the test runner that checks its accuracy and demonstrates its usage because I'm thorough like that.

The best part is, if you want to incorporate a different type of inline post-processing the capability is already there. I'm not sure what other hooks I'm going to add yet, maybe onParseEntry, onPreParse, onPostParse.

Have fun. Test it out. Let me know if it breaks.

[evanplaice]

From [email protected] on October 06, 2012 23:22:06

Status: Fixed

[evanplaice]

From [email protected] on October 08, 2012 11:42:08

Thanks, will try this and let you know. http://techdiary-viki.blogspot.in

[evanplaice]

From [email protected] on October 10, 2012 21:07:53

Owner: [email protected]

[evanplaice]

From [email protected] on October 16, 2012 01:25:48

I have written about how I used this package: http://techdiary-viki.blogspot.in/2012/10/javascript-using-ajax-jquery-csv-to.html

[evanplaice]

From [email protected] on October 16, 2012 14:34:48

Nice,

Status: Verified