hxlstandard/libhxl-js

JavaScript support library for the Humanitarian Exchange Language (HXL) data standard.

libhxl-js's Issues

After column rename row.coulmns returns old columns

hxlSet.columns.forEach(function(c){
    console.log(c.displayTag);
})

return new columns tags

hxlSet.forEach(function(r,i){
    r.columns.forEach(function(c){
        console.log(c.displayTag);
    })
})

returns old column tags

Support D3 and JQuery CSV parsing

Currently, we support only Papa Parse -- support the other libraries as well, if present.

Same headers in columns mean withColumns returns wrong column number

withColumns function returns the wrong column number if the headers are the same in multiple columns.

Previously this dataset had a column with header Country (the same as column 1 and column 4) and it gave the wrong result until the header was changed.

rough code to test:

dataset.withColumns(['#country+code'].forEach(function(row,col,rowindex){
 console.log(col)
});

Output gave column number as 1, but was expecting 4.

Make filters simpler

Fix filters to allow a single string spec, and to parse filter expressions for select from strings.

.withColumns fails if no match is found for tag.

dataset.withColumns(['#activity','#sector']).columns.forEach(function (column, index) {
console.log("Column " + index + " is " + column.displayTag);
});

fails with

hxl.js:1294 Uncaught TypeError: Cannot read property 'match' of null

Look into Node.js compatibility guidelines

Originally suggested by @benrudolph in #1

http://en.wikipedia.org/wiki/CommonJS

Filters fail if every column is not tagged

Make cache() filter more efficient

Right now, the cache() filter saved Row objects rather than just data arrays. Rewrite to cache only the raw data, and create Row objects on demand.

Investigate Excel libraries

Is there an Excel library for Javascript? That would make it possible to read .xls and .xlsx files directly, without the need to go through the HXL Proxy.

Add schema validation

Add schema validation similar to that in https://github.com/HXLProxy/libhxl-python

Suggested by @SimonbJohnson to support possible HDX Data Check prototype.

Add caching filter

The cache() or cached() filter will save a snapshot of the filter chain at its current point, so that earlier filters don't get rerun after expensive operations. Some filters (like count() and future sort()) will cache automatically.

Are there numeric min/max methods for datasets?

Make 0.3beta release

• check CHANGELOG
• merge to master
• run tests
• tag on GitHub
• make GitHub release
• update wiki

Add sort() filter

Add a sort() filter (automatically caching).

Be flexible with string matching in Select

Normalise whitespace and case, to avoid trivial mismatches.

Export data array

Allow exporting a complete data array, without or without headers. Requested by @SimonbJohnson

Add rename() filter

Need a method to rename columns (header and tag), similar to the rename method in libhxl-python, but with the extra requirement that we should be able to pick a column by relative index as well as tag. Requirement from @SimonbJohnson

Details

[9:38:34 AM] Simon Johnson: Hi David, I’m just having a play with hxl.js

As part of hxlmerge I want to give reoccurring core hashtags unique hl tags e.g. 
#org,#org to #org+hm1,#org+hm2

Is there an easy way to replace a tag…looking at the code it looks like I will have to change rawdata and also set savedColumns to null ?
[9:39:07 AM] David Megginson: Did I implement rename() in libhxl-js? Let me check ...
[9:40:12 AM] David Megginson: No, I didn't.  I realise that it wouldn't do what you want anyway, because it would rename all #org to the same thing.
[9:41:10 AM] David Megginson: I should be able to implement that as a simple filter. How soon do you need it?
[9:42:42 AM] Simon Johnson: I’m working on it now, but can pop over to other tasks if needed
[9:42:56 AM] Simon Johnson: before you start, here is my approach to the merge
[9:43:27 AM] David Megginson: I'll look into it today.  The Python rename() works like this:

myHXL.rename('#org', '#org+funder')
[9:43:45 AM] Simon Johnson: give unique id to hxl tags, create final data with all tags from both sets, iterate over rows filling the data in
[9:44:19 AM] David Megginson: Two options:

1. An optional positional arg: myHXL.rename('#org', '#org+hm1', 0); myHXL.rename('#org', '#org+hm2', 1);
[9:44:49 AM] David Megginson: 2. Allow an absolute column position as the first arg: myHXL.rename(5, '#org.hm1'); myHXL.rename(6, '#org.hm2');
[9:45:43 AM] David Megginson: Probably also implies a lookup method like getIndices(), e.g. var indices = myHXL.getIndices('#org');
[9:45:55 AM] Simon Johnson:  1 is less work for me on my particular use case
[9:46:03 AM] Simon Johnson: Yes… 2 would require the extra method
[9:46:37 AM] David Megginson: Which do you think is more sustainable?
[9:46:44 AM] David Megginson: (and more generally useful)?
[9:48:06 AM] Simon Johnson: I can’t think of a use case of the second that wouldn’t be fulfilled by the first

Node compatabillity

Hi David,

This is awesome; I'd love this for JS. I was thinking it might be worthwhile to make it compatible for node and not just the browser as most file reading will be done on server. Also in terms of design, I'd suggest something like a callback function when the file has been read (ex: https://github.com/wdavidw/node-csv). Node is single threaded so it depends on making all the long running tasks (like file reading) as asynchronous tasks which can be completed without blocking the main thread. I can help out with PR if that'd be useful. I think we should adhere to the standard specified in CommonJS (http://en.wikipedia.org/wiki/CommonJS). Underscore and D3 are also good examples of JS libraries that are compatible on the server and the browser