colearendt/cellist

Making beautiful music out of cell-lists, list-columns, nested lists, and the like

cellist

The goal of cellist is to turn nested list columns into tidy data_frames.

Example

This is a basic example which shows you how to solve a common problem:

## basic example code

Key Functions

• col_spec and related items (col_list, col_object, col_double, etc.)
• guess_spec
• spread_list
• gather_list
• inverse operations to rebuild the list?
• mappings from json_schema objects to col_spec?
• helpers to move from xml2 and jsonlite objects to nested lists

API

Should be able to do something like col_list(), col_list_spread(), col_list_gather()... Should also be able to nest specs in col_list... something like the following...

col_spec(
  list(
    d = col_double()
    , int = col_integer()
    , obj_raw = col_list()
    , obj_spread = col_list_spread(
      a = col_double
      , name = col_character()
    )
    , obj_gather = col_list_gather(
      b = col_integer()
      , name = col_character()
    )
    , arr_raw = col_list()
    , arr_spread = col_list_spread(
      1 = col_integer()
      , 2 = col_integer()
    )
    , arr_gather = col_list_gather(
      col_integer()
    )
  )
)

This API seems a little unweildy, but it seems that you would be able to pull out the collector functionality into a separate package and make it extensible (so the code is not defined for readr and tidylist)! These collectors make use of the name to do look-up by reference. This is not unlike readr, who also has a col_names parameter. The difference is that in this case, I think asked-for fields should be returned, even if not present.

The real power comes in something like guess_spec that will generate a spec for you... you could also conceive of generating a spec from a JSON Schema / XML schema object!

This also needs to be do-able by integer reference, i.e. list(1,"a","b") would grab the 1st object of a list, the first "a" key, and then the first "b" key.

Open Questions

• How should we handle the existing list? Should we hold to "copy on reference" or should we modify the list using "do not repeat yourself"... pull the items out? Probaby pull the items out... some items are not reversible
• maybe worth creating callbacks for naming the columns...? This at least for unnamed columns... named columns will be preserved? What about handling nested behavior, though... maybe a callback for that too

To Do

• Create tests to define what the internal functionality should be doing (since I cannot keep it straight otherwise...)
• Integrate purrr more natively
• Handle name conflicts
• col_spec needs to be changed to col_types and parsed by col_spec_standardise, much like readr does

cellist's Issues

Basic implementation of `gather_list`

In tidyjson, this was called gather_array or gather_keys or gather_object, depending on the context. The basic idea is that you gather the objects in a single row into many rows.

Outline desired functionality for `write_list`

This is the inverse operation to gather, spread, etc. It is desirable to users to write JSON, XML, and other types of nested-list data. It would be great to be able to use similar syntax to invert the cellist operations. This would transform a tidy dataset and output JSON, XML, or some other nested object.

https://twitter.com/abresler/status/942815614971928577

Basic implementation of `alter_list`

Here we would have the ability to change the objects of a list. i.e. turning character into numeric, etc.

Basic implementation of `enter_key`

The analogous operation in tidyjson is to enter_object. It can be very useful for discarding the information that you do not care about. I wonder whether there should be a similar option for an array... i.e. "just take the first."

In any case, one of the difficult questions here is: what do you do with rows that do not have this key? I prefer to give that option to the user. They should be able to do either (and we should handle NULL / NA nicely)

Scope support for `json_schema`, XML type definition, etc.

It would be very cool to either generate a json_schema object for validation, interpret a json schema object for parsing, or something of that nature. Anything to provide a "nice" default behavior to users.