simple conversion of DF to numerical matrix

[juliadata]

simple conversion of DF to numerical matrix

[doobwa]

For strings (and PooledDataVecs more generally), it would make sense (at least in my applications) to replace them with their refs. Thoughts?

[tshort]

Chris, could you explain a bit more? With PooledDataVecs, we're already storing a reference to a string. Or, is this about "dummy variables"? If so, that makes sense if a Float64 can point somehow be a ref.

[doobwa]

I was thinking x = convert_to_float(df) shouldn't necessarily create dummy variables for strings (since that's what expand does). Instead, for each PooledDataVec in df we would place its refs in x (promoted to Float64 for Uint16). But, if the goal is to make it easy to go from DataFrame to statistical modeling, then maybe automatically expanding to dummy variables would be a safer default.

[tshort]

Thanks for the explanation, Chris. It's beyond my expertise.

[ianfiske]

I'm new to JuliaData development, so I hope you don't mind my 2 cents. I think that Harlan's intent was probably meant as a shortcut to generate a model matrix using all columns in a DataFrame. In essence, something like model_matrix(:(~ col1 + col2 + ... + colp), df)). This would be a useful way to generate a model matrix for statistical modeling without typing in all of the column names. Along those lines, dummy variable (or other contrast methods) would be the way to go for PooledDataVecs. Of course, Harlan can speak up if this was or was not his intention.

[doobwa]

I originally misread your comment. I agree, that would be a good feature.

[HarlanH]

Hm, I'm not sure if I remember my original intentions here. But at the moment, yes, it might just be equivalent to model_matrix on all columns. Note that DataVec{ASCIIString} should get dummies too, and that PooledDataVec{Float64} probably shouldn't (although that'd be a weird situation).

[StefanKarpinski]

Can you guys explain what you mean by "refs" here? I'm a bit confused.

[HarlanH]

Oh, just that PooledDataVecs have integer indexes into the pool, and one option for generating a numerical matrix from a DataFrame, as an alternative to creating dummy variables for categorical values, is to use those indexes (converted to floats) as the values. I don't think it's a good idea. It conflates implementation details (the pool) with appropriate behavior for categorical values. It'd be odd, but occasionally useful, to replace categorical variables with some sort of numerical value (especially if they're ordinal, ala survey responses). But we also need to be able to support DataVec{ASCIIString}, which doesn't have underlying integers at all.

[StefanKarpinski]

Ah, ok. I see what you're saying. Thanks for the explanation.

[johnmyleswhite]

I closed this a while back.