Comments (11)
For strings (and PooledDataVecs more generally), it would make sense (at least in my applications) to replace them with their refs
. Thoughts?
from dataframes.jl.
Chris, could you explain a bit more? With PooledDataVecs, we're already storing a reference to a string. Or, is this about "dummy variables"? If so, that makes sense if a Float64 can point somehow be a ref
.
from dataframes.jl.
I was thinking x = convert_to_float(df)
shouldn't necessarily create dummy variables for strings (since that's what expand
does). Instead, for each PooledDataVec in df
we would place its refs in x
(promoted to Float64 for Uint16). But, if the goal is to make it easy to go from DataFrame to statistical modeling, then maybe automatically expanding to dummy variables would be a safer default.
from dataframes.jl.
Thanks for the explanation, Chris. It's beyond my expertise.
from dataframes.jl.
I'm new to JuliaData development, so I hope you don't mind my 2 cents. I think that Harlan's intent was probably meant as a shortcut to generate a model matrix using all columns in a DataFrame. In essence, something like model_matrix(:(~ col1 + col2 + ... + colp), df))
. This would be a useful way to generate a model matrix for statistical modeling without typing in all of the column names. Along those lines, dummy variable (or other contrast methods) would be the way to go for PooledDataVecs. Of course, Harlan can speak up if this was or was not his intention.
from dataframes.jl.
I originally misread your comment. I agree, that would be a good feature.
from dataframes.jl.
Hm, I'm not sure if I remember my original intentions here. But at the moment, yes, it might just be equivalent to model_matrix on all columns. Note that DataVec{ASCIIString}
should get dummies too, and that PooledDataVec{Float64}
probably shouldn't (although that'd be a weird situation).
from dataframes.jl.
Can you guys explain what you mean by "refs" here? I'm a bit confused.
from dataframes.jl.
Oh, just that PooledDataVecs have integer indexes into the pool, and one option for generating a numerical matrix from a DataFrame, as an alternative to creating dummy variables for categorical values, is to use those indexes (converted to floats) as the values. I don't think it's a good idea. It conflates implementation details (the pool) with appropriate behavior for categorical values. It'd be odd, but occasionally useful, to replace categorical variables with some sort of numerical value (especially if they're ordinal, ala survey responses). But we also need to be able to support DataVec{ASCIIString}, which doesn't have underlying integers at all.
from dataframes.jl.
Ah, ok. I see what you're saying. Thanks for the explanation.
from dataframes.jl.
I closed this a while back.
from dataframes.jl.
Related Issues (20)
- expression functors (in particular: `over`) HOT 14
- show in html is a bit hard to read HOT 8
- leftjoin! is actually copying reference instead of value?! HOT 1
- Column unioning when appending data HOT 4
- Tests of `describe` and `multithreading` fail in Julia-1.10.0-beta3 HOT 13
- feature request: `shift` HOT 2
- Allow adding columns to DataFrameRow just like to SubDataFrame HOT 1
- error when unique! a empty dataframe HOT 1
- ggplot2 has added an `after_stat`
- feature request: allow `skipmissing` column types HOT 3
- `combine` on grouped df return empty df when args is empty HOT 2
- DF description/missing color is unreadable on dark themes HOT 44
- Add rename!(::DataFrame, ::Pair{Regex, SubstitutionString}) method
- GroupBy then combine changes column order HOT 2
- Inconsistent Mean Calculation in Grouped DataFrame Compared to Overall DataFrame HOT 2
- What is the best way to write large DataFrames efficiently and with high performance in Julia while minimizing memory usage? HOT 4
- Segmentation Fault when reading compressed file HOT 1
- Revisit spreading for `AsTable` output` HOT 6
- Better error message when forming a DataFrame from a vector of dictionaries with missing data. HOT 2
- `describe` is slow HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframes.jl.