Comments (4)
Some more thoughts about this. The problem with ADVs inheriting from AbstractVector is that it complicates operations that need to ignore NAs. Consider mean(dv)
. If dv
has NAs, I think we can agree that the operation should fail. That works now. :) But if we have mean(naIgnore(dv))
, it's unclear how the mean(AbstractVector)
method should work. When the loop gets to dv[i]
, where that element is an NA, what should ref
return?
Instead, I think it's better to have a set of numerical methods that work on iterable types. There's already a sum
method that does this. See base/reduce.jl
. It would be easy to write mean
and any number of other methods that use duck-typing (since we don't have an Iterable type and multiple inheritance, yet) to efficiently calculate numerical summaries of an ADV with NAs.
Recommend we close this and replace it with an issue to implement statistical summary methods that work on iterables.
from dataframes.jl.
One way around the problem is to have naIgnore(dv)
return a different type that would not be an AbstractVector. That's the direction I was heading with regular arrays (see issue #54). I like this better than stuffing a flag into DataVecs. One of the reasons I like it is that if you can inherit from AbstractVectors, then many functions work for the case where there are no NAs. Then, adding support for NAs involves writing iterables to support the filter and replace options.
Either way, a lot more work needs to be done to improve support for DataVecs, and iterables are a good way to go. As far as this particular issue, I'll leave that up to you.
from dataframes.jl.
Hm, that would make sense to me if the primary goal was to avoid meta-data
inside DataVecs. But I don't think that's a good principle here. The
distinction between Data and non-Data is that Data has context aside from
its values. In particular, for statistical or measured data, it has
missing-values, as well as things like the nominal/ordinal/interval/ratio
distinction, which I'm planning to add soon to cover some of the issues
around categorical data we've been running into.
I'd prefer that we have the principle that a method either works for Data
or it doesn't, and that it doesn't depend on whether or not the Data has
NAs. You can always naignore() or nareplace() to get a non-Data vector that
can be processed by anything.
Other thoughts?
On Mon, Sep 24, 2012 at 9:37 AM, Tom Short [email protected] wrote:
One way around the problem is to have naIgnore(dv) return a different
type that would not be an AbstractVector. That's the direction I was
heading with regular arrays (see issue #54https://github.com/HarlanH/JuliaData/issues/54).
I like this better than stuffing a flag into DataVecs. One of the reasons I
like it is that if you can inherit from AbstractVectors, then many
functions work for the case where there are no NAs. Then, adding support
for NAs involves writing iterables to support the filter and replace
options.Either way, a lot more work needs to be done to improve support for
DataVecs, and iterables are a good way to go. As far as this particular
issue, I'll leave that up to you.—
Reply to this email directly or view it on GitHubhttps://github.com/HarlanH/JuliaData/issues/23#issuecomment-8818927.
from dataframes.jl.
Closed by 8d9ce5c
from dataframes.jl.
Related Issues (20)
- Segmentation Fault when reading compressed file HOT 1
- Revisit spreading for `AsTable` output` HOT 6
- Better error message when forming a DataFrame from a vector of dictionaries with missing data. HOT 2
- `describe` is slow HOT 3
- CartesianIndex error in Julia 1.11 HOT 4
- `DataFrame(x=Int[], y=Int)` HOT 3
- Add comparison function for dataframes which can handle both isapprox and isequal column types HOT 2
- unique fails with column-type FixedDecimal HOT 5
- mapcols! should modify the parent of a SubDataFrame HOT 11
- Feature request: Pairs in stack HOT 2
- Grouped DataFrame with array elements fails to combine HOT 4
- error when combining a grouped empty dataframe using `first` HOT 6
- Short circuit && on subset? HOT 1
- Integer strings as colnames/selectors are error prone HOT 2
- Suggestion - Matrix Syntax for hcat (as well as vcat) HOT 4
- Document custom generation of column names in manual HOT 9
- `join` should not introduce `Missing` types to schema HOT 1
- Consider removing Tables.allocatecolumn in vcat
- DataFrame(t::Table) converts PooledVector columns HOT 2
- Sampling GroupedDataFrames (rand) HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframes.jl.