jeffwong / pdist Goto Github PK
View Code? Open in Web Editor NEWDistance functions on subsets of matrices
Distance functions on subsets of matrices
While pdist
works great it handles NAs in a different way of dist
When calculating the distance between 2 vectors dist
not only ignores NAs but scales the distance to the length of the vector.
dist
help file description of this behaviour:
Missing values are allowed, and are excluded from all computations involving the rows within which they occur. Further, when Inf values are involved, all pairs of values are excluded when their contribution to the distance gave NaN or NA. If some columns are excluded in calculating a Euclidean, Manhattan, Canberra or Minkowski distance, the sum is scaled up proportionally to the number of columns used. If all pairs are excluded when calculating a particular distance, the value is NA.
This is an example of the difference of behaviour:
v1 <- c(1,2,3,4,NA)
v2 <- c(5,NA,4,3,5)
pdist(v1,v2)@dist
[1] 4.24264
dist(rbind(v1,v2))
v1
v2 5.477226
# scaling by sqrt(5/3)
pdist(v1,v2)@dist * sqrt(length(v1)/(length(v1) - sum(is.na(v2) | is.na(v1))))
[1] 5.477225
I wrote a function to compute the scaling but is a way slower than dist
:
pdist.w.scale <- function(X,Y)
{
if (!is.matrix(X))
X = as.matrix(X)
if (!is.matrix(Y))
Y = as.matrix(Y)
distances <- matrix(pdist(X,Y)@dist, ncol=nrow(X), byrow = TRUE)
#count NAs
na.count <- sapply(1:nrow(X),function(i){rowSums(is.na(Y) | is.na(X[i,]))})
#scaling to number of cols
distances * sqrt(ncol(X)/(ncol(X) - na.count))
}
It would be great if the scaling feature (by default or as a option) was incorporated to pdist
.
the c function to pdist returns an array of floats which naturally maps to a vector. Reshaping this into a matrix can add a lot of overhead, so this should be factored out into a seperate as.matrix function
Running the following command will cause a segmentation fault and crash the R session:
pdist(matrix(0, 3, 3), list())
*** caught segfault ***
address (nil), cause 'memory not mapped'
Traceback:
1: .C("Rpdist", X.vec, Y.vec, nx, ny, p, distances = distances, NAOK = T)
2: pdist(matrix(1, 3, 3), list())
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:
My quite unexhaustive debugging suggests that this only happens when Y=list()
but that c()
, NULL
, data.frame()
or list(1:3)
are fine. The following also works:
pdist(list(), matrix(0, 3, 3))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.