does it work well with big data?

[geostat-framework]

does it work well with big data?

[rth]

@moonlighf Well, some people who have successfully run PyKrige with a few 100k points, see #29 (comment)

In your case, I might make sense to,

1. build the variogram model on a small random subset of your points (say 10k-50k). You could use sklearn.utils.resample.
2. then run your kriging on the full dataset with backend="C" and by providing the n_closest_points parameter.

Though looking into this further it further it looks like we don't support this yet see #29 (comment)
So I think we would need to fix #29 for this to work.

In general, don't run any algorithm on a very large datset. Start by using a small subsample, see how long it takes then increase the dataset size. This way you can predict how long it will take to run on the full dataset without "waiting days" .

@basaks or @kvanlombeek might have other suggestions.

Also make sure you use the 1.3.2 version from PyPi or the development version from github.

[rth]

Could you please provide more details about the size of your data (number of points, etc), the corresponding run time in Arcgiss, if you run it on a single cpu or on a cluster, etc?
See scaling benchmarks in #37

[moonlighf]

@rth there is about millions of points and i have spent several days in arcgis, it doesn't get result. my computer is only single cpu .

[MuellerSeb]

Closing due to inactivity. Feel free to reopen if needed.