Increase speed for large data sets

[LKremer]

When plotting millions of points, counting the number of neighbors of each point is extremely slow. The current algorithm calculates the pairwise distance for all points. This could be optimized, for instance with this approach. Other ideas are welcome.

[seasmith]

Have you seen the RANN package?

I don't know too much about it but I know nngeo uses RANN to implement nearest-neighbors.

[LKremer]

Haven't tried this exact package yet, but I tried dbscan::pointdensity which also uses the same C++ library (ANN). It's roughly twice as fast when plotting ~100k points. I'm currently testing it.

[LKremer]

I added method="kde2d"to the geom and stat. It is indeed faster for many points (~ >20k) but slower for few points.

[olechnwin]

How do you use the kde2d method? I've downloaded the latest version of this package from github but geom_pointdensity still do not have method parameter?

[LKremer]

Hi @olechnwin,
you just use geom_pointdensity(method ="kde2d") and it should work. Maybe you still have an old version of the package loaded, or the old version is somehow still installed on your system? You could try restarting your R session, then explicitly uninstalling the old version and then reinstalling the GitHub version

[Aariq]

I've been testing some alternatives because MASS::kde2d runs out of memory with the dataset I'm working with. So far I've tried:

MASSExtra::kde_2d(): seems faster, promises to scale better, but still runs out of memory
KernSmoth::bkde2D(): works, and pretty fast too. Requires a larger than default gridsize and errors unless I increase it.

Some other options are summarized in this (in progress?) paper: https://vita.had.co.nz/papers/density-estimation.pdf