mmuurr · GitHub

top_n (and possibly other ranking functions) within group_by chains got really slow in the latest master branch version ( c7ca374 ).
Performance is fine when the number of groups is small, but it appears (in my light testing) that the slowdown is a (rapidly-growing) function of the number of groups.
Observe:

library(tidyverse); library(magrittr)
n_groups <- 10e3
n_obs <- 100e3
x <- sample(n_groups, n_obs, TRUE)
y <- runif(n_obs)
df <- tibble(x, y)
system.time(foo1 <- df %>% group_by(x) %>% top_n(1, y))
#    user  system elapsed
#  4.868   8.328  13.195
system.time(foo2 <- df %>% group_by(x) %>% arrange(desc(y)) %>% mutate(ix = row_number()) %>% filter(ix == 1) %>% select(-ix))
#    user  system elapsed
#   0.440   0.008   0.448
system.time(foo3 <- df %>% group_by(x) %>% arrange(desc(y)) %>% slice(1))
#    user  system elapsed
#  0.112   0.000   0.113
identical(sort(foo1$y), sort(foo2$y)) ## TRUE
identical(sort(foo2$y), sort(foo3$y)) ## TRUE

top_n performance previously appeared to be similar to the 'naive' foo2 and foo3 variants above.


Here's a performance analysis for n_groups varying between 1,000 and 10,000:

n_groups <- seq(1e3, 10e3, by = 1e3)
n_obs <- 100e3
results <- lapply(n_groups, function(n) {
    print(n)
    df <- tibble(x = sample(n, n_obs, TRUE), y = runif(n_obs))
    t1 <- system.time(df %>% group_by(x) %>% top_n(1, y))
    t2 <- system.time(df %>% group_by(x) %>% arrange(desc(y)) %>% mutate(ix = row_number()) %>% filter(ix == 1) %>% select(-ix))
    t3 <- system.time(df %>% group_by(x) %>% arrange(desc(y)) %>% slice(1))
    lst(t1, t2, t3)
}) %>% setNames(n_groups)
map_df(results, function(x) map_df(x, "elapsed"), .id = "n_groups")
 # A tibble: 10 x 4
    n_groups     t1    t2    t3
       <chr>  <dbl> <dbl> <dbl>
  1     1000  1.663 0.201 0.122
  2     2000  3.192 0.268 0.129
  3     3000  4.038 0.226 0.099
  4     4000  5.128 0.269 0.100
  5     5000  6.328 0.297 0.104
  6     6000  7.413 0.323 0.103
  7     7000  8.737 0.358 0.107
  8     8000 10.009 0.399 0.107
  9     9000 11.118 0.418 0.111
 10    10000 12.487 0.453 0.112

Read the original on github.com ↗