I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
Comments
I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
There are many ways to process larger than RAM objects in R.
The dbplyr package makes it possible to use a local db table as if they are in-memory data frames (https://dbplyr.tidyverse.org/).
https://diskframe.com/ is also a pretty great solution that processes data by chunks.
Take a look at data.table https://cran.r-project.org/web/packages/data.table/
Sorry to butt in, but could you point me to a resource on processing data in chunks using Python?