Open
Description
Using RDS to read/write/store data is slow and not portable. We may want to switch the data pipeline or dashboard to being written in Python in the future, so using a format that Python can easily read would be preferable.
Looking at a comparison of different formats and packages, storing as a CSV and reading/writing using data.table
via fread
and fwrite
seems pretty good. feather
could also be an option (is supported in both R and Python) but it's not clear if standard dataframe and dplyr
procedures work with it. data.table
data can be seamlessly processed either with dplyr
or with faster data.table
syntax.