question about truncate_on_load for bi-server (follow-up to issue #71)

[paulkaefer]

If I have truncate_on_load set to false per @boonhapus' suggestion here, will there be duplicate data added to the table if I run this daily?

[boonhapus]

If you're running daily with --from-date and --to-date set to yesterday's date, then you'll not have any duplicates. If you have overlap in the dates, then there's a chance for duplicated data.

Currently the Syncer doesn't take into account any of the data in the store itself. I do want to add UPSERT / MERGE capability in a future update.

[paulkaefer]

@boonhapus for now, if I wanted to just set truncate_on_load=true, would it essentially replace the table each day with the full history?

How many days of history are kept in TS?

[boonhapus]

If you're on Cloud, it's kept for 36 months. If you're on Software, it's 6 months by default. The API we use to fetch the data is fairly slow though (it's the best we've got 😥), so it could be difficult to fetch the entire size of the table each day.

[boonhapus]

Another strategy you could employ is to use --from-date set to yesterday as well as pass the --include-today flag, then you could load this to a RAW schema and implement a follow up job to MERGE into the final table.