More thorough integration tests for pg, redshift, snowflake, bq#27
Conversation
| How this test works: | ||
| - Assumes you have a Postgresql instance with the test data loaded (see README) | ||
| - Assumes you have configured corresponding ENV vars in .env (see README) |
There was a problem hiding this comment.
Is this is a readme somewhere?
There was a problem hiding this comment.
Updated to include paths to the README, that's referring to the data-transfer/pontoon/README
| stream_size = ds.size(stream) | ||
| if stream_size == 0: | ||
| progress.message("No records to process for this stream") | ||
| continue |
There was a problem hiding this comment.
Is it possible to have this at a higher level instead of having it in each destination connector?
There was a problem hiding this comment.
it'd definitely be possible in the orchestration layer (somewhere around here) to remove zero-length streams from the Dataset before passing it to destination.write(), but I kind of think the connectors should be able to gracefully handle an empty stream since nothing in the interface contract prevents passing one in. Another thing we could do (but might actually be less clear) would be making the Dataset.streams iterator skip empty streams, but that limits a connector's ability to do something (bookkeeping / logging?) with empty streams.
2 participants
Uh oh!
There was an error while loading. Please reload this page.