Sample that demonstrates how to convert CSV files hosted on COS to data stored in a RDBMS #180
Conversation
Co-authored-by: Sascha Schwarze <[email protected]>
Co-authored-by: Sascha Schwarze <[email protected]>
Co-authored-by: Sascha Schwarze <[email protected]>
Co-authored-by: Sascha Schwarze <[email protected]>
Co-authored-by: Sascha Schwarze <[email protected]>
Co-authored-by: Sascha Schwarze <[email protected]>
cos-to-sql/app.mjs
Outdated
| const pgClient = await getPgClient(secretsManager, smPgSecretId); | ||
|
|
||
| // | ||
| // Perform a single SQL insert statement per user |
There was a problem hiding this comment.
It would be better to do bulk inserts to reduce load on the database.
Conceptually, using upserts here might also make sense, but does not need to be changed.
There was a problem hiding this comment.
Yeah, that would a lot of sense for this specific use case. But at the same time I wanted to mimic a flow in which complex information is processed and handled line be line
There was a problem hiding this comment.
I thought about your comment. While I kept the loop and individual db operations per line, I adjusted the table and switched to use upserts. Thanks for the feedback
|
|
||
| export async function getPgClient(secretsManager, secretId) { | ||
| const fn = "getPgClient "; | ||
| const startTime = Date.now(); |
There was a problem hiding this comment.
What is your motivation on this @SaschaSchwarze0 ?
Do you want to track independent durations for the SM and the PG init operaion?
There was a problem hiding this comment.
Ah, I see. Yes, I assumed you wanted to measure the secret fetching.
cos-to-sql/utils/db.mjs
Outdated
| const startTime = Date.now(); | ||
| console.log(`${fn} > firstName: '${firstName}', lastName: '${lastName}'`); | ||
| return new Promise(function (resolve, reject) { | ||
| const queryText = "INSERT INTO users(firstname,lastname) VALUES($1, $2)"; |
There was a problem hiding this comment.
To continue the best practive you have followed elsewhere, what about measuring and logging statement durations ?
There was a problem hiding this comment.
Not sure what you mean.
such statements are backed into the call on each DB operation, COS operation and SM operation. From my perspective, we are good.
Co-authored-by: Sascha Schwarze <[email protected]>
This PR provides