feat: Expose query results as an IO object. #3
Conversation
| @@ -19,6 +22,33 @@ typedef struct { | |||
| struct local_result_v2 *c_result; | |||
| } LocalResult; | |||
|
|
|||
| VALUE rb_io_from_buffer(const char *buf, long len) { | |||
There was a problem hiding this comment.
Admittedly generated this via ChatGPT. One downside here is that we're copying the buffer into a pipe, which means we're still duplicating the byte array rather than wrapping it in an IO object that knows how to read from it.
| @@ -29,6 +29,7 @@ def build_query_string(query_str, output_format) | |||
| format_suffix = case output_format.downcase | |||
| when "csv" then " FORMAT CSVWithNames" | |||
| when "json" then " FORMAT JSON" | |||
| when "jsoneachrow" then " FORMAT JSONEachRow" | |||
There was a problem hiding this comment.
Allows us to parse each line separately each time we yield it, rather than needing to parse a potentially large result set in one go.
There was a problem hiding this comment.
Think we can benefit from some type of enum for all supported formats as well.
|
Will get back to this for some proper cleanup - opening this PR early for any possible comments and suggestions. |
|
Nice addition, @nelson-vantage; I'll take a closer look today. |
Gracias! I actually wouldn't merge this PR – it doesn't work as it should because the IO writer will block if the pipe object doesn't have enough space for the results. Still think we can improve the efficiency of the library by (a) not creating a new string every time we call |
Sketch of one possible approach to exposing the query results as an IO object instead of a singular string. Rushed through the interface – I believe ideally we replace the current
bufwithbuf_io.