Api improvements: compression, json serialization/streaming

[nuclearcat]

When i started to test rotonda with RouteViews MRT import (quite big) and /routes endpoint i noticed that there is significant initial delay, then there is huge spike in memory consumption and last, answer is quite large (9GB+) and not compressed.

I implemented compression, which might be significant for today prices, especially at cloud (244Mb vs 9GB!), and implemented serialization, where it outputs data to client gradually, and instantly, as soon as it start to receive results from search_routes, without buffering much. I set chunk size 256KB for now, it looks like transfer speed is at least not worse (maybe better in my case due lower memory pressure).

Here is some tests results:

Vanilla (no compression, no streaming):

curl -D - \
    http://127.0.0.1:9090/api/v1/ribs/ipv4unicast/routes \
    -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:35 --:--:--     0
HTTP/1.1 200 OK
content-type: application/json
vary: accept-encoding
transfer-encoding: chunked
date: Fri, 26 Dec 2025 23:23:36 GMT

100 9766M    0 9766M    0     0   265M      0 --:--:--  0:00:36 --:--:-- 2559M

With compression:

curl -H 'Accept-Encoding: gzip' --compressed -D -     http://127.0.0.1:9090/api/v1/ribs/ipv4unicast/routes     -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:37 --:--:--     0
HTTP/1.1 200 OK
content-type: application/json
vary: accept-encoding
content-encoding: gzip
transfer-encoding: chunked
date: Fri, 26 Dec 2025 23:21:14 GMT

100  244M    0  244M    0     0  3298k      0 --:--:--  0:01:15 --:--:-- 6207k


With streaming without compression (disabled on client):
curl -D - \
    http://127.0.0.1:9090/api/v1/ribs/ipv4unicast/routes \
    -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
HTTP/1.1 200 OK
content-type: application/json
vary: accept-encoding
transfer-encoding: chunked
date: Fri, 26 Dec 2025 23:43:16 GMT

100 9766M    0 9766M    0     0   272M      0 --:--:--  0:00:35 --:--:--  290M

With streaming with compression:
curl -H 'Accept-Encoding: gzip' --compressed -D -     http://127.0.0.1:9090/api/v1/ribs/ipv4unicast/routes     -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0HTTP/1.1 200 OK
content-type: application/json
vary: accept-encoding
content-encoding: gzip
transfer-encoding: chunked
date: Fri, 26 Dec 2025 23:48:59 GMT

100  244M    0  244M    0     0  5722k      0 --:--:--  0:00:43 --:--:-- 5954k