Questions on Reproducing Main Results

[Y-debug-sys]

Hello, thanks for releasing the code and for the great paper. The idea is interesting, and the experimental results are strong and impressive.

I‘m trying to run the code and reproduce the results reported in the paper, but I has a few quick and tricky questions:

1. Is there a recommended custom way to run the code to reproduce the main experiments (e.g., scripts, configs, or specific settings for different size)? For me, the current version does not seem to support this without any guidance.
2. How is the reported storage (memory) space computed? Does it include all auxiliary data structures?
3. How is throughput and latency measured in the experiments?
4. For different data stream datasets, is it necessary to retrain a corresponding size meta model?
5. What does {}_task_file.npz represent, and how can I generate the corresponding embeddings for my own dataset?

Thanks again for making the code available. Best regards!

[FFY0]

Hello, and thank you for your attention.

As an early exploration, Meta-Sketch must be retrained for each dataset and each memory size. Here, “memory size” refers only to the size of the memory itself. Other network parameters are excluded, since they are amortized across many instances and, in future work, will be negligible compared with the memory size.

Regarding the code: because different datasets and memory sizes require customized settings, the repository provides a working example for the WordQuery dataset with 9KB memory size under MetaSketch_TPAMI/ExpCode/WordQueryBasicSketch/. You should be able to run this example directly.

I also strongly recommend more recent work, LegoSketch (ICML 2025) (https://github.com/FFY0/LegoSketch_ICML). By addressing scalability at the architectural level, LegoSketch naturally resolves the issues of extending to new datasets and memory sizes, thereby significantly reducing the difficulty of reproducing both training and evaluation.

Throughput and latency measurement: throughput is measured in terms of batches, while latency is measured as the time for a single operation executed sequentially.