Training from scratch

[lihuibng]

Is it possible for us to train a new magika from scratch, or finetune on a base model?

[reyammer]

Hello, thanks for reaching out!

In principle, yes, it's possible to train from scratch, but we are unfortunately unable to share the dataset for a number reasons. And about finetuning: we actually never tried that as training from scratch (with a new version of the dataset) is usually relatively fast. But in principle it should be possible?

May I ask what would be the specific need for retraining / finetuning? In general, we are of course very happy to hear about detection gaps that we could fix for every user... if we have a chance to make things better, we are definitively willing to put the time in!

If you have specific questions or you need specific guidance, please also feel free to follow up via email (magika-dev@google.com).

Thanks!

[pombredanne]

I was about to file the same or similar issue:

• is there a training dataset that we can use to reproduce or recreate a model?
• what is the procedure for retraining?
• why is the training dataset not public? I am nervous to use some arbitrary binary from an opaque provenance in general, and even more so when this is to reuse a binary plug in an open source project, and even more more so if that binary is not reproducible.

I reckon that 100M files is not a lot so in general I would be much more comfy to rebuild a proper model from an open dataset than use some opaque blob.

I am sure that you are perfectly sane and honest people. I like to trust but I prefer to verify, or to be able to verify, the same way you would unlikely let some random binary be used at Google.

[reyammer]

Hello,

Thanks for the thoughtful comments. We appreciate the "trust but verify" mindset—it’s exactly the right approach for security tools.

Regarding the dataset: You are correct that we cannot make it public, primarily due to licensing and usage restrictions. Additionally, modern versions of the model use runtime augmentation techniques that make reproducing the exact training set complex.

Regarding the "opaque binary" concern: I want to clarify the distinction between the code and the model artifacts to address your threat model concerns.

• The Code: The inference engine (the logic that actually parses files and executes instructions) is fully open source (see the /rust or /python directories). You can audit and build this from source.
• The Model: The .onnx file is not an executable binary; it is a serialized set of weights.

Because the inference logic is open and verifiable, the "model file" itself does not have the capability to execute arbitrary code. A compromised model would result in incorrect detections (misclassification), but it presents a significantly different risk profile than running an opaque executable binary.

We are happy to support community efforts to retrain the model on open datasets! However, while we are committed to maintaining the production inference system, we currently do not have the bandwidth to build and maintain a production-ready public retraining pipeline.

I hope this clarifies the situation and helps alleviate your concerns about integrating the tool.

[pombredanne]

@reyammer re: #1261 (comment) ..

Thank you ++ for these details!
Short of having a production-ready pipeline, would you be able to dump whatever code you have in a branch at least?

Also you wrote:

Additionally, modern versions of the model use runtime augmentation techniques that make reproducing the exact training set complex.

Can you elaborate what you mean and what this is practically?

[pombredanne]

@reyammer re: #1261 (comment)

Just curious, did you use AI to generate that comment?

[reyammer]

Hello, sorry for the long delay in getting back to you. Hectic times! Just wanted to share that we'll likely start a new training round quite soon and I'll use this occasion to wrap up and share an example with the new dataset pipeline we have been working on. It may not be prod ready, but I would be happy to answer questions / directions / pointers / etc. Sorry again for the silence!

About the augmentation: there are n augmentation techniques we use (like cutmix) and their implementation is currently not fully deterministic, so results vary (but only slightly, these have usually a low impact). One more todo to take care of...