Write script to evaluate the results of the tool

[marco-c]

The script should:

1. Train the model
2. Get a set of bugs from Bugzilla
3. For each bug:

• Get all signatures;
• Verify that the tool considers those signatures to be similar
• Verify that the tool, given a stack trace from one of the signatures, would have suggested a stack trace from one of the other signatures

Ideally, we have to somehow assign a score to the results of the tool, so we can compare different implementations (e.g. different word embeddings).

[marco-c]

To clarify, let's say a bug contains three signatures: A, B and C.

There are two simple things we can to to verify the results of the tool make sense:

1. Verify if the tool considers signatures taken in pairs to be similar (e.g. A and B, B and C, C and A).

2. Given in input a stack trace from A (then from B, then from C), the tool suggests X top similar stack traces. We can verify if these stack traces are either in A, B or C.

[marco-c]

Example of a bug containing multiple signatures: https://bugzilla.mozilla.org/show_bug.cgi?id=1333486.

[KirillTim]

Seems to be done