Test: Settings editor search improvements

[rzhao271]

Refs: #226039

• anyOS @isidorn
• anyOS @Yoyokrazy
• anyOS @justschen

Complexity: 4

Create Issue

---

Sanity testing section

1. Open the Settings UI.
2. Confirm that workbench.settings.useWeightedKeySearch is unchecked by default.
3. Search for "editor formonpast".
4. Confirm that editor.formatOnPaste shows up.
5. Search for "editor.formatonpaste".
6. Confirm that only editor.formatOnPaste shows up.
7. Search for "stickyscrolle".
8. Confirm that terminal.integrated.stickyScroll.enabled shows up.
9. Enable workbench.settings.useWeightedKeySearch by checking the checkbox.
10. Confirm that searching for "editor formonpast", "editor.formatonpaste", and "stickyscrolle" still yield the same expected setting(s) as before.
11. Confirm that searching for "editor fomonpast" yields editor.formatOnPaste.
12. Confirm that searching for "editor tab size" yields editor.tabSize as the first result.

Open testing section

Try out other queries to search for your favourite settings!
For this iteration, we plan to ship with workbench.settings.useWeightedKeySearch unchecked by default.
Please try searching with the setting enabled and searching with the setting disabled.
Let me know if there are any queries that fail to yield an expected result, especially if the query yields the expected result with the setting disabled but not with the setting enabled.

Known issues

Synonym search (e.g. "editor paste style") is still not supported.
Typo detection (e.g. "editor formstonpaste") is still not supported.

[isidorn]

Overall I had a much worse Settings Search experience when I turned on useWeightedKeySearch.
Best illustrated by these two #238933 and #238934

I see the papercut that we want to solve with this feature, but can we do it in a less destructive way?

[rzhao271]

@isidorn thanks for the testing! I wonder whether some results I thought were good with the setting on were actually with the setting off this whole time. However, the search experience with the setting off is also new to this iteration (the first sanity test should fail on the current Stable), so I will likely mention it on the release notes but not mention the preview setting.

[justschen]

@rzhao271 Overall I also think it performed a bit worse for me - I typically keep my setting queries short and rarely know the actual setting name. I provided another few examples #238934 but will keep the setting on and report on more.

Search for "stickyscrolle". Confirm that terminal.integrated.stickyScroll.enabled shows up.

On this one, note that it did show up, but it was the second to last entry out of all entries

[Yoyokrazy]

echoing the prior sentiment, my queries are typically one or two plain language words, never looking for the setting id itself. Typos are rare (takes little time to correct them), it's more common that I forgot the correct word from the setting title. If I do search for an ID, it'll be copied and pasted so that should always work.