Fix windows installer leaving files and error on plover_plugins #1186

Job	Run time
Analyze	15s
Test (Python 3.12)	-1s
Test (Qt GUI)	-1s
Test (Windows)	-1s
Test (Python Latest Supported)	-1s
Test (Linux)	-1s
Test (macOS)	-1s
Test (Python 3.11)	-1s
Test (Code Quality)	-1s
Build (macOS App)	0s
Test (Python 3.10)	-1s
Test (Packaging)	-1s
Build (Linux)	0s
Build (Windows)	0s
Notarize (macOS App)	0s
Build (macOS DMG)	0s
Notarize (macOS DMG)	0s
Release	0s
	5s

Provide feedback