Compress files in a single queue, not by directory #16
Description
The original decision to compress files by directory is flawed. It limits parallelisation to the number of files in a directory, and also means directory traversal and file identification cannot occur in parallel with file compression.
Would be much better, and cleaner, to have a single queue for files to compress and have a thread which finds files to compress and adds them to the queue.
One of the reasons for the original design was so nccompress could clean up after itself, and remove the tmpdir, and complain if it couldn't, which indicated a problem compressing a file. This was stupid. There are other ways to achieve this.