Fix shard ordering bug on some filesystems.#138
Conversation
* We were relying on the results of FileInputFormat.listStatus() returning a sorted set of statuses when loading multiple parts files. This is not a safe assumption on some files systems which return them in essentially random order. This corrects the problem by overriding listStatus in FileSplitInputFormat and sorting the results. * See broadinstitute/gatk#5881 for additional information.
|
@heuermh Could you take a look at this? It's a pretty nasty bug that has apparently been around for a while but was only manifesting locally on macs. It started happening in GATK's travis builds for some unknown reason ( I don't know what the configuration change was that made it start...). @jamesemery Please take a look at this as well. |
jamesemery
left a comment
jamesemery
left a comment
There was a problem hiding this comment.
This looks good, I'm not sure what the best way to test a transient-Filesystem dependent error like this would be...
| <groupId>org.disq-bio</groupId> | ||
| <artifactId>disq</artifactId> | ||
| <version>0.3.7-SNAPSHOT</version> | ||
| <version>0.3.7-bugfix-SNAPSHOT</version> |
There was a problem hiding this comment.
This change isn't necessary for this pull request -- versioning will be handled by the Maven release plugin on the next release cycle.
There was a problem hiding this comment.
Woops! That's funny, I literally told @tedsharpe the same thing on his last PR and then I did it myself...
|
Requested the version change be dropped, otherwise looks good to me. |
|
On further investigation disq is only passing because the reader tests don't assert that the files we get back are in the correct order... I'm seeing the sam issue with SAM files that isn't fixed by this patch. (I didn't even know we could write sharded sam files...) |
|
I suspect we may have the same issue with vcf too? |
3 participants
This is not a safe assumption on some files systems which return them in essentially random order.
This corrects the problem by overriding listStatus in FileSplitInputFormat and sorting the results.