-
-
Notifications
You must be signed in to change notification settings - Fork 865
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
V3 : Fix GIF, PNG, and WEBP Edge Case Handling #2882
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copilot reviewed 5 out of 16 changed files in this pull request and generated no comments.
Files not reviewed (11)
- tests/Images/External/ReferenceOutput/GifDecoderTests/Issue1962_Rgba32_issue1962_tiniest_gif_1st.png: Language not supported
- tests/Images/External/ReferenceOutput/GifDecoderTests/Issue2012BadMinCode_Rgba32_issue2012_drona1.png: Language not supported
- tests/Images/Input/Gif/issues/issue_2866.gif: Language not supported
- tests/Images/Input/Webp/issues/Issue2866.webp: Language not supported
- src/ImageSharp/Formats/Gif/GifEncoderCore.cs: Evaluated as low risk
- tests/ImageSharp.Tests/TestUtilities/ReferenceCodecs/MagickReferenceDecoder.cs: Evaluated as low risk
- src/ImageSharp/Formats/Webp/Lossy/Vp8Decoder.cs: Evaluated as low risk
- src/ImageSharp/Formats/Webp/BitReader/BitReaderBase.cs: Evaluated as low risk
- tests/ImageSharp.Tests/Formats/Png/PngEncoderTests.cs: Evaluated as low risk
- tests/ImageSharp.Tests/TestImages.cs: Evaluated as low risk
- src/ImageSharp/Formats/Webp/WebpBlendMethod.cs: Evaluated as low risk
Comments suppressed due to low confidence (2)
tests/ImageSharp.Tests/Formats/Gif/GifEncoderTests.cs:389
- [nitpick] The test method name GifEncoder_CanDecode_Issue2866 is misleading since it is testing encoding, not decoding. It should be renamed to GifEncoder_CanEncode_Issue2866.
public void GifEncoder_CanDecode_Issue2866<TPixel>(TestImageProvider<TPixel> provider)
src/ImageSharp/Formats/Webp/WebpAnimationDecoder.cs:89
- The comment should reflect the correct type for backgroundColor, which is TPixel instead of Color.
/// <param name="backgroundColor">The default background color of the canvas in.</param>
This is quite a huge change, but I'm planning to take a high-level look in the coming days. |
Thanks! I have a V4 version also that I’m going to push, lots of similarities but also some additional changes. |
…elMap{TPixel}.cs Co-authored-by: Anton Firszov <[email protected]>
@JimBobSquarePants I think it would be good to add a few more test case to verify everything works as expected. Here is a github repository with some gif images generated with imagemagick: gif-test-suite, which I think could be useful testcases. I can help with adding tests, I have some time in the next few days. |
That would be awesome if you could thanks! |
With some of test images I have added with f80aa76, the output of ImageSharp does not match what imagemagick would produce, which may indicate an error. The output from ImageSharp looks like this: I am not sure yet why. I will try to figure out what's wrong tomorrow. To extract the images with imagemagick, the following command can be used:
|
@brianpopow That looks like something to do with palette reading though the last frame suggests an additional bug. |
@JimBobSquarePants It seems that the issue is a regression from version 2.1.9 to 3.0.0. It is an issue with handling the disposal method The weird thing is that when I save the frames like this:
The images still look black and white. I am not sure what I do wrong here with saving the frames. |
I can fix restore to previous behavior easily enough but I'm seeing some odd behavior in ImageMagick for this image and others. It doesn't seem to follow the gif89a specification at all regarding GCE transparency and is setting entries at a given index with an alpha of zero regardless of whether there is a transparency flag. @dlemstra Any idea why this is? |
@brianpopow I figured out why the output is wrong when cloning. The current code attempts to encode the image using the palette data and bit depth extracted from the individual frames however the new frame contains all the data from the combined frame not just te delta. We shouldn't attempt to preserve the palette when converting between formats. I'll push a fix for this today to this branch. |
@JimBobSquarePants: I am glad you found the issue, I had trouble making sense out of this behavior. |
@SpaceCheetah Fixed with 71357d2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the long delay. This is great work and clearly an improvement over the current state. Correct me if I'm wrong, but in my understanding this PR is dealing with 2 relatively separate concerns: (1) fixing edge cases including #2866, (2) preventing quality loss to address #2862.
I have no concerns with (1), but I think there might be some space for improvements regarding (2).
In 4ace814 I did some benchmark extensions. While the original Gif benchmark is targeted at comparing ImageSharp to System.Drawing, benchmarking with a custom decoder setup isn't very representative when we want to track impactful performance changes within the library, since most users will likely go with the default decoder config. For that reason the commit splits the benchmark into two, where EncodeGif_DefaultEncoder
benchmarks the default encoder performance. I also added DecodeEncodeGif_*
to have benchmark that approximates the typical end-to-end user scenario.
These benchmarks show ~50% regression for EncodeGif_DefaultEncoder
and 20%-40% regression for DecodeEncodeGif_DefaultEncoder
which would land in a 3.1.*
revision update.
Config
BenchmarkDotNet=v0.13.0, OS=Windows 10.0.26100
Intel Core i9-10900K CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=9.0.201
[Host] : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT
DefaultJob : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT
EncodeGif_DefaultEncoder
release/3.1.x
| Method | TestImage | Mean | Error | StdDev | Ratio | RatioSD |
|--------------------- |--------------- |-------------:|------------:|------------:|-------:|--------:|
| 'System.Drawing Gif' | Gif/cheers.gif | 417.4 us | 8.19 us | 13.68 us | 1.00 | 0.00 |
| 'ImageSharp Gif' | Gif/cheers.gif | 258,576.9 us | 5,151.66 us | 5,512.22 us | 623.78 | 32.37 |
| | | | | | | |
| 'System.Drawing Gif' | Gif/leo.gif | 110.1 us | 2.20 us | 5.14 us | 1.00 | 0.00 |
| 'ImageSharp Gif' | Gif/leo.gif | 23,045.7 us | 370.19 us | 309.12 us | 210.78 | 12.16 |
PR
| Method | TestImage | Mean | Error | StdDev | Ratio | RatioSD |
|--------------------- |--------------- |-------------:|------------:|-------------:|-------:|--------:|
| 'System.Drawing Gif' | Gif/cheers.gif | 417.7 us | 8.26 us | 10.45 us | 1.00 | 0.00 |
| 'ImageSharp Gif' | Gif/cheers.gif | 405,657.8 us | 6,656.34 us | 10,748.74 us | 977.41 | 30.18 |
| | | | | | | |
| 'System.Drawing Gif' | Gif/leo.gif | 109.2 us | 2.18 us | 4.74 us | 1.00 | 0.00 |
| 'ImageSharp Gif' | Gif/leo.gif | 30,104.5 us | 348.21 us | 325.72 us | 276.99 | 14.94 |
DecodeEncodeGif_DefaultEncoder
release/3.1.x
| Method | TestImage | Mean | Error | StdDev | Ratio | RatioSD |
|-------------------------------- |--------------- |-----------:|----------:|----------:|------:|--------:|
| SystemDrawing | Gif/cheers.gif | 7.140 ms | 0.1384 ms | 0.2732 ms | 1.00 | 0.00 |
| ImageSharp_DefaultEncoder | Gif/cheers.gif | 426.230 ms | 7.9217 ms | 7.4100 ms | 58.02 | 2.29 |
| | | | | | | |
| SystemDrawing | Gif/leo.gif | 2.021 ms | 0.0390 ms | 0.0571 ms | 1.00 | 0.00 |
| ImageSharp_DefaultEncoder | Gif/leo.gif | 35.703 ms | 0.5295 ms | 0.4953 ms | 17.91 | 0.67 |
PR
| Method | TestImage | Mean | Error | StdDev | Ratio | RatioSD |
|-------------------------------- |--------------- |-----------:|-----------:|-----------:|------:|--------:|
| SystemDrawing | Gif/cheers.gif | 7.247 ms | 0.1412 ms | 0.2279 ms | 1.00 | 0.00 |
| ImageSharp_DefaultEncoder | Gif/cheers.gif | 586.890 ms | 11.1193 ms | 20.0504 ms | 81.43 | 2.99 |
| | | | | | | |
| SystemDrawing | Gif/leo.gif | 2.142 ms | 0.0428 ms | 0.0761 ms | 1.00 | 0.00 |
| ImageSharp_DefaultEncoder | Gif/leo.gif | 42.617 ms | 0.8244 ms | 0.8097 ms | 20.28 | 1.24 |
Ideas:
- Given that Quality loss when quantizing an image that already has (fewer than) 256 colors #2862 complains about quality loss for images with smaller palettes, I wonder if it would make sense to dynamically choose between Coarse and Hybrid depending on the palette size?
- If the former idea is not feasible for whatever reason, shouldn't we keep
ColorMatchingMode.Coarse
as default, at least for 3.1? Note that Quality loss when quantizing an image that already has (fewer than) 256 colors #2862 seems to be the only complaint about the quality gap which makes me think that there might be potentially more users worried about performance.
src/ImageSharp/Processing/Processors/Quantization/EuclideanPixelMap{TPixel,TCache}.cs
Show resolved
Hide resolved
where TPixel : unmanaged, IPixel<TPixel> => colorMatchingMode switch | ||
{ | ||
ColorMatchingMode.Hybrid => new EuclideanPixelMap<TPixel, HybridCache>(configuration, palette, transparentIndex), | ||
ColorMatchingMode.Exact => new EuclideanPixelMap<TPixel, NullCache>(configuration, palette, transparentIndex), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ColorMatchingMode.Exact => new EuclideanPixelMap<TPixel, NullCache>(...)
Nit: This is a bit confusing. Does ColorMatchingMode.Exact
mean that there is no caching? Shouldn't then we rename ExactCache
to something like AccurateCache
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, it might be good to add common EuclideanPixelMap<TPixel, TCache>
instantiations to AotCompilerTools
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, Exact means no cache. Very slow but the results are based on distance matching only. I could rename the internal cache instance if that makes thing clearer. I'll also add to the compiler tools.
Thanks @antonfirsov that's very much appreciated! What I was trying to do by using Hybrid as default was to remove any surprises using the default pipeline, however with the introduction of explicit options that provides clear indication of the default behavior and the power for users to override. I'll default back to coarse caching to maintain current performance. Using explicit options is the best approach IMO as the length of the palette doesn't directly affect the accuracy of matching. 256 distinct colors would lead to less collisions than 256 shades of gray using coarse caching. |
@antonfirsov That's the additional changes made now. There's something weird going on when the coarse cache is enabled. I'm getting less frames than I should which suggests some kind of introduced memory corruption. Actually, the encoded PNG seems fine, it's our decoder that's messing something up. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
private readonly IMemoryOwner<short> bucketsOwner; | ||
private MemoryHandle bucketsHandle; | ||
private short* buckets; | ||
|
||
// Entries array: stores up to 256 entries. | ||
private readonly IMemoryOwner<Entry> entriesOwner; | ||
private MemoryHandle entriesHandle; | ||
private Entry* entries; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thoughts for future optimization:
I think it's possible to speed up the AccurateCache by replacing the buckets -> entries
double dispatch logic and the linked-list collision management with a single contigous buffer where a constant number of entries is designated for each bucket. Entries belonging to the same bucked would be layed out adjacently in the memory which would improve memory locality. (This would use more memory, but it would be still much less than what CoarseCache is using.)
Moreover, both the coarse and the accurate cache has an index calculation step (GetCoarseIndex
and the (key >> 16) ^ (key >> 8) ^ key) & 0x1FF
thingie) which can be pre-caluclated for batches of pixels using SIMD. This would require tweaks on the Quantizer API though.
My assumption is that if we would implement both of these changes it would greatly improve quantizer perf, however this assumption has to be proven by profiling first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Intrigued by the AccurateCache
suggestions....
I'm not sure how we could pre-batch the keys though given that they could be any uint
value represented by packed Rgba32
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we had a bulk quantization overload like:
public interface IQuanizer<TPixel> {
public QuantizeColors(ReadOnlySpan<TPixel> pixels, Span<byte> output);
}
We could first do all the TPixel -> index
mappings into a buffer, and then use the indices to look up colors in the second step. Actually, even the second step could be partially vectorized thanks to AVX2 & AVX512 scatter/gather instructions, e.g. VPGATHERDD/_mm256_i32gather_epi32
(Avx2.GatherVector256
).
I believe this could do magic to quantizer perf, therefore I recommend to stick with a SIMD-friendly cache algorithms in #2894, so we can vectorize them later. Personally, I prefer some memory penalty over the speed penalty since I think our current quantization is very slow. We need to aim for speeding it up without hurting quality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I see how that would work. I had no idea such a method existed!!
I'm not sure our quantization is actually all that slow though. System.Drawing seems to pass-through the image when it's multi-framed if its unchanged so comparing it against that is not a fair test without performing some sort of operation on the individual frames first.
You'll find that if we benchmark a single frame GIF the results are much closer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah System.Drawing numbers seem to be fake to me as well. We should rather benchmark against the whatever library that powers Chromium. I may be wrong, but based on what is possible, unfortunately, I don't think our numbers will be close.
Prerequisites
Description
Fixes #2866
Fixes #2862
There's quite a lot going on here:
I'll have to manually rework this for V4 as the code base has migrated significantly there.