Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3 : Fix GIF, PNG, and WEBP Edge Case Handling #2882

Merged
merged 29 commits into from
Mar 27, 2025

Conversation

JimBobSquarePants
Copy link
Member

@JimBobSquarePants JimBobSquarePants commented Feb 7, 2025

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

Fixes #2866
Fixes #2862

There's quite a lot going on here:

  • Both GIF and WEBP decoders were not handling frame disposal properly.
  • GIF Decoder background color handling was incorrect.
  • GIF Encoder incorrectly used global palette for local root frame.
  • WEBP Decoder did not clear some buffers on load where it should.
  • PNG Encoder palette animations did not work properly
  • Fixed pixel sampling in Wu and Octree quantizers
  • Improved accuracy for matching first 512 colors in EuclidianPixelMap.

I'll have to manually rework this for V4 as the code base has migrated significantly there.

Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 5 out of 16 changed files in this pull request and generated no comments.

Files not reviewed (11)
  • tests/Images/External/ReferenceOutput/GifDecoderTests/Issue1962_Rgba32_issue1962_tiniest_gif_1st.png: Language not supported
  • tests/Images/External/ReferenceOutput/GifDecoderTests/Issue2012BadMinCode_Rgba32_issue2012_drona1.png: Language not supported
  • tests/Images/Input/Gif/issues/issue_2866.gif: Language not supported
  • tests/Images/Input/Webp/issues/Issue2866.webp: Language not supported
  • src/ImageSharp/Formats/Gif/GifEncoderCore.cs: Evaluated as low risk
  • tests/ImageSharp.Tests/TestUtilities/ReferenceCodecs/MagickReferenceDecoder.cs: Evaluated as low risk
  • src/ImageSharp/Formats/Webp/Lossy/Vp8Decoder.cs: Evaluated as low risk
  • src/ImageSharp/Formats/Webp/BitReader/BitReaderBase.cs: Evaluated as low risk
  • tests/ImageSharp.Tests/Formats/Png/PngEncoderTests.cs: Evaluated as low risk
  • tests/ImageSharp.Tests/TestImages.cs: Evaluated as low risk
  • src/ImageSharp/Formats/Webp/WebpBlendMethod.cs: Evaluated as low risk
Comments suppressed due to low confidence (2)

tests/ImageSharp.Tests/Formats/Gif/GifEncoderTests.cs:389

  • [nitpick] The test method name GifEncoder_CanDecode_Issue2866 is misleading since it is testing encoding, not decoding. It should be renamed to GifEncoder_CanEncode_Issue2866.
public void GifEncoder_CanDecode_Issue2866<TPixel>(TestImageProvider<TPixel> provider)

src/ImageSharp/Formats/Webp/WebpAnimationDecoder.cs:89

  • The comment should reflect the correct type for backgroundColor, which is TPixel instead of Color.
/// <param name="backgroundColor">The default background color of the canvas in.</param>

@JimBobSquarePants JimBobSquarePants changed the title V3 : Fix GIF and WEBP Edge Case Handling V3 : Fix GIF, PNG, and WEBP Edge Case Handling Feb 25, 2025
@antonfirsov
Copy link
Member

This is quite a huge change, but I'm planning to take a high-level look in the coming days.

@JimBobSquarePants
Copy link
Member Author

This is quite a huge change, but I'm planning to take a high-level look in the coming days.

Thanks! I have a V4 version also that I’m going to push, lots of similarities but also some additional changes.

@brianpopow
Copy link
Collaborator

@JimBobSquarePants I think it would be good to add a few more test case to verify everything works as expected. Here is a github repository with some gif images generated with imagemagick: gif-test-suite, which I think could be useful testcases.

I can help with adding tests, I have some time in the next few days.

@JimBobSquarePants
Copy link
Member Author

That would be awesome if you could thanks!

@brianpopow
Copy link
Collaborator

brianpopow commented Mar 3, 2025

With some of test images I have added with f80aa76, the output of ImageSharp does not match what imagemagick would produce, which may indicate an error.
For example, the image:
animated_transparent_frame_restoreprev_loop
animated_transparent_frame_restoreprev_loop.gif

The output from ImageSharp looks like this:

output_0

output_1

output_2

output_3

I am not sure yet why. I will try to figure out what's wrong tomorrow.
Also notice: It could be not related to changes introduced with this PR.

To extract the images with imagemagick, the following command can be used:

 magick convert -coalesce animated_transparent_frame_restoreprev_loop.gif[0-4] out.png

@JimBobSquarePants
Copy link
Member Author

@brianpopow That looks like something to do with palette reading though the last frame suggests an additional bug.

@brianpopow
Copy link
Collaborator

@brianpopow That looks like something to do with palette reading though the last frame suggests an additional bug.

@JimBobSquarePants It seems that the issue is a regression from version 2.1.9 to 3.0.0. It is an issue with handling the disposal method RestoreToPrevious. When I change it back to how it was in 2.1.9, the images look again like expected in the ActualOutput folder.

The weird thing is that when I save the frames like this:

var img = Image.Load<Rgba32>(imagePath);
for (int i = 0; i < img.Frames.Count; i++)
{
    using Image frameImage = img.Frames.CloneFrame(i);

    frameImage.Save($"output_{i}.png");
}

The images still look black and white. I am not sure what I do wrong here with saving the frames.

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 6, 2025

@brianpopow That looks like something to do with palette reading though the last frame suggests an additional bug.

@JimBobSquarePants It seems that the issue is a regression from version 2.1.9 to 3.0.0. It is an issue with handling the disposal method RestoreToPrevious. When I change it back to how it was in 2.1.9, the images look again like expected in the ActualOutput folder.

The weird thing is that when I save the frames like this:

var img = Image.Load<Rgba32>(imagePath);
for (int i = 0; i < img.Frames.Count; i++)
{
    using Image frameImage = img.Frames.CloneFrame(i);

    frameImage.Save($"output_{i}.png");
}

The images still look black and white. I am not sure what I do wrong here with saving the frames.

I can fix restore to previous behavior easily enough but I'm seeing some odd behavior in ImageMagick for this image and others. It doesn't seem to follow the gif89a specification at all regarding GCE transparency and is setting entries at a given index with an alpha of zero regardless of whether there is a transparency flag. @dlemstra Any idea why this is?

@JimBobSquarePants
Copy link
Member Author

@brianpopow I figured out why the output is wrong when cloning. The current code attempts to encode the image using the palette data and bit depth extracted from the individual frames however the new frame contains all the data from the combined frame not just te delta. We shouldn't attempt to preserve the palette when converting between formats. I'll push a fix for this today to this branch.

@brianpopow
Copy link
Collaborator

@brianpopow I figured out why the output is wrong when cloning. The current code attempts to encode the image using the palette data and bit depth extracted from the individual frames however the new frame contains all the data from the combined frame not just te delta. We shouldn't attempt to preserve the palette when converting between formats. I'll push a fix for this today to this branch.

@JimBobSquarePants: I am glad you found the issue, I had trouble making sense out of this behavior.

@SpaceCheetah
Copy link
Contributor

SpaceCheetah commented Mar 21, 2025

Seems the PNG encoder still has trouble for multi-frame cases with "previous" dispose mode. Should make sure it works for all blend modes too, or change the blend mode so it does. Here's a case where the encoder fails, but the decoder succeeds:
Input:
test
Encoder output:
Encode_APng_test

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 24, 2025

Seems the PNG encoder still has trouble for multi-frame cases with "previous" dispose mode. Should make sure it works for all blend modes too, or change the blend mode so it does. Here's a case where the encoder fails, but the decoder succeeds: Input: test Encoder output: Encode_APng_test

@SpaceCheetah Fixed with 71357d2

Encode_APng_Issue2882_Issue_2882

Copy link
Member

@antonfirsov antonfirsov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the long delay. This is great work and clearly an improvement over the current state. Correct me if I'm wrong, but in my understanding this PR is dealing with 2 relatively separate concerns: (1) fixing edge cases including #2866, (2) preventing quality loss to address #2862.

I have no concerns with (1), but I think there might be some space for improvements regarding (2).

In 4ace814 I did some benchmark extensions. While the original Gif benchmark is targeted at comparing ImageSharp to System.Drawing, benchmarking with a custom decoder setup isn't very representative when we want to track impactful performance changes within the library, since most users will likely go with the default decoder config. For that reason the commit splits the benchmark into two, where EncodeGif_DefaultEncoder benchmarks the default encoder performance. I also added DecodeEncodeGif_* to have benchmark that approximates the typical end-to-end user scenario.

These benchmarks show ~50% regression for EncodeGif_DefaultEncoder and 20%-40% regression for DecodeEncodeGif_DefaultEncoder which would land in a 3.1.* revision update.

Config

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.26100
Intel Core i9-10900K CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=9.0.201
  [Host]     : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT
  DefaultJob : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT

EncodeGif_DefaultEncoder

release/3.1.x

|               Method |      TestImage |         Mean |       Error |      StdDev |  Ratio | RatioSD |
|--------------------- |--------------- |-------------:|------------:|------------:|-------:|--------:|
| 'System.Drawing Gif' | Gif/cheers.gif |     417.4 us |     8.19 us |    13.68 us |   1.00 |    0.00 |
|     'ImageSharp Gif' | Gif/cheers.gif | 258,576.9 us | 5,151.66 us | 5,512.22 us | 623.78 |   32.37 |
|                      |                |              |             |             |        |         |
| 'System.Drawing Gif' |    Gif/leo.gif |     110.1 us |     2.20 us |     5.14 us |   1.00 |    0.00 |
|     'ImageSharp Gif' |    Gif/leo.gif |  23,045.7 us |   370.19 us |   309.12 us | 210.78 |   12.16 |

PR

|               Method |      TestImage |         Mean |       Error |       StdDev |  Ratio | RatioSD |
|--------------------- |--------------- |-------------:|------------:|-------------:|-------:|--------:|
| 'System.Drawing Gif' | Gif/cheers.gif |     417.7 us |     8.26 us |     10.45 us |   1.00 |    0.00 |
|     'ImageSharp Gif' | Gif/cheers.gif | 405,657.8 us | 6,656.34 us | 10,748.74 us | 977.41 |   30.18 |
|                      |                |              |             |              |        |         |
| 'System.Drawing Gif' |    Gif/leo.gif |     109.2 us |     2.18 us |      4.74 us |   1.00 |    0.00 |
|     'ImageSharp Gif' |    Gif/leo.gif |  30,104.5 us |   348.21 us |    325.72 us | 276.99 |   14.94 |

DecodeEncodeGif_DefaultEncoder

release/3.1.x

|                          Method |      TestImage |       Mean |     Error |    StdDev | Ratio | RatioSD |
|-------------------------------- |--------------- |-----------:|----------:|----------:|------:|--------:|
|                   SystemDrawing | Gif/cheers.gif |   7.140 ms | 0.1384 ms | 0.2732 ms |  1.00 |    0.00 |
|       ImageSharp_DefaultEncoder | Gif/cheers.gif | 426.230 ms | 7.9217 ms | 7.4100 ms | 58.02 |    2.29 |
|                                 |                |            |           |           |       |         |
|                   SystemDrawing |    Gif/leo.gif |   2.021 ms | 0.0390 ms | 0.0571 ms |  1.00 |    0.00 |
|       ImageSharp_DefaultEncoder |    Gif/leo.gif |  35.703 ms | 0.5295 ms | 0.4953 ms | 17.91 |    0.67 |

PR

|                          Method |      TestImage |       Mean |      Error |     StdDev | Ratio | RatioSD |
|-------------------------------- |--------------- |-----------:|-----------:|-----------:|------:|--------:|
|                   SystemDrawing | Gif/cheers.gif |   7.247 ms |  0.1412 ms |  0.2279 ms |  1.00 |    0.00 |
|       ImageSharp_DefaultEncoder | Gif/cheers.gif | 586.890 ms | 11.1193 ms | 20.0504 ms | 81.43 |    2.99 |
|                                 |                |            |            |            |       |         |
|                   SystemDrawing |    Gif/leo.gif |   2.142 ms |  0.0428 ms |  0.0761 ms |  1.00 |    0.00 |
|       ImageSharp_DefaultEncoder |    Gif/leo.gif |  42.617 ms |  0.8244 ms |  0.8097 ms | 20.28 |    1.24 |

Ideas:

where TPixel : unmanaged, IPixel<TPixel> => colorMatchingMode switch
{
ColorMatchingMode.Hybrid => new EuclideanPixelMap<TPixel, HybridCache>(configuration, palette, transparentIndex),
ColorMatchingMode.Exact => new EuclideanPixelMap<TPixel, NullCache>(configuration, palette, transparentIndex),
Copy link
Member

@antonfirsov antonfirsov Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ColorMatchingMode.Exact => new EuclideanPixelMap<TPixel, NullCache>(...)

Nit: This is a bit confusing. Does ColorMatchingMode.Exact mean that there is no caching? Shouldn't then we rename ExactCache to something like AccurateCache?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it might be good to add common EuclideanPixelMap<TPixel, TCache> instantiations to AotCompilerTools.

Copy link
Member Author

@JimBobSquarePants JimBobSquarePants Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, Exact means no cache. Very slow but the results are based on distance matching only. I could rename the internal cache instance if that makes thing clearer. I'll also add to the compiler tools.

@JimBobSquarePants
Copy link
Member Author

Thanks @antonfirsov that's very much appreciated!

What I was trying to do by using Hybrid as default was to remove any surprises using the default pipeline, however with the introduction of explicit options that provides clear indication of the default behavior and the power for users to override. I'll default back to coarse caching to maintain current performance.

Using explicit options is the best approach IMO as the length of the palette doesn't directly affect the accuracy of matching. 256 distinct colors would lead to less collisions than 256 shades of gray using coarse caching.

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 26, 2025

@antonfirsov That's the additional changes made now.

There's something weird going on when the coarse cache is enabled. I'm getting less frames than I should which suggests some kind of introduced memory corruption.

Actually, the encoded PNG seems fine, it's our decoder that's messing something up.

Copy link
Member

@antonfirsov antonfirsov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Comment on lines +216 to +223
private readonly IMemoryOwner<short> bucketsOwner;
private MemoryHandle bucketsHandle;
private short* buckets;

// Entries array: stores up to 256 entries.
private readonly IMemoryOwner<Entry> entriesOwner;
private MemoryHandle entriesHandle;
private Entry* entries;
Copy link
Member

@antonfirsov antonfirsov Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts for future optimization:

I think it's possible to speed up the AccurateCache by replacing the buckets -> entries double dispatch logic and the linked-list collision management with a single contigous buffer where a constant number of entries is designated for each bucket. Entries belonging to the same bucked would be layed out adjacently in the memory which would improve memory locality. (This would use more memory, but it would be still much less than what CoarseCache is using.)

Moreover, both the coarse and the accurate cache has an index calculation step (GetCoarseIndex and the (key >> 16) ^ (key >> 8) ^ key) & 0x1FF thingie) which can be pre-caluclated for batches of pixels using SIMD. This would require tweaks on the Quantizer API though.

My assumption is that if we would implement both of these changes it would greatly improve quantizer perf, however this assumption has to be proven by profiling first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intrigued by the AccurateCache suggestions....

I'm not sure how we could pre-batch the keys though given that they could be any uint value represented by packed Rgba32.

Copy link
Member

@antonfirsov antonfirsov Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we had a bulk quantization overload like:

public interface IQuanizer<TPixel> {
    public QuantizeColors(ReadOnlySpan<TPixel> pixels, Span<byte> output);
}

We could first do all the TPixel -> index mappings into a buffer, and then use the indices to look up colors in the second step. Actually, even the second step could be partially vectorized thanks to AVX2 & AVX512 scatter/gather instructions, e.g. VPGATHERDD/_mm256_i32gather_epi32 (Avx2.GatherVector256).

I believe this could do magic to quantizer perf, therefore I recommend to stick with a SIMD-friendly cache algorithms in #2894, so we can vectorize them later. Personally, I prefer some memory penalty over the speed penalty since I think our current quantization is very slow. We need to aim for speeding it up without hurting quality.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I see how that would work. I had no idea such a method existed!!

I'm not sure our quantization is actually all that slow though. System.Drawing seems to pass-through the image when it's multi-framed if its unchanged so comparing it against that is not a fair test without performing some sort of operation on the individual frames first.

You'll find that if we benchmark a single frame GIF the results are much closer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah System.Drawing numbers seem to be fake to me as well. We should rather benchmark against the whatever library that powers Chromium. I may be wrong, but based on what is possible, unfortunately, I don't think our numbers will be close.

@JimBobSquarePants JimBobSquarePants merged commit 4a8ff2b into release/3.1.x Mar 27, 2025
8 checks passed
@JimBobSquarePants JimBobSquarePants deleted the js/fix-2866 branch March 27, 2025 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants