Skip to content

Fix out-of-bounds read in wavelet decomposition on small images (e.g., thumbnails) - #21461

Merged
TurboGit merged 1 commit into
darktable-org:masterfrom
masterpiga:fix_wavelet_oob
Jun 30, 2026
Merged

TurboGit merged 1 commit into
darktable-org:masterfrom
masterpiga:fix_wavelet_oob

Conversation

@masterpiga

Copy link
Copy Markdown
Collaborator

Problem

The à trous wavelet decomposition in common/dwt.c averages each pixel with neighbours hscale pixels to the left/right (and rows above/below), where hscale doubles each scale level. At the borders, out-of-range neighbours are reflected back inside using 2*width-2-(col+hscale).

The horizontal pass capped the scale at MIN(1 << lev, width). When hscale reaches width — a small image (e.g. thumbnail/mipmap export) combined with a high wavelet level — the reflection of the last column underflows to -1. Since the index is an unsigned size_t, it wraps to a huge value and the code reads gigabytes past the buffer → SIGSEGV.

This reliably crashed retouch during thumbnail export (_image_load_job_run → pixelpipe → retouch → dwt_decompose). The vertical pass already capped correctly at height-1; the horizontal pass did not.

Fix

  • dwt_decompose_horiz: cap hscale at width-1, mirroring the vertical pass.
  • dwt_denoise_vert_1ch / dwt_denoise_horiz_1ch: same size-1 cap. The horizontal denoise kernel additionally used three disjoint loops that overlap and double-process columns when 2*hscale > width; reworked the loop bounds into a true non-overlapping partition and reflected the far neighbour in the edge loop. The hot middle loop keeps its vectorized form.
  • OpenCL (dwt.cl + host): clamp sc at size-1 and reflect the far neighbour in the left/top kernel branches, matching the CPU path.

Behaviour is unchanged in the valid range (hscale < width); only the previously-crashing over-scaled case now produces a well-defined max-reflection result.

macOS crash report

crash_report.txt

Co-authored with Claude.

@masterpiga masterpiga added this to the 5.6.1 milestone Jun 30, 2026
@masterpiga masterpiga added bugfix pull request fixing a bug difficulty: trivial some changes in a couple of functions scope: image processing correcting pixels labels Jun 30, 2026
@masterpiga

Copy link
Copy Markdown
Collaborator Author

@TurboGit also in this case, I tentatively set 5.6.1 as the milestone. The bug is not 5.6-related, but the change is a genuine bugfix.

@masterpiga masterpiga added the priority: medium core features are degraded in a way that is still mostly usable, software stutters label Jun 30, 2026

@TurboGit TurboGit left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll merge on master for now and if regression tests are clean tomorrow and my testing later today is good I'll merge for 5.6.1.

@TurboGit
TurboGit merged commit 6af3e2f into darktable-org:master Jun 30, 2026
5 checks passed
@jenshannoschwalm

Copy link
Copy Markdown
Collaborator

Just checked integration tests also for OpenCL with just-now commits. All good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix pull request fixing a bug difficulty: trivial some changes in a couple of functions priority: medium core features are degraded in a way that is still mostly usable, software stutters scope: image processing correcting pixels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants