Fix out-of-bounds write in path mask border gap-filling

[masterpiga]

This is arguably a rare latent bug that I was lucky enough to stumble upon in the middle of an editing session.

Fireworks ensued.

Problem

darktable crashes (SIGSEGV, heap buffer overflow) while moving the mouse during path-mask creation. Backtrace:

_path_get_pts_border  (inlined _path_points_fill_border_gaps)
_path_get_points_border
dt_masks_gui_form_create
_path_events_mouse_moved

Root cause

_path_points_fill_border_gaps() builds an arc between two border points. When the next segment is collapsed — its corner and both control handles essentially coincident — _path_border_get_XY computes a zero derivative and returns DT_INVALID_COORDINATE (-FLT_MAX) for the border point bmax. From there:

1. dt_fast_hypotf() on that point overflows the radius to +Inf.
2. The arc length l = (a2 - a1) * fmaxf(r1, r2) becomes Inf; (int)Inf saturates to INT_MAX.
3. 2*(l-1) overflows int to a small negative value, so dt_masks_dynbuf_reserve_n() skips its buffer-growth check (and rewinds pos) and returns a pointer into the unenlarged buffer.
4. The fill loop for(int i = 1; i < l; i++) then performs ~2.1 billion sequential writes, running off the end of the buffer into unmapped memory.

Why it's rare

The crash needs bmax to be exactly DT_INVALID_COORDINATE, i.e. an exactly-zero segment derivative. The caller deliberately samples the border at t = 0.00001 rather than t = 0 precisely to avoid this: at that offset even an ordinary sharp-corner node (handle on the corner) still yields a small non-zero tangent, so bmax is valid and the arc length stays small. Only a fully degenerate segment (corner ≈ both handles) makes the tangent round to zero. Such segments occur only transiently — e.g. dropping a node almost on top of another, or a node mid-creation before its handles spread — and only with nb >= 3, which gates this code path. Once the precondition is met the overflow fires deterministically; the rarity is entirely in reaching that degenerate geometry.

Proposed fix

In _path_points_fill_border_gaps():

• Bail out early if any input point (bmin/bmax/cmax) is DT_INVALID_COORDINATE. This compares against a finite sentinel, so it holds regardless of fast-math flags.
• Defensive guard: reject a non-finite radius and clamp the arc length to INT_MAX/4 before the int cast, so 2*(l-1) can never overflow.

Co-authored with Claude.

[jenshannoschwalm]

Do you have any idea how that happend in your session?

[masterpiga]

Do you have any idea how that happend in your session?

LOL

Not exactly. Something weird happened, and one edge of a mask was pulled very far away. I don't know how that happened, maybe a wrong gesture/key combo. I just saw for a fraction of a second an extremely stretched segment, and then darktable went kaput.

I can't exclude that the weird thingy that exposed this bug was a bug in itself, but I don't know really. It was all very fast.

[TurboGit]

Thanks!

[TurboGit]

Needs a release note entry. TIA.