Thread Deadlock when placing

[JanCantCode]

Server Implementation

Paper

Server Version

1.21.11

Describe the bug

We are running the newest version of your FAWE fork on canvas 1.21.11, in combination with a plugin that uses the FAWE api to frequently place certain schematics in the world (fixing stuff that gets broken in the pvp area)
This works fine for a certain amount of time, but after a few hours FAWE becomes completely unresponsive, i.e most commands stop working without any error being thrown. this is becaues the Threads responsible for placing chunks in the world appear to be running into a deadlock, where they are waiting for some other thread to free up a chunk section.

I have taken the time to create a thread dump but due to the nature of StampedLock's, it doesnt give much information in regards to who is holding the relevant locks.

To Reproduce

I have not yet attempted to create a setup where this behaviour can be reproduced consistently, but i am assuming its related to a race condition that occurs when multiple invocations of the FAWE api in an async context overlap and attempt to touch the same regions? I have search our logs for FAWE related exceptions that would imply that some exception in the critical code of the section swapping causes the lock to not be released, but have found nothing.

Expected behaviour

FAWE should not deadlock itsself.

Screenshots / Videos

No response

Error log (if applicable)

No response

Fawe Debugpaste

abc

Are you interested in contributing a fix?

Yes

[R00tB33rMan]

What does your implementation look like? I only ask, since we never had issues with FAWE relative to straight-up locking when the actual schematic implementation in the actual plugin is improperly done; however, this does point out to a degree the source of the issue. I have a decent sum of prevalent issues to resolve, I'm afraid, so I'll likely need to get to this quite soon and patch this up, on top of:

• //regen failing on 1.21.11 Folia/Canvas
• -e tags fail to copy entities across regions
• Lighting/region thread crash in extremely race conditions for chunk buffer filling
• (And now this)

We'll push some changes as soon as we can to address your issue and the other two when I'm free!

[JanCantCode]

What does your implementation look like

We use a public plugin I wrote, called AsyncArenas that allows you to define an "arena", for which a schematic-like file is saved in a folder and then read at runtime into FAWE-clipboards, and then placed at intervals declared in a config. notably, we have multiple schematics that overlap with each other resetting simoultaniously. The actual schematic placement implementation looks as follows:

Bukkit.getAsyncScheduler()
            .runNow(
               AsyncArenas.getInstance(),
               task -> {
                  AsyncArenas.getInstance().getLogger().info("Regenerating Arena: " + name);
                  World weditWorld = holder.world();
                  EditSession session = WorldEdit.getInstance().newEditSession(weditWorld);

                  try {
                     Operation operation = new com.sk89q.worldedit.session.ClipboardHolder(holder.clipboard())
                        .createPaste(session)
                        .to(holder.clipboard().getMinimumPoint())
                        .build();
                     Operations.complete(operation);
                  } catch (Throwable var10) {
                     if (session != null) {
                        try {
                           session.close();
                        } catch (Throwable var9) {
                           var10.addSuppressed(var9);
                        }
                     }

                     throw var10;
                  }

                  if (session != null) {
                     session.close();
                  }

                  AsyncArenas.getInstance().getLogger().info("Successfully Regenerated Arena: " + name);

                  for (Runnable callback : callbacks) {
                     Bukkit.getGlobalRegionScheduler().run(AsyncArenas.getInstance(), scheduledTask -> callback.run());
                  }
               }
            );

this is running bukkits async scheduler, and isnt scheduled using fawes task manager (but iirc Operations#complete delegates that somewhere down the line). Also notably after modifying our configuration in a way that guarantees all schematic placements that overlap in the same area are delayed relative to each other by atleast a few seconds, FAWE hasnt deadlocked after 5 hours of uptime, an extent of time that would usually result it in deadlocking, which would confirm the theory of the deadlock being related to that. thanks in advance for a patch whenever, this is a great fork
(also we used to get the chunk buffer crash every 1-2 days, but that abruptly stopped happening a few weeks ago)