ACI SSoT Sync is not atomic, failures are not rolled back.

[kingfetty]

Environment

• Python version: 3.11
• Nautobot version: 2.3.16
• nautobot-ssot version: 3.4.0

Expected Behavior

Upon ACI Sync failure and changes applied should be undone.

Observed Behavior

ACI sync jobs that fail after making changes leave artifacts behind. This can cause unexpected behavior on subsequent syncs.

Steps to Reproduce

1. Configure Nautobot and ACI SSoT
2. Setup a failure scenario by creating an object that will exist in the ACI fabric but do not apply the configured ACI tag to the object
3. Run the sync job, the job will make many changes to the system, then encounter the error and halt. The changes will remain in the system.

[jdrew82]

I think this is a misunderstanding of how SSoT works. The intention is to be idempotent but not necessarily atomic. In Nautobot 1.x the Jobs run() function was wrapped in an atomic wrapper which enabled this functionality but was removed in Nautobot 2.x due to issues with the datasets being too large and causing failures during imports. In order to enable this we'd need to determine at which level we wish to have the changes and then add atomic wrappers everywhere. This also wouldn't necessarily ensure the exact functionality that is expected here.

[kingfetty]

Very good points about the size of the commits. I've encountered several times where sync jobs fail partially leaving artifacts behind that cause further errors on future syncs. Particularly with things like the ACI integration that requires proper tags to be placed on all the objects it touches. If the job fails without applying the tags, you're left with a lot of manual cleanup.

[jdrew82]

@kingfetty I do agree that there is a problem there and I've seen this with other integrations too.