fix: add Windows-specific LMDB cleanup handling #1144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft

jaelliot wants to merge 5 commits into WebOfTrust:main from jaelliot:jay/fix-windows-lmdb-cleanup

.github/workflows/python-app-ci.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -20,10 +20,10 @@ jobs:
  
        steps:

          - uses: actions/checkout@v4

          - name: Set up Python 3.12.6+

          - name: Set up Python 3.14.2

            uses: actions/setup-python@v5

            with:

              python-version: '>=3.12.6'

              python-version: '3.14.2'

          - name: Install libsodium (macOS)

            if: runner.os == 'macOS'

    @@ -37,6 +37,16 @@ jobs:
  
              echo "DYLD_FALLBACK_LIBRARY_PATH=$(brew --prefix)/lib:/usr/local/lib:/usr/lib" >> "$GITHUB_ENV"

              echo "DYLD_LIBRARY_PATH=$(brew --prefix)/lib" >> "$GITHUB_ENV"

          - name: Install libsodium (Windows)

            if: runner.os == 'Windows'

            run: |

              $url = "https://download.libsodium.org/libsodium/releases/libsodium-1.0.20-stable-msvc.zip"

              $output = "$env:TEMP\libsodium.zip"

              Invoke-WebRequest -Uri $url -OutFile $output

              Expand-Archive -Path $output -DestinationPath "$env:TEMP\libsodium"

              Copy-Item "$env:TEMP\libsodium\libsodium\x64\Release\v143\dynamic\libsodium.dll" -Destination "$env:windir\System32\libsodium.dll"

            shell: pwsh

          - name: Install dependencies

            run: |

              python -m pip install --upgrade pip

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -110,6 +110,7 @@ venv/ @@
     ENV/
     env.bak/
     venv.bak/
+    venv-windows
     # Spyder project settings
     .spyderproject
@@ Expand Down @@

src/keri/db/dbing.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -45,11 +45,14 @@ @@
     """
+    import gc
     import os
     import platform
     import shutil
     import stat
+    import sys
     import tempfile
+    import time
     from collections import abc
     from contextlib import contextmanager
     from typing import Union
@@ Expand Down Expand Up / @@ -460,17 +463,58 @@ def version(self, val): @@
         def close(self, clear=False):
             """
             Close lmdb at .env and if clear or .temp then remove lmdb directory at .path
             Parameters:
                clear is boolean, True means clear lmdb directory
+            Note:
+                On Windows, LMDB uses mandatory file locks that may persist briefly
+                after env.close(). We add explicit sync and retry logic to ensure
+                locks are released before attempting directory cleanup.
             """
             if self.env:
                 try:
+                    # Explicitly sync to ensure all writes are flushed
+                    self.env.sync(force=True)
                     self.env.close()
-                except:
+                    # Give Windows time to release file handles immediately after close
+                    if sys.platform == 'win32':
+                        time.sleep(0.01)  # 10ms initial delay
+                except Exception:
                     pass
             self.env = None
+            # Windows-specific: Retry cleanup with exponential backoff
+            # Windows holds file locks longer than Unix, causing PermissionError
+            if clear and sys.platform == 'win32':
+                # Force garbage collection to release any lingering file handles
+                gc.collect()
+                # Windows with clear=True: Retry with exponential backoff
+                max_retries = 10
+                base_delay = 0.1  # Start with 100ms
+                last_error = None
+                for attempt in range(max_retries):
+                    try:
+                        return super(LMDBer, self).close(clear=clear)
+                    except (PermissionError, OSError) as e:
+                        last_error = e
+                        # On Windows, any PermissionError or OSError during cleanup
+                        # is likely a file lock issue - retry with backoff
+                        if attempt < max_retries - 1:
+                            # Exponential backoff: 100ms, 200ms, 400ms, 800ms, 1.6s...
+                            delay = base_delay * (2 ** attempt)
+                            time.sleep(delay)
+                            # Force garbage collection between retries
+                            gc.collect()
+                        else:
+                            # Final attempt failed, re-raise the last error
+                            raise last_error
+                # Shouldn't reach here but satisfy type checker
+                return False  # type: ignore[unreachable]
+            # Non-Windows or not clearing, no retry needed
             return super(LMDBer, self).close(clear=clear)
         def getVer(self):
@@ Expand Down @@

tests/app/test_forwarding.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -19,14 +19,25 @@
  
    from keri.peer import exchanging

    from keri.spac import payloading

    # Import Windows port utilities for cross-platform testing

    try:

        from .windows_ports import get_available_port

    except ImportError:

        # Fallback if windows_ports module not available

        def get_available_port(port):

            return port

    def test_postman(seeder):

        with habbing.openHab(name="test", transferable=True, temp=True) as (hby, hab), \

                habbing.openHby(name="wes", salt=core.Salter(raw=b'wess-the-witness').qb64, temp=True) as wesHby, \

                habbing.openHby(name="repTest", temp=True) as recpHby:

            mbx = storing.Mailboxer(name="wes", temp=True)

            wesDoers = indirecting.setupWitness(alias="wes", hby=wesHby, mbx=mbx, tcpPort=5634, httpPort=5644)

            # Use Windows-compatible ports

            tcp_port = get_available_port(5634)

            http_port = get_available_port(5644)

            wesDoers = indirecting.setupWitness(alias="wes", hby=wesHby, mbx=mbx, tcpPort=tcp_port, httpPort=http_port)

            wesHab = wesHby.habByName("wes")

            seeder.seedWitEnds(hby.db, witHabs=[wesHab])

            seeder.seedWitEnds(wesHby.db, witHabs=[wesHab])

    @@ -96,7 +107,9 @@ def test_essr_stream(seeder):
  
            app = falcon.App()

            httpEnd = indirecting.HttpEnd(rxbs=recpHab.psr.ims)

            app.add_route("/", httpEnd)

            server = http.Server(port=5555, app=app)

            # Use Windows-compatible port

            http_port = get_available_port(5555)

            server = http.Server(port=http_port, app=app)

            httpServerDoer = http.ServerDoer(server=server)

            kvy = eventing.Kevery(db=hab.db)

    @@ -115,7 +128,7 @@ def test_essr_stream(seeder):
  
                                            role=kering.Roles.controller,

                                            stamp=help.nowIso8601()))

            msgs.extend(recpHab.makeLocScheme(url='http://127.0.0.1:5555',

            msgs.extend(recpHab.makeLocScheme(url=f'http://127.0.0.1:{http_port}',

                                              scheme=kering.Schemes.http,

                                              stamp=help.nowIso8601()))

            hab.psr.parse(ims=msgs)

    @@ -197,7 +210,10 @@ def test_essr_mbx(seeder):
  
                habbing.openHby(name="repTest", temp=True) as recpHby:

            mbx = storing.Mailboxer(name="wes", temp=True)

            wesDoers = indirecting.setupWitness(alias="wes", hby=wesHby, mbx=mbx, tcpPort=5634, httpPort=5644)

            # Use Windows-compatible ports

            tcp_port = get_available_port(5634)

            http_port = get_available_port(5644)

            wesDoers = indirecting.setupWitness(alias="wes", hby=wesHby, mbx=mbx, tcpPort=tcp_port, httpPort=http_port)

            wesHab = wesHby.habByName("wes")

            seeder.seedWitEnds(hby.db, witHabs=[wesHab])

            seeder.seedWitEnds(wesHby.db, witHabs=[wesHab])

tests/app/windows_ports.py

-Original file line number
+Diff line change
@@ -0,0 +1,92 @@
+    # -*- encoding: utf-8 -*-
+    """
+    Windows-specific utilities for testing
+    """
+    import socket
+    import sys
+    def find_available_port(start_port=8000, max_attempts=100):
+        """
+        Find an available port for Windows testing
+        Windows blocks access to ports in the 5000-5999 range with WinError 10013.
+        This function finds an available port starting from 8000.
+        Args:
+            start_port (int): Port to start searching from (default: 8000)
+            max_attempts (int): Maximum number of ports to try
+        Returns:
+            int: Available port number
+        Raises:
+            RuntimeError: If no available port is found
+        """
+        for port in range(start_port, start_port + max_attempts):
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            try:
+                sock.bind(('127.0.0.1', port))
+                sock.close()
+                return port
+            except socket.error:
+                sock.close()
+                continue
+        raise RuntimeError(f"No available ports found in range {start_port}-{start_port + max_attempts}")
+    def get_test_ports(count=5):
+        """
+        Get a list of available ports for testing
+        Args:
+            count (int): Number of ports to find
+        Returns:
+            list[int]: List of available port numbers
+        """
+        ports = []
+        start_port = 8000
+        for i in range(count):
+            port = find_available_port(start_port + len(ports))
+            ports.append(port)
+        return ports
+    # Windows-specific port mappings for tests
+    if sys.platform == 'win32':
+        WINDOWS_TEST_PORTS = get_test_ports(10)
+        # Map commonly used test ports to available Windows ports
+        PORT_MAPPING = {
+: WINDOWS_TEST_PORTS[0],  # test_essr_stream uses this
+: WINDOWS_TEST_PORTS[1],  # test_essr_mbx tcpPort
+: WINDOWS_TEST_PORTS[2],  # test_essr_mbx httpPort
+: WINDOWS_TEST_PORTS[3],  # other witness ports
+: WINDOWS_TEST_PORTS[4],  # other witness ports
+: WINDOWS_TEST_PORTS[5],  # other witness ports
+: WINDOWS_TEST_PORTS[6],  # other witness ports
+: WINDOWS_TEST_PORTS[7],  # default witness ports
+: WINDOWS_TEST_PORTS[8],  # other witness ports
+: WINDOWS_TEST_PORTS[9],  # other witness ports
+        }
+    else:
+        PORT_MAPPING = {}
+    def get_available_port(requested_port):
+        """
+        Get an available port, using Windows mapping if needed
+        Args:
+            requested_port (int): The port originally requested
+        Returns:
+            int: Available port (mapped on Windows, original on other platforms)
+        """
+        if sys.platform == 'win32' and requested_port in PORT_MAPPING:
+            return PORT_MAPPING[requested_port]
+        return requested_port

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: add Windows-specific LMDB cleanup handling #1144

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

fix: add Windows-specific LMDB cleanup handling #1144

Are you sure you want to change the base?

Uh oh!

fix: add Windows-specific LMDB cleanup handling #1144

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!