Skip to content

Python: enable overlay compilation + extractor overlay support #20337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
d10c merged 15 commits into from
Oct 16, 2025
Merged

Python: enable overlay compilation + extractor overlay support #20337

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
dac50fa
Enable overlay compilation in lib/qlpack.yml
d10c Aug 12, 2025
4c9ca93
Turn on overlay support in codeql-extractor.yml
d10c Aug 11, 2025
a88d339
Add overlay builtins to python dbscheme
d10c Aug 11, 2025
1c3a7f2
Add database upgrade/downgrade scripts
d10c Aug 11, 2025
1574b5f
Add synthetic data to dbscheme.stats for `databaseMetadata`/`overlayC…
d10c Aug 20, 2025
29b1a74
Support CODEQL_PATH_TRANSFORMER env var in python path renamer
d10c Aug 11, 2025
e0cf719
Path transformer: handle Windows-style paths
d10c Sep 10, 2025
49b18db
Python extractor: in overlay mode, traverse only changed files
d10c Aug 11, 2025
6f208e9
Write overlay metadata at end of extraction.
d10c Aug 11, 2025
1a9683f
Add `@top` database type
d10c Oct 1, 2025
7174d4c
Overlay.qll: discard predicates
d10c Aug 18, 2025
20d4e42
Add consistency query (exactly one path for every entity)
d10c Oct 2, 2025
9d7c524
Basic python overlay tests
d10c Oct 2, 2025
ece1210
Add change note.
d10c Oct 2, 2025
c749607
Bump python extractor version to 7.1.5
d10c Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions python/extractor/semmle/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,10 @@ def close(self):
self.module_queue.put(None)
for p in self.procs:
p.join()
if 'CODEQL_EXTRACTOR_PYTHON_OVERLAY_BASE_METADATA_OUT' in os.environ:
with open(os.environ['CODEQL_EXTRACTOR_PYTHON_OVERLAY_BASE_METADATA_OUT'], 'w', encoding='utf-8') as f:
metadata = {}
json.dump(metadata, f)
self.logger.info("Processed %d modules in %0.2fs", len(self.import_graph.done), time.time() - self.start_time)

def stop(self, timeout=2.0):
Expand Down