Split detection engine into pkg/quickdup library, keep cmd/quickdup as CLI - #10
Merged
Conversation
…s CLI The detection engine (parsing, strategies, pattern growth, filtering, caching) moves to pkg/quickdup as an importable library with an exported API (ActiveStrategy, CommentPrefix, DetectPatterns, ParseFilesWithCache, FilterPatterns, plus a single-call AnalyzeFiles convenience wrapper). cmd/quickdup keeps only CLI concerns: flag parsing, terminal rendering/theming, JSON report I/O, and the --compare git-worktree orchestration. Also backports two small fixes proven out in a downstream vendored copy of this engine: - strategy_wordonly.go/strategy_inlineable.go/strategy_normalizedindent.go/ strategy_wordindent.go: dedup near-identical Entry/Strategy boilerplate into shared FirstWordEntry/indentEntry/indentStrategy types (strategy_indent_shared.go). - filter.go: check the ignore.json directory's MkdirAll error before writing, instead of ignoring it. - similarity.go: drop the unused computeAverageTokenSimilarity function. Debug progress reporting (previously an unconditional/debug-gated fmt.Printf) now goes through an injectable ProgressOutput writer so the library doesn't print by default; cmd/quickdup points it at os.Stdout to preserve existing CLI output exactly. Verified: go build/vet/test all pass; smoke-tested the CLI binary (default scan, --debug, --strategy word-indent caching, --compare).
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Refactors core parsing/strategy logic into pkg/quickdup with shared implementations, adds a one-shot AnalyzeFiles API, and updates the CLI to consume the library package.
Changes:
- Added shared first-word/indent strategy plumbing (
firstWordStrategy,indentStrategy) and moved word-only/indent strategies intopkg/quickdup. - Introduced package-level state (
ActiveStrategy,CommentPrefix, progress output) and a serializedAnalyzeFilesconvenience API. - Began migrating the CLI to call into
pkg/quickdupand removed duplicated strategy implementations undercmd/quickdup.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/quickdup/strategy_wordonly.go | Adds word-only strategy backed by shared first-word mechanics. |
| pkg/quickdup/strategy_wordindent.go | Adds word+indent strategy and C-style comment stripping. |
| pkg/quickdup/strategy_normalizedindent.go | Adds normalized-indent strategy and delta normalization. |
| pkg/quickdup/strategy_indent_shared.go | Introduces shared entry/strategy helpers for hashing/signature/scoring and blocked hashes. |
| pkg/quickdup/strategy_firstword_test.go | Adds tests to ensure shared parsing/scoring semantics remain intact. |
| pkg/quickdup/state.go | Introduces package-level state and progress logging hook. |
| pkg/quickdup/detector_test.go | Adds tests for detection/growth/overlap behavior and AnalyzeFiles integration. |
| pkg/quickdup/analyze.go | Adds AnalyzeFiles convenience API with per-extension comment prefix defaults and strategy selection. |
| cmd/quickdup/types.go | Changes package declaration (impacts CLI build layout). |
| cmd/quickdup/strategy_wordonly.go | Removes CLI-local word-only strategy implementation (now in pkg). |
| cmd/quickdup/strategy_wordindent.go | Removes CLI-local word-indent strategy implementation (now in pkg). |
| cmd/quickdup/strategy_normalizedindent.go | Removes CLI-local normalized-indent strategy implementation (now in pkg). |
| cmd/quickdup/strategy_inlineable.go | Switches inlineable strategy to shared first-word entry/strategy behavior. |
| cmd/quickdup/strategy.go | Changes package declaration (impacts CLI build layout). |
| cmd/quickdup/similarity.go | Changes package declaration and removes average token similarity helper. |
| cmd/quickdup/parser.go | Changes package declaration and switches to ActiveStrategy/CommentPrefix globals. |
| cmd/quickdup/output.go | Updates output rendering to accept pkg/quickdup types. |
| cmd/quickdup/main.go | Updates CLI to set/use pkg/quickdup globals and APIs. |
| cmd/quickdup/filter.go | Changes package declaration and switches to ActiveStrategy; minor ignore-file creation hardening. |
| cmd/quickdup/entry.go | Changes package declaration (impacts CLI build layout). |
| cmd/quickdup/detector.go | Renames/exports DetectPatterns, switches progress printing to progressf. |
| cmd/quickdup/compare.go | Changes package declaration and switches JSON types to pkg/quickdup. |
| cmd/quickdup/cache.go | Exports cache helpers and changes package declaration (impacts CLI build layout). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Active strategy (set from --strategy flag) | ||
| var activeStrategy Strategy | ||
| var debugEnabled bool | ||
| "github.com/asynkron/Asynkron.QuickDup/pkg/quickdup" |
|
|
||
| // Load user-ignored hashes from ignore.json | ||
| userIgnored := LoadIgnoredHashes(folder, *strategyName) | ||
| userIgnored := quickdup.LoadIgnoredHashes(folder, *strategyName) |
Comment on lines
+279
to
+282
| cache = quickdup.LoadCache(folder, *strategyName) | ||
| } | ||
|
|
||
| fileData, cacheHits, cacheMisses := parseFilesWithCache(files, cache) | ||
| fileData, cacheHits, cacheMisses := quickdup.ParseFilesWithCache(files, cache) |
| // Save updated cache | ||
| if !*noCache && cacheMisses > 0 { | ||
| saveCache(folder, *strategyName, files, fileData) | ||
| quickdup.SaveCache(folder, *strategyName, files, fileData) |
| detectStart := time.Now() | ||
| PrintDetectStart() | ||
| patterns := detectPatterns(fileData, len(fileData), *minOccur, *minSize, *maxSize, *keepOverlaps) | ||
| patterns := quickdup.DetectPatterns(fileData, len(fileData), *minOccur, *minSize, *maxSize, *keepOverlaps) |
Comment on lines
+105
to
+110
| for _, file := range files { | ||
| ext := strings.ToLower(filepath.Ext(file)) | ||
| CommentPrefix = commentPrefixesByExt[ext] | ||
| if CommentPrefix == "" { | ||
| CommentPrefix = "//" | ||
| } |
| MinSimilarity: minSimilarity, | ||
| UserIgnored: map[uint64]bool{}, | ||
| }) | ||
| _ = strategyName |
| IndentDelta: indentDelta, | ||
| Word: word, | ||
| SourceLine: sourceLine, | ||
| hashBytes: []byte(fmt.Sprintf("%d|%s\n", indentDelta, word)), |
| func PrintGitHubAnnotations(matches []PatternMatch, top int, githubLevel string, gitDiff string, changedFiles map[string]bool) { | ||
| func PrintGitHubAnnotations(matches []quickdup.PatternMatch, top int, githubLevel string, gitDiff string, changedFiles map[string]bool) { | ||
| annotationCount := 0 | ||
| for _, m := range matches[:top] { |
| // PrintMatches prints the top matches with their locations | ||
| func PrintMatches(matches []PatternMatch, top int) { | ||
| func PrintMatches(matches []quickdup.PatternMatch, top int) { | ||
| for i, m := range matches[:top] { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pkg/quickdupas an importable library with an exported API:ActiveStrategy,CommentPrefix,DetectPatterns,ParseFilesWithCache,LoadCache/SaveCache,FilterPatterns(already exported), plus a new single-callAnalyzeFiles/AnalyzeOptionsconvenience wrapper.cmd/quickdupkeeps only CLI concerns: flag parsing, terminal rendering/theming (output.go), JSON report I/O, and the--comparegit-worktree orchestration (compare.go).ProgressOutputwriter (defaults to discard) instead of printing straight to stdout —cmd/quickduppoints it atos.Stdoutso CLI output is unchanged.Backports
Two changes proven out in a downstream vendored copy of this engine:
strategy_wordonly.go/strategy_inlineable.go/strategy_normalizedindent.go/strategy_wordindent.go: dedup near-identicalEntry/Strategyboilerplate into sharedFirstWordEntry/indentEntry/indentStrategytypes (newstrategy_indent_shared.go).filter.go: check the.quickdupdirectory'sMkdirAllerror before writingignore.json, instead of ignoring it.similarity.go: drop the unusedcomputeAverageTokenSimilarityfunction.Test plan
go build ./...go vet ./...go test ./...(newpkg/quickduptest suite:detector_test.go,strategy_firstword_test.go)--debug,--strategy word-indent(cache read/write path),--compare(git worktree subprocess flow)🤖 Generated with Claude Code