Skip to content

Commit e35742e

Browse files
author
Ubuntu
committed
Merge branch 'fix/test-suite-compatibility' into main
Resolved conflicts in generated files (package-lock.json and tsconfig.tsbuildinfo files)
2 parents b9dfa5e + a462545 commit e35742e

26 files changed

Lines changed: 4318 additions & 143 deletions

‎CI_TEST_FAILURE_INVESTIGATION.md‎

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# CI Test Failure Investigation Prompt
2+
3+
## Context
4+
The mcp-debugger project has 1019 tests total, but the CI workflow excludes e2e and Python tests to maintain a green build. When running the FULL test suite in the CI environment (using Act), we found that 1014 tests pass and only 4 tests actually fail. This means we're unnecessarily excluding 83+ passing tests from CI.
5+
6+
## The 4 Failing Tests
7+
8+
### 1. Full Debug Session - Multiple Breakpoints Test
9+
**File**: `tests/e2e/full-debug-session.test.ts`
10+
**Test**: "python debugging > should handle multiple breakpoints correctly"
11+
**Error**: Timeout waiting for session to reach paused/stopped state
12+
**Pattern**: `[EventUtils] Timeout waiting for session ... to reach any of: paused, stopped. Current state: 'running'. Waited 10000ms`
13+
14+
### 2. Container Smoke Test - Path Handling
15+
**File**: `tests/e2e/mcp-server-smoke-container.test.ts`
16+
**Test**: "should handle paths naturally in container mode"
17+
**Error**: Docker cleanup error: `docker rm mcp-path-test-...`
18+
**Pattern**: Docker container management issue
19+
20+
### 3-4. SSE Transport Tests (2 failures)
21+
**File**: `tests/e2e/mcp-server-smoke-sse.test.ts`
22+
**Tests**:
23+
- "should successfully debug fibonacci.py via SSE transport"
24+
- "should work when SSE server is spawned from different working directory"
25+
**Error**: `SSE server exited with code null`
26+
**Pattern**: Server process management in CI environment
27+
28+
## Investigation Questions
29+
30+
### For Each Failing Test:
31+
32+
#### 1. What is the test actually testing?
33+
- What feature/capability is being validated?
34+
- Is this a critical path or edge case?
35+
- What would break if this test were removed?
36+
37+
#### 2. Why does it fail in CI but pass locally?
38+
- **Docker-in-Docker issues?** (Act runs in Docker, tests spawn Docker containers)
39+
- **Process management differences?** (SSE server lifecycle)
40+
- **Timing/performance issues?** (10s timeout might be too short in CI)
41+
- **Network/port binding differences?** (localhost vs container networking)
42+
- **File system differences?** (volume mounts, permissions)
43+
44+
#### 3. What type of failure is this?
45+
46+
**Category A: Test Design Issue**
47+
- Test assumes local environment characteristics
48+
- Hardcoded timeouts too aggressive for CI
49+
- Test cleanup not robust enough
50+
- Should be modified to work in CI
51+
52+
**Category B: Implementation Bug**
53+
- Our code doesn't handle CI environment correctly
54+
- Missing error handling for edge cases
55+
- Race conditions exposed by slower CI environment
56+
- Implementation should be fixed
57+
58+
**Category C: Inappropriate for CI**
59+
- Requires resources not available in CI (like Docker-in-Docker)
60+
- Tests interactive/GUI features
61+
- Tests platform-specific behavior
62+
- Should be skipped in CI with proper justification
63+
64+
## Specific Investigation Tasks
65+
66+
### Task 1: Analyze Multiple Breakpoints Test
67+
```bash
68+
# Look at the test implementation
69+
cat tests/e2e/full-debug-session.test.ts | grep -A 50 "should handle multiple breakpoints"
70+
71+
# Check timeout configuration
72+
grep -r "10000" tests/e2e/full-debug-session.test.ts
73+
74+
# See if there's retry logic
75+
grep -r "retry\|attempt\|waitFor" tests/e2e/full-debug-session.test.ts
76+
```
77+
78+
**Questions**:
79+
- Is 10 seconds enough time for Python/debugpy to initialize in a containerized CI environment?
80+
- Does the test properly wait for debugpy to be ready before setting breakpoints?
81+
- Could this be a race condition where breakpoints are set before the debugger is attached?
82+
83+
### Task 2: Analyze Container Path Test
84+
```bash
85+
# Look at container management
86+
grep -B5 -A10 "handle paths naturally" tests/e2e/mcp-server-smoke-container.test.ts
87+
88+
# Check Docker cleanup logic
89+
grep -r "docker rm\|cleanup\|teardown" tests/e2e/
90+
91+
# Look for Docker-in-Docker configuration
92+
grep -r "DOCKER_HOST\|docker.sock" tests/
93+
```
94+
95+
**Questions**:
96+
- Is the test trying to use Docker-in-Docker?
97+
- Are container names conflicting in parallel test runs?
98+
- Is cleanup happening in the wrong order?
99+
100+
### Task 3: Analyze SSE Server Tests
101+
```bash
102+
# Check SSE server spawn logic
103+
grep -B10 -A10 "SSE server exited" tests/e2e/mcp-server-smoke-sse.test.ts
104+
105+
# Look at server lifecycle management
106+
grep -r "spawn\|fork\|exec" tests/e2e/mcp-server-smoke-sse.test.ts
107+
108+
# Check port management
109+
grep -r "PORT\|port\|3000\|listen" tests/e2e/mcp-server-smoke-sse.test.ts
110+
```
111+
112+
**Questions**:
113+
- Is the SSE server failing to start due to port conflicts?
114+
- Are we properly waiting for the server to be ready?
115+
- Is there a missing dependency in the CI environment?
116+
117+
## Decision Matrix
118+
119+
For each failing test, classify and decide:
120+
121+
| Test | Category | Action | Justification |
122+
|------|----------|--------|---------------|
123+
| Multiple breakpoints | A/B/C? | Fix test/Fix impl/Skip | Why? |
124+
| Container paths | A/B/C? | Fix test/Fix impl/Skip | Why? |
125+
| SSE transport 1 | A/B/C? | Fix test/Fix impl/Skip | Why? |
126+
| SSE transport 2 | A/B/C? | Fix test/Fix impl/Skip | Why? |
127+
128+
## Recommended Actions
129+
130+
### If Category A (Test Issue):
131+
1. Increase timeouts for CI environment
132+
2. Add retry logic with exponential backoff
133+
3. Improve test isolation and cleanup
134+
4. Add CI-specific test configuration
135+
136+
### If Category B (Implementation Issue):
137+
1. Fix the actual bug in the implementation
138+
2. Add proper error handling
139+
3. Make code more resilient to different environments
140+
4. Add logging to help diagnose future issues
141+
142+
### If Category C (Inappropriate for CI):
143+
1. Skip with clear justification
144+
2. Document what manual testing is required
145+
3. Consider alternative test approaches
146+
4. Add to release checklist for manual verification
147+
148+
## Success Criteria
149+
150+
The goal is to:
151+
1. **Enable maximum test coverage in CI** - Run as many of the 1019 tests as possible
152+
2. **Document clear rationale** - For any test we skip, explain WHY
153+
3. **Improve reliability** - Tests should pass consistently, not flakily
154+
4. **Maintain feature coverage** - Don't lose important validations
155+
156+
## Next Steps
157+
158+
1. Run each failing test in isolation in Act environment
159+
2. Add debug logging to understand exact failure points
160+
3. Try fixes in order of impact (easiest first)
161+
4. Create PR with either:
162+
- Fixed tests that now pass in CI
163+
- Fixed implementation that handles CI properly
164+
- Properly skipped tests with clear skip conditions
165+
166+
## Example Skip Pattern
167+
168+
```typescript
169+
describe.skipIf(process.env.CI && !process.env.DOCKER_HOST)(
170+
'Docker-in-Docker tests',
171+
() => {
172+
// Tests that require Docker-in-Docker capability
173+
test('should handle container paths', () => {
174+
// This test requires Docker daemon access from within Act container
175+
// Skipped in CI unless DOCKER_HOST is configured for Docker-in-Docker
176+
});
177+
}
178+
);
179+
```
180+
181+
## Investigation Commands
182+
183+
```bash
184+
# Run just the failing tests in Act
185+
act -W test-full-ci.yml -j test-all --env TEST_FILTER="full-debug-session"
186+
187+
# Run with verbose logging
188+
act -W test-full-ci.yml -j test-all --env DEBUG="*" --env LOG_LEVEL="debug"
189+
190+
# Check Docker-in-Docker setup
191+
act -W test-full-ci.yml -j test-all --privileged --container-options "-v /var/run/docker.sock:/var/run/docker.sock"
192+
193+
# Test with extended timeouts
194+
act -W test-full-ci.yml -j test-all --env TEST_TIMEOUT_MULTIPLIER=3
195+
```
196+
197+
## Final Questions to Answer
198+
199+
1. **Are we testing the right things?** - Do these e2e tests provide value or are they redundant?
200+
2. **Can we make tests more resilient?** - Add retries, better waits, smarter assertions?
201+
3. **Should CI mirror production?** - Or is it OK to have some differences?
202+
4. **What's the maintenance cost?** - Of fixing vs skipping vs leaving excluded?
203+
204+
---
205+
206+
This investigation should determine whether each failing test represents:
207+
- A test that needs fixing to work in CI
208+
- A bug in our implementation that CI exposed
209+
- A legitimate reason to skip in CI (like platform-specific tests)

0 commit comments

Comments
 (0)