process-improvements-todo.md

Process Improvements for Testing (Ranked by Implementation Difficulty)

This document outlines ten recommendations to improve the testing process and iteration time for the Orca Task project based on analysis of the current implementation, ranked from easiest to hardest to implement.

1. Implement Configurable Ports and URLs (Easy) ✅ IMPLEMENTED

Problem: Hardcoded ports (5000, 3000, 8080) lead to conflicts with other processes.

Solution:

• Make all service ports configurable via environment variables
• Update code to use environment variables instead of hardcoded values
• Create a port availability check before tests start

Example config file:

// config.ts
export const serviceConfig = {
  middleServerPort: process.env.MIDDLE_SERVER_PORT || 5001,
  taskServerPort: process.env.TASK_SERVER_PORT || 3000,
  pythonServerPort: process.env.PYTHON_SERVER_PORT || 8080
};

Implementation Results:

• Successfully updated middle-server/src/index.ts to use configurable host and port
• Updated task/src/constant.ts to use environment variables for the middle server URL
• Updated task/src/orca.ts to use environment variables for the Python server and command
• Updated task/docker-container/app.py and task/docker-container/utils.py to use environment variables
• Tests confirmed middle-server successfully running on custom port 5002
• Documented all configurable variables in updated .env-sample

2. Add Test Timeouts and Resilience (Easy)

Problem: Tests hang indefinitely when something goes wrong.

Solution:

• Add explicit timeouts to all Jest tests
• Implement retry logic for flaky operations
• Add better error handling with descriptive messages

// Example with timeout and retry
it("should performs the core logic task", async () => {
  const round = 1;
  
  // Add timeout to test
  jest.setTimeout(10000);
  
  // Retry logic
  let attempts = 0;
  let success = false;
  while (attempts < 3 && !success) {
    try {
      await taskRunner.task(round);
      success = true;
    } catch (e) {
      attempts++;
      if (attempts >= 3) throw e;
      await new Promise(r => setTimeout(r, 1000));
    }
  }
  
  const value = await namespaceWrapper.storeGet("value");
  expect(value).toBeDefined();
  expect(value).not.toBeNull();
}, 15000);

3. Implement Watch Mode with Fast Feedback (Easy)

Problem: Testing the full lifecycle is slow and not needed for many code changes.

Solution:

• Set up watch mode with targeted test runs
• Create focused test suites for specific components
• Implement hot reloading for faster dev/test cycles

# Example command
yarn test:watch --testPathPattern=submission

4. Mock External Dependencies in Tests (Moderate)

Problem: The Jest test (main.test.ts) attempts to interact with real services (middle-server, IPFS), slowing tests and causing flakiness.

Solution:

• Create mock implementations of external services used in testing
• Use Jest mocks for fetch calls to middle-server
• Mock IPFS storage operations
• Add a test-specific config flag to bypass actual Python process spawning

Example:

// Mock the fetch call to middle-server
jest.spyOn(global, 'fetch').mockImplementation((url) => {
  if (url.includes('/fetch-todo')) {
    return Promise.resolve({
      json: () => Promise.resolve({ 
        status: 200, 
        data: { todoID: 'mock-id', todo: 'ComputeFibonacci', input: 10 }
      })
    });
  }
  // Other fetch mocks...
});

5. Add Test-Specific Debug Tooling (Moderate)

Problem: When tests fail, it's hard to diagnose why.

Solution:

• Add verbose logging in test mode
• Create test-specific visualization for state transitions
• Add snapshots of key data structures at critical points

// Enhanced debugging
if (process.env.TEST_DEBUG) {
  console.log(JSON.stringify({
    state: "pre-submission",
    taskState: await namespaceWrapper.getTaskState({}),
    timestamp: Date.now()
  }, null, 2));
}

6. Separate Unit from Integration Tests (Moderate)

Problem: The current test suite combines multiple test types in one long-running test suite.

Solution:

• Split tests into unit, integration, and E2E categories
• Create separate test commands in package.json
• Allow running only specific test categories

{
  "scripts": {
    "test:unit": "jest --testMatch='**/*.unit.test.ts'",
    "test:integration": "jest --testMatch='**/*.integration.test.ts'",
    "test:e2e": "jest --testMatch='**/*.e2e.test.ts'"
  }
}

7. Implement In-Memory Database for Testing (Moderate to Hard)

Problem: Tests might rely on persistent state between runs.

Solution:

• Use in-memory database implementations for testing
• Reset database state between tests
• Initialize with test fixtures

// Before each test
beforeEach(async () => {
  // Reset the in-memory database
  await namespaceWrapper.resetTestStore();
  
  // Load test fixtures
  await namespaceWrapper.storeSet("testFixture", { ... });
});

8. Implement Environment-Aware Configuration (Hard)

Problem: Different environments (test, development, production) have different requirements.

Solution:

• Create environment-specific configuration files
• Add a configuration service that loads the right config based on NODE_ENV
• Include test-specific shortcuts and optimizations

// config/test.ts, config/development.ts, config/production.ts
// Different configurations for different environments

// Service to load the right config
const config = loadConfig(process.env.NODE_ENV || 'development');

9. Automated Setup and Teardown Script (Hard)

Problem: Manual steps to start/stop services and clean up between test runs.

Solution:

• Create a unified setup script that:
  • Checks for running services and stops them
  • Starts required services in the right order
  • Waits for services to be ready before tests start
  • Automatically tears down all services after tests
  • Cleans up temporary files and database state

// Example automated setup script
async function setupTestEnvironment() {
  // Kill any existing processes on test ports
  await killProcessOnPort(config.middleServerPort);
  await killProcessOnPort(config.taskServerPort);
  
  // Start services if needed and not mocked
  if (!config.useMocks.middleServer) {
    await startMiddleServer();
    await waitForService(`http://localhost:${config.middleServerPort}`);
  }
  
  // Clean temp files
  rimraf.sync('./tmp');
  fs.mkdirSync('./tmp');
  
  console.log('Test environment ready');
}

10. Containerize the Testing Environment with Docker Compose (Hardest)

Problem: Setting up dependencies (Node.js version, Python, Flask) across different developer environments leads to "works on my machine" issues.

Solution: Create a docker-compose.yml that includes:

• A Node.js container for the task
• A Node.js container for the middle-server
• A Python container with Flask pre-installed
• Shared volume mounts for code changes

Benefit: One command (docker-compose up) brings up a consistent testing environment regardless of the host machine.

These improvements would significantly reduce the time spent on setting up the testing environment, minimize flaky tests, and provide faster feedback during development iterations.