Create Smart Fallback Mechanisms for Complex Reframing Scenarios

[sdntsng]

Problem Statement

Our intelligent reframing system (Issues #22, #23) will encounter scenarios where optimal subject-focused cropping isn't possible due to:

• Multiple important subjects that can't fit in target aspect ratio
• Fast-moving or erratic subject movement that breaks tracking
• Poor lighting or quality that prevents reliable subject detection
• Content types that don't have clear subjects (landscapes, graphics, presentations)
• Ultra-wide content that would lose too much information when cropped

Currently, our system has only basic center-crop fallback, resulting in poor user experience when intelligent reframing fails.

Current Implementation

File: backend/src/routes/reframe.js

Current fallback logic:

// Minimal fallback to center crop
const fallbackFilter = `crop=${targetWidth}:${targetHeight}:(iw-ow)/2:(ih-oh)/2`;

Issues:

• No graceful degradation options
• Users have no choice when intelligent reframing fails
• Important content may be lost with aggressive cropping
• No visual indication of fallback mode usage

Research Insights

Google AutoFlip Approach

AutoFlip implements sophisticated fallback strategies:

• Letterboxing with blurred background when content can't be cropped
• Background color fill for minimal distraction
• Content-aware padding using edge pixel extension
• Zoom-out option to retain more content at cost of smaller subject size

Industry Best Practices

• Progressive degradation - Try multiple strategies before giving up
• User choice - Let users select preferred fallback method
• Visual feedback - Clearly indicate when fallback is used
• Quality preservation - Prioritize content retention over perfect framing

Proposed Solution

Phase 1: Multi-Strategy Fallback System

Implement cascading fallback options with user control:

// Enhanced fallback system
class ReframeFallbackManager {
  constructor() {
    this.strategies = [
      'intelligent-crop',    // Primary: Subject detection + smooth movement
      'zoom-out-crop',      // Secondary: Wider crop to retain more content  
      'letterbox-blur',     // Tertiary: Letterbox with blurred background
      'letterbox-color',    // Quaternary: Letterbox with solid color
      'center-crop'         // Final: Simple center crop
    ];
  }
  
  async processReframe(videoData, targetAspectRatio, userPreferences = {}) {
    const results = [];
    
    for (const strategy of this.strategies) {
      try {
        const result = await this.tryStrategy(strategy, videoData, targetAspectRatio);
        
        if (this.validateResult(result, userPreferences)) {
          return {
            success: true,
            strategy: strategy,
            result: result,
            fallbacksAttempted: results.length
          };
        }
        
        results.push({ strategy, error: result.error });
        
      } catch (error) {
        console.log(`Strategy ${strategy} failed:`, error.message);
        results.push({ strategy, error: error.message });
      }
    }
    
    // All strategies failed
    return {
      success: false,
      strategies: results,
      error: 'All reframing strategies failed'
    };
  }
  
  async tryStrategy(strategy, videoData, aspectRatio) {
    switch (strategy) {
      case 'intelligent-crop':
        return await this.intelligentCrop(videoData, aspectRatio);
        
      case 'zoom-out-crop':
        return await this.zoomOutCrop(videoData, aspectRatio);
        
      case 'letterbox-blur':
        return await this.letterboxWithBlur(videoData, aspectRatio);
        
      case 'letterbox-color':
        return await this.letterboxWithColor(videoData, aspectRatio);
        
      case 'center-crop':
        return await this.centerCrop(videoData, aspectRatio);
        
      default:
        throw new Error(`Unknown strategy: ${strategy}`);
    }
  }
  
  validateResult(result, userPreferences) {
    // Check if result meets user's quality criteria
    const minContentRetention = userPreferences.minContentRetention || 0.7;
    const maxLetterboxRatio = userPreferences.maxLetterboxRatio || 0.3;
    
    if (result.contentRetention < minContentRetention) {
      return false;
    }
    
    if (result.letterboxRatio > maxLetterboxRatio) {
      return false;
    }
    
    return true;
  }
}

Phase 2: Advanced Fallback Strategies

Implement sophisticated alternatives to basic cropping:

// Zoom-out cropping - retain more content at smaller scale
async function zoomOutCrop(videoData, aspectRatio) {
  const { width: targetWidth, height: targetHeight } = getAspectRatioDimensions(aspectRatio);
  
  // Calculate optimal zoom-out factor to retain important content
  const subjects = await detectAllSubjects(videoData);
  const contentBounds = calculateContentBounds(subjects);
  
  // Determine minimum zoom to keep all subjects
  const zoomFactor = calculateOptimalZoom(contentBounds, targetWidth, targetHeight);
  
  const cropFilter = `crop=${targetWidth}:${targetHeight}:(iw-${targetWidth})/2:(ih-${targetHeight})/2,scale=${Math.floor(targetWidth * zoomFactor)}:${Math.floor(targetHeight * zoomFactor)}`;
  
  return {
    filter: cropFilter,
    strategy: 'zoom-out-crop',
    zoomFactor: zoomFactor,
    contentRetention: calculateContentRetention(contentBounds, zoomFactor),
    letterboxRatio: 0
  };
}

// Letterbox with blurred background
async function letterboxWithBlur(videoData, aspectRatio) {
  const { width: targetWidth, height: targetHeight } = getAspectRatioDimensions(aspectRatio);
  
  // Create blurred background layer
  const blurFilter = `[0:v]scale=${targetWidth}:${targetHeight}:force_original_aspect_ratio=increase,crop=${targetWidth}:${targetHeight},gblur=sigma=20[blurred]`;
  
  // Overlay original video scaled to fit
  const overlayFilter = `[0:v]scale=${targetWidth}:${targetHeight}:force_original_aspect_ratio=decrease[scaled];[blurred][scaled]overlay=(W-w)/2:(H-h)/2`;
  
  return {
    filter: `${blurFilter};${overlayFilter}`,
    strategy: 'letterbox-blur',
    contentRetention: 1.0, // All content retained
    letterboxRatio: calculateLetterboxRatio(videoData.aspectRatio, aspectRatio)
  };
}

// Letterbox with solid color background  
async function letterboxWithColor(videoData, aspectRatio, backgroundColor = '#000000') {
  const { width: targetWidth, height: targetHeight } = getAspectRatioDimensions(aspectRatio);
  
  // Create solid color background
  const colorFilter = `color=${backgroundColor}:size=${targetWidth}x${targetHeight}[bg]`;
  
  // Scale and overlay original video
  const overlayFilter = `[0:v]scale=${targetWidth}:${targetHeight}:force_original_aspect_ratio=decrease[scaled];[bg][scaled]overlay=(W-w)/2:(H-h)/2`;
  
  return {
    filter: `${colorFilter};${overlayFilter}`,
    strategy: 'letterbox-color',
    backgroundColor: backgroundColor,
    contentRetention: 1.0,
    letterboxRatio: calculateLetterboxRatio(videoData.aspectRatio, aspectRatio)
  };
}

Phase 3: User Preference System

Allow users to configure fallback behavior:

// User preferences for fallback behavior
interface ReframeFallbackPreferences {
  // Strategy preferences (ordered by preference)
  preferredStrategies: string[];
  
  // Quality thresholds
  minContentRetention: number;      // 0-1, minimum content that must be retained
  maxLetterboxRatio: number;        // 0-1, maximum acceptable letterbox ratio
  
  // Appearance preferences
  letterboxColor: string;           // Color for letterbox backgrounds
  blurIntensity: number;           // 0-20, blur intensity for blurred backgrounds
  
  // Performance preferences
  maxProcessingTime: number;        // Maximum time to spend trying strategies
  enablePreview: boolean;          // Generate preview for user approval
}

// API endpoint for user configuration
router.post('/reframe/configure', async (req, res) => {
  const { transcriptId, preferences } = req.body;
  
  // Store user preferences
  await UserPreferences.upsert({
    transcriptId: transcriptId,
    reframePreferences: preferences
  });
  
  res.json({
    success: true,
    message: 'Reframe preferences saved'
  });
});

Technical Implementation

Enhanced Reframe Endpoint

// Updated reframe.js with fallback integration
router.post('/generate', async (req, res) => {
  try {
    const { transcriptId, aspectRatio, startTime, endTime, preferences = {} } = req.body;
    
    // Load user preferences
    const userPrefs = await UserPreferences.findOne({ transcriptId }) || {};
    const reframePrefs = { ...DEFAULT_PREFERENCES, ...userPrefs.reframePreferences, ...preferences };
    
    // Initialize fallback manager
    const fallbackManager = new ReframeFallbackManager();
    
    // Attempt reframing with fallback strategies
    const result = await fallbackManager.processReframe(
      videoData,
      aspectRatio,
      reframePrefs
    );
    
    if (\!result.success) {
      return res.status(400).json({
        success: false,
        error: 'All reframing strategies failed',
        details: result.strategies
      });
    }
    
    // Generate video using selected strategy
    await generateReframedVideo(inputPath, outputPath, result);
    
    res.json({
      success: true,
      reframedVideoUrl: outputUrl,
      strategy: result.strategy,
      fallbacksAttempted: result.fallbacksAttempted,
      metadata: {
        contentRetention: result.result.contentRetention,
        letterboxRatio: result.result.letterboxRatio || 0,
        processingTime: Date.now() - startTime
      }
    });
    
  } catch (error) {
    console.error('Reframe with fallback failed:', error);
    res.status(500).json({
      success: false,
      error: 'Reframing failed',
      details: error.message
    });
  }
});

Quality Assessment Metrics

function calculateContentRetention(originalBounds, processedBounds) {
  const originalArea = originalBounds.width * originalBounds.height;
  const retainedArea = processedBounds.width * processedBounds.height;
  return retainedArea / originalArea;
}

function calculateLetterboxRatio(sourceAspectRatio, targetAspectRatio) {
  if (sourceAspectRatio === targetAspectRatio) return 0;
  
  // Calculate how much of the frame will be letterbox
  const scaleFactor = Math.min(
    targetAspectRatio.width / sourceAspectRatio.width,
    targetAspectRatio.height / sourceAspectRatio.height
  );
  
  const usedArea = scaleFactor * sourceAspectRatio.width * sourceAspectRatio.height;
  const totalArea = targetAspectRatio.width * targetAspectRatio.height;
  
  return 1 - (usedArea / totalArea);
}

Files to Modify

1. backend/src/routes/reframe.js

   • Integrate ReframeFallbackManager
   • Add user preference handling
   • Update response format with strategy metadata

2. backend/src/utils/reframeFallbackManager.js (new file)

   • Fallback strategy implementations
   • Quality assessment functions
   • User preference validation

3. backend/src/models/UserPreferences.js (new file)

   • MongoDB schema for user reframe preferences
   • Default preference values

4. frontend/src/components/ReframeModal.tsx

   • Add fallback preference controls
   • Display strategy information in results
   • Show quality metrics to users

User Experience Enhancements

Fallback Strategy Preview

// Allow users to preview different fallback strategies
const ReframeFallbackPreview: React.FC = ({ transcript, aspectRatio }) => {
  const [strategies, setStrategies] = useState([]);
  const [selectedStrategy, setSelectedStrategy] = useState('intelligent-crop');
  
  const generatePreviews = async () => {
    const previews = await Promise.all([
      generatePreview('intelligent-crop'),
      generatePreview('letterbox-blur'),
      generatePreview('letterbox-color'),
      generatePreview('center-crop')
    ]);
    
    setStrategies(previews);
  };
  
  return (
    <div className="fallback-preview">
      <h3>Reframing Strategy Options</h3>
      
      {strategies.map(strategy => (
        <div key={strategy.name} className="strategy-option">
          <img src={strategy.previewUrl} alt={strategy.name} />
          <div className="strategy-info">
            <h4>{strategy.name}</h4>
            <p>Content Retention: {(strategy.contentRetention * 100).toFixed(1)}%</p>
            <p>Processing Time: ~{strategy.estimatedTime}s</p>
          </div>
          <Button onClick={() => setSelectedStrategy(strategy.name)}>
            Select
          </Button>
        </div>
      ))}
    </div>
  );
};

Testing Strategy

Fallback Trigger Testing

• Force subject detection failures
• Test with various video aspect ratios
• Test with poor quality/lighting videos
• Test with ultra-wide and ultra-tall content
• Test with content having no clear subjects

Quality Assessment

• Measure content retention accuracy
• User preference compliance testing
• A/B testing of fallback vs. no-fallback
• Processing time impact measurement

Success Metrics

• Reframing success rate >95% (with fallbacks)
• User satisfaction with fallback options
• Average content retention >80% across all strategies
• Processing time increase <50% for fallback attempts
• Clear user feedback about which strategy was used
• Graceful handling of all edge cases

Implementation Priority

Effort: Medium (4-5 days)
Priority: Medium
Risk: Low (builds on existing reframe system)
Dependencies: Subject Detection (#22) enhances but not required

This enhancement ensures our reframing system provides excellent results even in challenging scenarios, building user confidence and reducing support requests.