Refactor: Decouple Orchestration and API Layers in NV-Ingest

[drobison00]

Summary

This PR refactors nv-ingest to cleanly separate the orchestration layer from the API elements, allowing finer-grained usage of individual work elements via the API. This improves modularity and makes it easier to invoke specific pipeline components independently.

Changes

Separation of Concerns

• Moved all API-related code into src/api/ to centralize API functionality.
• Moved all orchestration code into src/nv_ingest/framework/, ensuring a clearer distinction between execution flow and API access.
• Client code remains unchanged in client/.

Standardized API Interface

• Introduced a common API signature for stage-level execution:

  (
      df_extraction_ledger: pd.DataFrame, 
      config: Dict[str, Any], 
      execution_trace_log: Optional[List[Any]] = None
  ) -> Tuple[pd.DataFrame, Dict]

• This ensures consistency across API-exposed pipeline stages.

Benefits

• Enables fine-grained execution of pipeline elements through the API.
• Simplifies code structure by clearly separating orchestration from API concerns.
• Makes it easier to extend and maintain API interactions independently of orchestration logic.

Next Steps

• Additional testing to ensure smooth orchestration/API interaction.
• Documentation updates to reflect new API structure.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.