graph LR
Data_Preparation_Formatting_Component["Data Preparation & Formatting Component"]
Evaluation_Orchestration_Component["Evaluation Orchestration Component"]
Evaluation_Data_Management_Component["Evaluation Data Management Component"]
Benchmark_Specific_Evaluation_Component["Benchmark-Specific Evaluation Component"]
Evaluation_Utilities_Component["Evaluation Utilities Component"]
Data_Preparation_Formatting_Component -- "provides formatted data to" --> Evaluation_Data_Management_Component
Evaluation_Orchestration_Component -- "coordinates with" --> Evaluation_Data_Management_Component
Evaluation_Orchestration_Component -- "delegates evaluation tasks to" --> Benchmark_Specific_Evaluation_Component
Evaluation_Orchestration_Component -- "utilizes" --> Evaluation_Utilities_Component
Evaluation_Data_Management_Component -- "supplies data to" --> Evaluation_Orchestration_Component
Evaluation_Data_Management_Component -- "supplies data to" --> Benchmark_Specific_Evaluation_Component
Evaluation_Data_Management_Component -- "receives data from" --> Data_Preparation_Formatting_Component
Benchmark_Specific_Evaluation_Component -- "receives data from" --> Evaluation_Data_Management_Component
Benchmark_Specific_Evaluation_Component -- "uses" --> Evaluation_Utilities_Component
Benchmark_Specific_Evaluation_Component -- "provides results to" --> Evaluation_Orchestration_Component
Evaluation_Utilities_Component -- "supports" --> Evaluation_Orchestration_Component
Evaluation_Utilities_Component -- "supports" --> Benchmark_Specific_Evaluation_Component
The Data & Evaluation Module is a critical subsystem within the LLaVA project, focusing on preparing datasets for the LLaVA model and rigorously assessing its performance.
Responsible for transforming raw datasets (e.g., SQA) into the LLaVA-specific JSONL format, including the construction and formatting of chatbot-style prompts. This component ensures data consistency and proper structuring for subsequent training or evaluation.
Related Classes/Methods:
scripts.convert_sqa_to_llava.convert_to_llava:8-46scripts.convert_sqa_to_llava_base_prompt.build_prompt_chatbot:221-241
Serves as the primary driver for general LLaVA model evaluation. It coordinates the overall assessment process, including data loading, model inference, and result aggregation across various benchmarks.
Related Classes/Methods:
Handles the preparation, loading, and batching of evaluation datasets. It provides flexibility for diverse datasets by managing custom data loading mechanisms, ensuring data is efficiently supplied to evaluation routines.
Related Classes/Methods:
Implements the specific evaluation logic, metrics, and procedures tailored for individual benchmarks (e.g., MMBench, general VQA, ScienceQA, M4C, TextVQA). These modules apply domain-specific rules and scoring mechanisms to assess model performance accurately.
Related Classes/Methods:
llava.eval.model_vqa_mmbench.eval_model:54-140llava.eval.model_vqa.eval_model:29-84llava.eval.model_vqa_science.eval_model:29-94llava.eval.m4c_evaluatorllava.eval.eval_textvqa.eval_single:35-51
Provides common utility functions for image processing (parsing, loading), text processing (normalization, formatting), and metric computation (e.g., ANLS, accuracy) used across various evaluation benchmarks. These utilities ensure consistent data handling and metric calculation.
Related Classes/Methods: