TRSLLAVA - Beyond Pixels Reproduction

Automated image description generation for remote sensing datasets using Gemini API.

Project Structure

TRSLLAVA/
├── config.py                 # Configuration file
├── gemini_processor.py       # Main processing script
├── token_estimator.py        # Token usage estimation
├── requirements.txt          # Python dependencies
├── README.md                 # This file
├── results/                  # Output directory
│   ├── rsicd_descriptions.json
│   └── rsitmd_descriptions.json
├── cache/                    # Intermediate cache
└── logs/                     # Processing logs

Quick Start

1. Install Dependencies

cd ~/TRSLLAVA
pip install -r requirements.txt

2. Configure API Key

Edit config.py and set your Gemini API key:

GEMINI_API_KEY = "your_actual_api_key_here"

Or set via environment variable:

export GEMINI_API_KEY="your_actual_api_key_here"

3. Estimate Token Usage

python token_estimator.py

4. Process Datasets

Process both datasets:

python gemini_processor.py --dataset both

Process only RSICD:

python gemini_processor.py --dataset rsicd

Process only RSITMD:

python gemini_processor.py --dataset rsitmd

5. Check Results

Results are saved in results/ directory as JSON files:

# View statistics
python -m json.tool results/rsicd_descriptions.json | head -50

# Count descriptions
python -c "import json; data=json.load(open('results/rsicd_descriptions.json')); print(f'Total: {len(data)}, Success: {sum(1 for v in data.values() if v.get(\"description\"))}')"

Token & Cost Estimation

Dataset	Images	Input Tokens	Output Tokens	Cost (Pro)	Cost (Flash)
RSICD	10,921	~14M	~1.6M	~$50	~$1.5
RSITMD	4,743	~6M	~0.7M	~$22	~$0.7
Total	15,664	~20M	~2.3M	~$72	~$2.2

Recommendation: Use Gemini 1.5 Flash (gemini-1.5-flash) to save ~97% on costs.

Configuration Options

Edit config.py to customize:

• Model: gemini-1.5-pro (quality) or gemini-1.5-flash (speed/cost)
• Batch size: How often to save intermediate results
• Retry settings: For failed API calls
• Rate limiting: Delay between requests
• Prompt: Customize description generation prompt

Features

• ✅ Automatic checkpointing: Resume from where you left off
• ✅ Batch processing: Save progress periodically
• ✅ Error handling: Automatic retries with exponential backoff
• ✅ Token tracking: Monitor usage during processing
• ✅ Detailed logging: Full logs in logs/ directory
• ✅ Progress statistics: Real-time progress updates

Output Format

Each description JSON file contains:

{
  "image_name.jpg": {
    "description": "A detailed natural language description of the remote sensing image...",
    "path": "/full/path/to/image.jpg",
    "timestamp": "2026-01-20T12:34:56.789012"
  },
  "failed_image.jpg": {
    "description": null,
    "path": "/path/to/failed_image.jpg",
    "error": "Failed to generate",
    "timestamp": "2026-01-20T12:34:56.789012"
  }
}

Troubleshooting

Rate Limiting

If you hit rate limits, increase RATE_LIMIT_DELAY in config.py:

RATE_LIMIT_DELAY = 2  # Increase from 1 to 2 seconds

Out of Memory

If processing large datasets, reduce BATCH_SIZE:

BATCH_SIZE = 5  # Reduce from 10 to 5

API Errors

Check your logs in logs/processing_*.log for detailed error messages.

Next Steps (After Description Generation)

1. Use descriptions for retrieval: Implement BERT encoding
2. Evaluate quality: Compare with ground truth captions
3. Optimize prompts: Test different prompts for better descriptions

References

• Beyond Pixels paper: [Link to paper]
• Gemini API docs: https://ai.google.dev/gemini-api/docs
• RSICD dataset: https://github.com/201528014227051/RSICD_optimal
• RSITMD dataset: https://github.com/201528014227051/RSITMD