AI Expense Agent

An intelligent Telegram bot for automatic expense tracking and analysis through voice, photos, and text input.

Overview

The AI Expense Agent is an advanced Telegram bot that automatically captures and analyzes your expenses. It can understand voice messages, scan receipts, and process text to intelligently categorize and store your purchases.

Key Features

Input Methods

• Voice Recognition: Simply speak your expenses
• Receipt Scanner: Take photos of receipts for automatic import
• Text Input: Type your expenses directly

Intelligent Features

• Automatic Store Recognition: Corrects common spelling errors in supermarket names
• Product Categorization: Automatically assigns items to appropriate categories
• Price Validation: Checks prices for plausibility
• Budget Tracking: Monitors your monthly spending
• RAG-powered Store Correction: Use AI to correct store names
• Multi-modal Processing: Combines voice, OCR, and text analysis

Installation and Setup

Prerequisites

• Python 3.8 or higher
• Telegram account
• OpenAI API key

Step 1: Clone Repository

git clone [repository-url]
cd ai-expense-agent

Step 2: Install Dependencies

Create a requirements.txt file with the following content:

# Core dependencies
openai-whisper
opencv-python
numpy
pytesseract
openai
rapidfuzz
python-telegram-bot
python-dotenv
nest-asyncio

Then install:

pip install -r requirements.txt

Step 3: System Dependencies

Install Tesseract OCR on your system:

Ubuntu/Debian:

sudo apt-get install tesseract-ocr tesseract-ocr-deu

macOS:

brew install tesseract tesseract-lang

Windows: Download from: https://github.com/UB-Mannheim/tesseract/wiki

Step 4: Configure Environment Variables

Create a .env file in the root directory:

TELEGRAM_BOT_TOKEN=your_telegram_bot_token
OPENAI_API_KEY=your_openai_api_key

Step 5: Start the Bot

python main_bot.py

Creating a Telegram Bot

1. Open Telegram and search for @BotFather
2. Send /newbot and follow the instructions
3. Note down the bot token
4. Add the token to your .env file

Usage

Getting Started

1. Start the bot with /start
2. Set your monthly budget: /set_budget 500
3. Begin tracking expenses

Voice Messages

Speak naturally about your purchases:

• "Was at REWE and bought bread for 2 euros"
• "Spent 15 euros today on milk and cheese"
• "Went to ALDI, shampoo for 3 euros"

The bot automatically recognizes:

• Store names (even with spelling errors)
• Product names
• Prices
• Quantities

Receipt Scanner

1. Take a photo of your receipt
2. Send the photo to the bot
3. The bot automatically scans all items
4. Confirm import for lower recognition quality

Text Input

Write your expenses:

• "I bought beer for 2 euros"
• "Was at Rossmann, shower gel 4 euros"
• "Kaufland: pasta 1.50 euros"

Available Commands

Basic Commands

• /start - Start bot and show help
• /help - Help and command overview
• /status - Current bot status

Reports

• /report - Show last 10 expenses
• /report_daily - Today's expenses
• /report_weekly - Last 7 days expenses
• /report_monthly - Monthly overview

Budget and Analysis

• /budget_check - Current budget status
• /set_budget [amount] - Set monthly budget
• /insights - AI-powered expense analysis

Advanced Analytics

• /category_stats - Expenses by category
• /store_comparison - Compare spending between stores
• /top_products - Most purchased products
• /spending_trends - Spending trends and predictions

Project Structure

ai-expense-agent/
├── main/
│   └── main_bot.py                 # Main bot entry point
├── config/
│   └── settings.py                 # Configuration management
├── core/
│   ├── commands.py                 # Command handlers
│   └── message_handlers.py         # Message processing
├── services/
│   ├── speech_service.py           # Whisper speech recognition
│   ├── receipt_scanner.py          # OCR and receipt processing
│   ├── food_recognition_service.py # Product recognition and categorization
│   ├── rag_system.py               # Store name correction
│   └── database_service.py         # Database operations
├── models/
│   └── expense.py                  # Data models
├── utils/
│   └── formatters.py               # Message formatting
├── data/                           # Database and temporary files
├── .env                            # Environment variables
└── requirements.txt                # Python dependencies

Supported Categories

The bot automatically recognizes these categories:

• Food: Bread, milk, meat, etc.
• Beverages: Water, juices, alcoholic drinks
• Drugstore: Shampoo, toothpaste, cosmetics
• Household: Cleaning supplies, toilet paper
• Sweets: Chocolate, chips, ice cream
• Transport: Gas, public transport
• Health: Medicine, pharmacy
• Electronics: Tech items
• Clothing: Apparel and shoes

Supported Stores

The bot recognizes most German supermarket and drugstore chains:

• Supermarkets: REWE, EDEKA, KAUFLAND, REAL
• Discounters: ALDI, LIDL, NETTO, PENNY, NORMA
• Drugstores: DM, ROSSMANN, Müller
• Others: TEGUT, GLOBUS, BUDNI

Even with spelling errors, the bot correctly identifies stores.

Tips for Best Results

Voice Messages

• Speak clearly and not too fast
• Mention store, product, and price
• Use simple, natural sentences

Receipt Scanner

• Ensure good lighting
• Photograph the entire receipt
• Keep phone straight above receipt
• Avoid shadows and reflections

Text Input

• Use clear formulations
• Specify prices in euros
• Mention the store for better categorization

Technical Architecture

Core Components

• Speech Recognition: OpenAI Whisper for local processing
• OCR Engine: Tesseract with German language support
• AI Analysis: OpenAI GPT-4 for text understanding
• RAG System: Intelligent store name correction
• Database: SQLite for local data storage

Data Flow

1. Input (Voice/Photo/Text) → Processing
2. Feature Extraction → AI Analysis
3. Store/Product Recognition → Categorization
4. Database Storage → Report Generation

Database and Storage

Local Database

• All data stored locally in SQLite database
• No cloud storage without consent
• Full control over your data
• Database location: data/expenses.db

Bot Not Responding

1. Check internet connection
2. Restart the bot
3. Verify API keys in .env file

Speech Recognition Issues

1. Speak more clearly and slowly
2. Reduce background noise
3. Use shorter sentences

Receipt Scanner Problems

1. Improve image quality
2. Better lighting conditions
3. Photograph complete receipt
4. Use voice messages as alternative

Wrong Categorization

• Bot learns over time
• Manually correct wrong assignments
• Use more specific product names

Privacy and Security

Local Processing

• Speech recognition happens locally
• Only anonymized data to OpenAI
• No storage of audio files

Development and Contributing

System Requirements

• RAM: Minimum 2GB
• Storage: 1GB for installation and database
• Internet: Stable connection for Telegram and OpenAI API

Dependencies

• OpenAI Whisper: Speech recognition
• Tesseract OCR: Text recognition in images
• Python-Telegram-Bot: Telegram integration
• SQLite: Local data storage
• OpenCV: Image processing
• RapidFuzz: Fuzzy string matching

Environment Setup

# Install Python dependencies
pip install -r requirements.txt

# Install system dependencies (Ubuntu)
sudo apt-get install tesseract-ocr tesseract-ocr-deu

# Set up environment variables
cp .env.example .env
# Edit .env with your API keys

---

Note: This bot processes financial data locally and uses AI services. Always review your data and ensure you're comfortable with the privacy implications before use.