Skip to content

Latest commit

 

History

History
600 lines (537 loc) · 17 KB

TECHFEATURE.md

File metadata and controls

600 lines (537 loc) · 17 KB

Meme Agent Technical Features Documentation

System Architecture

Core Components and File Architecture

meme-agent/
├── src/
│   ├── agents/                    # AI Agent System
│   │   ├── models/               # AI Model Implementations
│   │   ├── providers/            # AI Provider Integrations (Groq, DeepSeek)
│   │   │   ├── groq.provider.ts  # Groq API Integration for Tweet Generation
│   │   │   └── deepseek.provider.ts # DeepSeek Integration
│   │   ├── prompts/             # Agent Prompts & Templates
│   │   └── groups/              # Agent Group Configurations
│   ├── services/                 # Core Services
│   │   ├── ai/                  # AI Services
│   │   │   ├── tweetGenerator.ts # Tweet Content Generation
│   │   │   ├── types.ts         # AI Service Type Definitions
│   │   │   └── personality.ts   # Character-based Generation
│   │   ├── social/              # Social Media Integration
│   │   │   ├── twitter.ts       # Twitter API Integration
│   │   │   ├── MarketTweetCron.ts # Automated Market Updates
│   │   │   └── agentTwitterClient.ts # Twitter Client Implementation
│   │   ├── analysis/            # Analysis Services
│   │   │   ├── market-analysis.ts # Market Data Processing
│   │   │   └── transaction-analysis.ts # Transaction Analysis
│   │   └── blockchain/          # Blockchain Integration
│   ├── personality/              # Character System
│   │   ├── types.ts             # Character Schema Definitions
│   │   ├── loadCharacter.ts     # Character Loading Logic
│   │   └── traits/              # Character Traits Implementation
│   ├── infrastructure/           # Infrastructure Layer
│   │   ├── database/            # Database Management
│   │   │   ├── entities/        # TypeORM Entities
│   │   │   ├── schemas/         # MongoDB Schemas
│   │   │   └── services/        # Database Services
│   │   └── cache/              # Cache Management
│   ├── middleware/              # Express Middleware
│   │   ├── auth/               # Authentication
│   │   ├── validation/         # Request Validation
│   │   └── error/              # Error Handling
│   └── utils/                  # Utility Functions
├── characters/                  # Character Configurations
│   └── jenna.character.json    # Jenna Character Definition
├── tests/                      # Test Suites
│   ├── unit/                  # Unit Tests
│   │   └── services/
│   │       └── social/        # Twitter Service Tests
│   └── integration/           # Integration Tests
├── logs/                      # Application Logs
│   ├── social.log            # Social Media Integration Logs
│   └── market.log           # Market Analysis Logs
└── scripts/                  # Utility Scripts

Key Components for Twitter Integration

  1. Tweet Generation Pipeline

    • src/services/ai/tweetGenerator.ts: Core tweet generation logic
    • src/services/ai/personality.ts: Character-based content adaptation
    • src/services/social/MarketTweetCron.ts: Automated market updates

  2. Twitter Service Implementation

    • src/services/social/twitter.ts: Main Twitter service
    • src/services/social/agentTwitterClient.ts: Twitter API client
    • src/middleware/rateLimit.ts: Rate limiting implementation

  3. Character System Integration

    • src/personality/types.ts: Character schema definitions
    • src/personality/loadCharacter.ts: Character loading logic
    • characters/jenna.character.json: Character configuration

  4. Market Data Integration

    • src/services/analysis/market-analysis.ts: Market data processing
    • src/services/blockchain/defi/tradingEngine.ts: Trading functionality
    • src/services/analysis/transaction-analysis.ts: Transaction analysis

### Technology Stack

#### Core Technologies
- **Runtime**: Node.js (≥18.0.0)
  - ESM modules
  - Worker threads support
  - Native fetch API
- **Package Manager**: pnpm (≥8.0.0)
  - Workspace support
  - Fast, efficient installation
- **Language**: TypeScript 5.3
  - Strict type checking
  - ESM support
  - Decorators

#### Database Layer
- **PostgreSQL 14**
  - TypeORM integration
  - JSON(B) support
  - Full-text search
  - Partitioning
- **MongoDB 6**
  - Mongoose ODM
  - Time-series collections
  - Change streams
  - Atlas integration
- **Redis 7**
  - Caching layer
  - Pub/Sub system
  - Rate limiting
  - Session storage

#### AI Infrastructure
- **Primary Model**: DeepSeek-33B
  - 33 billion parameters
  - Custom fine-tuning
  - Specialized prompts
- **Secondary Models**
  - GPT-4 Turbo
  - Claude-3 Opus
  - Ollama (local)
  - Groq (cloud)

## Core Features

### 1. Transaction Management
- **Real-time Monitoring**
  - WebSocket connections
  - Transaction streaming
  - Block confirmation tracking
  - Mempool monitoring
- **Analysis Engine**
  - Price impact calculation
  - Slippage prediction
  - Gas optimization
  - MEV protection
- **Multi-DEX Support**
  - Jupiter integration
  - Orca integration
  - Best route finding
  - Split orders
- **Caching System**
  - Transaction history
  - Order book snapshots
  - Token metadata
  - Price feeds

### 2. Token Analysis
- **Contract Analysis**
  - Bytecode verification
  - Security audit
  - Ownership analysis
  - Permission checks
- **Market Analysis**
  - Volume profiling
  - Liquidity analysis
  - Holder distribution
  - Price correlation
- **Social Metrics**
  - Sentiment analysis
  - Trend detection
  - Influencer tracking
  - Community growth
- **Technical Analysis**
  - Custom indicators
  - Pattern recognition
  - Volatility metrics
  - Risk assessment

### 3. Social Media Integration

#### Twitter Integration Architecture
The Twitter integration follows the elizaOS pattern, using direct authentication without API tokens for improved reliability and maintainability.

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ AI Service │ │ Tweet Service │ │ Market Data │ │ (tweetGenerator)│────▶│ (twitter.ts) │◀────│ Services │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Character Configuration │ │ (personality & engagement settings) │ └─────────────────────────────────────────────────────────────────┘


#### Core Components

1. **Twitter Authentication Service** (`src/services/social/twitter.ts`)
   ```typescript
   interface TwitterAuthService {
     initialize(config: TwitterConfig): Promise<void>;
     authenticate(): Promise<AuthResult>;
     validateSession(): Promise<boolean>;
     handleAuthError(error: AuthError): Promise<void>;
   }
   
   interface TwitterConfig {
     username: string;        // Twitter account username
     password: string;        // Twitter account password
     email: string;          // Twitter account email
     mockMode?: boolean;     // Enable mock mode for development
     maxRetries?: number;    // Maximum authentication retries
     retryDelay?: number;    // Delay between retries (ms)
   }

Authentication Flow:

  1. Environment Variable Loading:

    class TwitterAuthManager {
  private loadCredentials(): TwitterConfig {
    return {
      username: process.env.TWITTER_USERNAME,
      password: process.env.TWITTER_PASSWORD,
      email: process.env.TWITTER_EMAIL,
      mockMode: process.env.TWITTER_MOCK_MODE === 'true',
      maxRetries: parseInt(process.env.TWITTER_MAX_RETRIES || '3'),
      retryDelay: parseInt(process.env.TWITTER_RETRY_DELAY || '5000')
    };
  }
}

  2. Session Management:

    interface SessionManager {
  initializeSession(): Promise<void>;
  validateSession(): Promise<boolean>;
  refreshSession(): Promise<void>;
  persistCookies(cookies: Cookie[]): Promise<void>;
}

  3. Error Handling:

    interface AuthErrorHandler {
  handleACIDChallenge(): Promise<void>;  // Handle Error 399
  handleSuspiciousLogin(): Promise<void>;
  implementRetryStrategy(): Promise<void>;
}

  4. Content Generation Service (src/services/ai/tweetGenerator.ts)

    interface TweetGenerator {
  generateMarketTweet(data: MarketData): Promise<string>;
  generateEngagementResponse(context: Context): Promise<string>;
  validateContent(tweet: string): Promise<ValidationResult>;
}

interface ContentRules {
  maxEmojis: 0;              // No emojis allowed
  maxHashtags: 0;            // No hashtags allowed
  minInterval: 300000;       // 5-minute minimum between tweets
  requireUnique: true;       // Ensure unique post formats
}

    Content Validation:

    class ContentValidator {
  private readonly rules: ContentRules = {
    maxEmojis: 0,
    maxHashtags: 0,
    minInterval: 300000,
    requireUnique: true
  };

  async validateTweet(content: string): Promise<ValidationResult> {
    // Ensure no emojis
    if (this.containsEmojis(content)) {
      return { valid: false, reason: 'Contains emojis' };
    }

    // Ensure no hashtags
    if (this.containsHashtags(content)) {
      return { valid: false, reason: 'Contains hashtags' };
    }

    // Check uniqueness
    if (!await this.isUniqueFormat(content)) {
      return { valid: false, reason: 'Similar format exists' };
    }

    return { valid: true };
  }
}

  5. Rate Limiting Service

    interface RateLimitManager {
  checkLimit(action: TwitterAction): Promise<boolean>;
  trackRequest(action: TwitterAction): void;
  getRemainingQuota(): RateLimit;
}

class TwitterRateLimiter implements RateLimitManager {
  private readonly limits = {
    tweets: { window: 900000, max: 300 },      // 15-minute window
    engagements: { window: 600000, max: 1000 } // 10-minute window
  };
}

Service Interactions

  1. Market Update Pipeline

    Market Data Collection → Data Validation → Content Generation → Spam Check → Post
[Jupiter/Helius] → [Validator] → [TweetGenerator] → [ContentRules] → [Twitter]

  2. Authentication Pipeline

    Load Credentials → Session Check → Auth Flow → Cookie Management → Ready
[.env] → [Validator] → [AuthService] → [SessionManager] → [TwitterService]

  3. Error Recovery Flow

    Error Detection → Classification → Retry Strategy → Session Refresh → Resume
[Monitor] → [ErrorHandler] → [RetryService] → [SessionManager] → [Service]

Integration Features

  • Data Collection

    • Twitter authentication & session management
    • Rate limit handling
    • Mock mode fallback
    • Cookie-based persistence

  • Analysis Pipeline

    • Market sentiment analysis
    • Tweet content validation
    • Character-aware generation
    • Response optimization

  • Engagement System

    • Real-time monitoring
    • Context-aware responses
    • Personality consistency
    • Rate limit awareness

  • Automated Actions

    • Market updates
    • Trading signals
    • Community engagement
    • Performance reporting

4. AI-Powered Trading

  • Decision Engine
    • Multi-model consensus
    • Risk evaluation
    • Market timing
    • Position sizing
  • Portfolio Management
    • Asset allocation
    • Rebalancing
    • Risk management
    • Performance tracking
  • Execution Engine
    • Order splitting
    • Timing optimization
    • Slippage control
    • Fee management
  • Performance Analysis
    • Trade analytics
    • Strategy backtesting
    • Risk metrics
    • Attribution analysis

System Optimizations

1. Performance Enhancements

  • Caching Strategy

    interface CacheConfig {
  ttl: number;
  maxSize: number;
  updateInterval: number;
}
    • Multi-level caching
    • Intelligent prefetching
    • Cache invalidation
    • Memory management

  • Database Optimization

    CREATE INDEX idx_token_analysis ON analysis_results 
USING GiST (token_address, created_at DESC);
    • Query optimization
    • Index management
    • Connection pooling
    • Data partitioning

  • API Management

    interface RateLimitConfig {
  windowMs: number;
  maxRequests: number;
  strategy: 'sliding' | 'fixed';
}
    • Rate limiting
    • Request batching
    • Response compression
    • Circuit breaking

2. Reliability Features

  • Error Handling

    class OperationalError extends Error {
  public readonly isOperational: boolean = true;
  constructor(
    public statusCode: number,
    message: string,
    public source?: string
  ) {
    super(message);
  }
}
    • Error classification
    • Retry mechanisms
    • Fallback strategies
    • Recovery procedures

  • Monitoring System

    interface MetricCollector {
  recordLatency(operation: string, duration: number): void;
  incrementCounter(metric: string): void;
  recordGauge(metric: string, value: number): void;
}
    • Performance metrics
    • Health checks
    • Resource monitoring
    • Alert system

3. Security Measures

  • API Security

    interface SecurityConfig {
  rateLimit: RateLimitConfig;
  apiKeyRotation: number;
  ipWhitelist: string[];
  requestValidation: ValidationConfig;
}
    • Authentication
    • Authorization
    • Input validation
    • Rate limiting

  • Data Protection

    interface EncryptionConfig {
  algorithm: string;
  keyRotation: number;
  saltRounds: number;
}
    • Encryption
    • Key management
    • Access control
    • Audit logging

Known Issues & Solutions

1. AI Model Integration

Issues & Solutions:

interface ModelFallback {
  primary: AIModel;
  fallback: AIModel[];
  threshold: number;
  maxRetries: number;
}
  • Implemented model rotation
  • Response validation
  • Latency monitoring
  • Error recovery

Twitter Content Generation:

interface TweetValidation {
  maxLength: number;
  contentRules: {
    noEmojis: boolean;
    noHashtags: boolean;
    requireUnique: boolean;
  };
  retryConfig: {
    maxAttempts: number;
    backoffMs: number;
  };
}
  • Character-aware generation
  • Market data integration
  • Content validation
  • Rate limit management

2. Transaction Processing

Issues & Solutions:

interface TransactionRetry {
  maxAttempts: number;
  backoffMs: number;
  conditions: RetryCondition[];
}
  • Dynamic gas adjustment
  • MEV protection
  • Confirmation tracking
  • Error recovery

3. Data Management

Issues & Solutions:

interface DataPartitioning {
  strategy: 'time' | 'hash' | 'range';
  interval: string;
  cleanup: CleanupConfig;
}
  • Sharding strategy
  • Backup procedures
  • Recovery protocols
  • Consistency checks

Future Optimizations

1. Performance Roadmap

  • GraphQL API implementation
  • WebSocket optimization
  • Database sharding
  • Cache optimization

2. Scalability Plans

  • Kubernetes deployment
  • Service mesh integration
  • Load balancing
  • Auto-scaling

3. Feature Pipeline

  • Advanced ML models
  • Social integration
  • Analytics dashboard
  • Automated testing

Dependencies & Integration

Core Dependencies

{
  "dependencies": {
    "@pythnetwork/client": "^2.17.0",
    "@solana/buffer-layout": "^4.0.1",
    "@switchboard-xyz/oracle": "^2.1.13",
    "amqplib": "^0.10.3",
    "bull": "^4.12.0",
    "graphql": "^16.8.1",
    "ioredis-cluster": "^2.1.1",
    "node-machine-learning": "^1.0.0",
    "technicalindicators": "^3.1.0",
    "ws": "^8.16.0"
  }
}

Development Tools

{
  "devDependencies": {
    "@types/amqplib": "^0.10.4",
    "@types/bull": "^4.10.0",
    "@types/ws": "^8.5.10",
    "artillery": "^2.0.3",
    "autocannon": "^7.14.0",
    "madge": "^6.1.0",
    "typescript-transform-paths": "^3.4.6"
  }
}

These components work together to provide:

  • Technical analysis
  • Machine learning capabilities
  • Message queue processing
  • Real-time communication
  • Performance testing
  • Dependency analysis