Observability for Node.js & Python

A comprehensive, secure observability solution for Node.js and Python applications with real-time monitoring, performance metrics, and agent authentication.

Features

• Real-time request monitoring with IP-based filtering and shared secret authentication
• Performance metrics and slow request tracking
• Detailed request/response logging with full context
• Snapshot functionality for state preservation and analysis
• Modern React dashboard with dark/light mode support
• Agent authentication with whitelist and shared secret validation
• IPv6/IPv4 compatibility for proper IP filtering
• Cross-platform support for Windows, Linux, and macOS
• Automatic URL detection for agent registration
• Agent management with deletion capabilities

Quick Start

One-Command Startup

# Start everything (Dashboard + Demo Apps)
./run-demo-apps.sh

This will launch:

• Dashboard: http://localhost:3000
• Node.js Demo: http://localhost:3010
• Python Demo: http://localhost:3011

Manual Setup

1. Install Dependencies

# Dashboard (React + Node.js)
cd observability/dashboard
npm install
npm run build

# Python Demo
cd ../../demo-python-app
pip install -r requirements.txt

# Node.js Demo (already has dependencies)
cd ../demo-nodejs-app
npm install

2. Start Dashboard

cd observability/dashboard
node server.js

3. Start Demo Apps (in separate terminals)

# Terminal 1 - Node.js Demo
cd demo-nodejs-app
node server.js

# Terminal 2 - Python Demo
cd demo-python-app
python app.py

Project Structure

observability-for-node-js/
├── README.md                          # This file
├── CONTRIBUTING.md                     # Contributing guidelines
├── LICENSE                            # MIT License
├── run-demo-apps.sh                   # Quick start script
├── demo-nodejs-app/                   # Node.js demo application
│   ├── server.js
│   ├── utils/agent.js
│   └── package.json
├── demo-python-app/                   # Python demo application
│   ├── app.py
│   ├── utils/agent.py
│   └── requirements.txt
└── observability/                      # Core observability system
    ├── agent/                         # Agent implementations
    │   ├── agent.js                   # Node.js agent
    │   ├── agent.py                   # Python agent
    │   └── agent.php                  # PHP agent (coming soon)
    └── dashboard/                     # React dashboard
        ├── src/                       # React source code
        ├── build/                     # Built React app
        ├── database.js               # SQLite database
        ├── setup.js                  # API routes
        ├── server.js                 # Dashboard server
        └── package.json

Security Configuration

Agent Whitelist Setup

Configure observability/dashboard/agent-whitelist.json:

{
  "allowedAgents": [
    {
      "agentId": "demo-nodejs-app-agent",
      "sharedSecret": "nodejs-demo-secret-key-9999",
      "allowedSources": ["localhost", "127.0.0.1", "::1"],
      "description": "Demo Node.js application"
    },
    {
      "agentId": "demo-python-app-agent", 
      "sharedSecret": "python-demo-secret-key-9999",
      "allowedSources": ["localhost", "127.0.0.1", "::1"],
      "description": "Demo Python Flask application"
    }
  ],
  "defaultAllowedSources": ["localhost", "127.0.0.1", "::1"],
  "strictMode": true,
  "logBlockedRequests": true,
  "requireSharedSecret": true
}

Integration Guide

Node.js Applications

const ObservabilityAgent = require('./utils/observability/agent');

const observabilityAgent = new ObservabilityAgent({
    agentId: 'your-agent-id',
    appName: 'Your App Name',
    appPort: 3000,
    sharedSecret: 'your-secret-key-here',
    observabilityUrl: 'http://localhost:3000',
    heartbeatInterval: 5000,
    batchSize: 10,
    flushInterval: 2000
});

app.use(observabilityAgent.middleware());

Python Applications

from utils.agent import ObservabilityAgent

observability_agent = ObservabilityAgent({
    'agentId': 'your-agent-id',
    'appName': 'Your App Name',
    'appPort': 3000,
    'sharedSecret': 'your-secret-key-here',
    'observabilityUrl': 'http://localhost:3000',
    'heartbeatInterval': 5000,
    'batchSize': 10,
    'flushInterval': 2000
})

before_request, after_request = observability_agent.middleware()
app.before_request(before_request)
app.after_request(after_request)

Demo Applications

Node.js Demo App (Port 3010)

• Agent ID: demo-nodejs-app-agent
• Shared Secret: nodejs-demo-secret-key-9999
• Test Endpoints:
  • GET /slow-endpoint - 1 second delay
  • GET /request-421 - Returns 421 status
  • GET /request-401 - Returns 401 status
  • GET /get-something - Basic GET endpoint
  • POST /post-something - Basic POST endpoint
  • PUT /put-something - Basic PUT endpoint
  • PATCH /patch-something - Basic PATCH endpoint
  • DELETE /delete-something - Basic DELETE endpoint

Python Flask Demo App (Port 3011)

• Agent ID: demo-python-app-agent
• Shared Secret: python-demo-secret-key-9999
• Same test endpoints as Node.js demo

Dashboard Features

Real-time Monitoring

• Request metrics and success/failure rates
• Response time tracking with interactive charts
• Agent status and heartbeat monitoring
• Automatic data refresh every 15 seconds

Request Logs

• Detailed logs of all monitored requests
• Filter by agent, status code, and time range
• Search functionality for specific endpoints
• Pagination for large datasets

Performance Analysis

• Identify slow endpoints and performance bottlenecks
• Configurable slow request thresholds
• Performance trends and patterns

Agent Management

• View all registered agents with URLs
• Monitor agent status and last heartbeat
• Delete agents from the system
• Automatic URL detection during registration

Snapshots

• Create point-in-time snapshots
• Export snapshot data as JSON
• Compare application states over time

API Reference

Authentication Endpoints

Agent Registration

POST /api/agents/register
Content-Type: application/json

{
  "agentId": "your-agent-id",
  "appName": "Your App Name", 
  "appPort": 3000,
  "appUrl": "http://localhost:3000",
  "sharedSecret": "your-secret-key-here"
}

Agent Heartbeat

POST /api/agents/{agentId}/heartbeat
Content-Type: application/json
x-shared-secret: your-secret-key-here

{
  "sharedSecret": "your-secret-key-here"
}

Metrics Submission

POST /api/metrics
Content-Type: application/json

{
  "agentId": "your-agent-id",
  "sharedSecret": "your-secret-key-here",
  "metrics": [
    {
      "data": {
        "method": "GET",
        "endpoint": "/api/users",
        "statusCode": 200,
        "duration": 150,
        "timestamp": "2025-01-19T05:15:02.240Z",
        "userAgent": "Mozilla/5.0...",
        "ip": "127.0.0.1"
      }
    }
  ]
}

Monitoring Endpoints

• GET /api/stats - Get current statistics and agent list
• GET /api/logs - Get paginated request logs
• GET /api/slow - Get slow requests (threshold: 500ms)
• DELETE /api/agents/{agentId} - Delete an agent
• POST /api/snapshots - Create a new snapshot
• GET /api/snapshots - List all snapshots
• GET /api/snapshots/:id - Get specific snapshot
• GET /api/snapshots/:id/export - Export snapshot as JSON
• DELETE /api/snapshots/:id - Delete a snapshot

Security Features

Multi-Factor Authentication

• IP Whitelist: Only allow agents from specified IP addresses/domains
• Shared Secret: Require cryptographic secret for agent registration and communication
• Strict Mode: Configurable enforcement of authentication policies
• Audit Logging: Complete logging of all authentication attempts

Network Compatibility

• IPv6 Support: Automatic conversion of IPv6-mapped IPv4 addresses
• Flexible Sources: Support for localhost, domains, and wildcards
• Request Validation: Proper handling of edge cases and undefined URLs

Dependencies

Dashboard

• Node.js 14+
• React 18+
• SQLite3 (for agent storage)
• Express.js (for API server)

Agents

• Node.js: Express 4.x (optional, for automatic monitoring)
• Python: Flask 2.x
• No external database required

Guarantees

What We Guarantee

• Performance: Minimal impact on application performance (< 5% overhead)
• Reliability: Built-in error handling and automatic recovery
• Security: End-to-end encryption for agent communication and IP-based access control
• Privacy: All data stays within your infrastructure - no external data transmission
• Compatibility: IPv6/IPv4 dual-stack support for modern networks

Non-Guarantees

• Persistence: Logs and snapshots are stored on local filesystem (no cloud backup)
• Scalability: Best suited for small to medium applications (high-traffic apps may need log rotation)
• High Availability: No built-in clustering (single dashboard instance)

Contributing

Contributions are welcome! Please read our contributing guidelines before submitting pull requests.

Development Setup

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test with the demo applications
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Author

Anastasios Bolkas

• GitHub: @anastasios-b
• Portfolio: anastasios-bolkas.tech

---

Getting Help

• Issues: Report bugs and request features on GitHub Issues
• Documentation: Check this README and inline code comments
• Demo: Run ./run-demo-apps.sh to see everything in action