web-framework-bench

Comprehensive benchmarking toolkit for comparing web frameworks across runtime performance, build metrics, hosting providers, and LLM code understanding.

Built for PickAFramework.com and extracted as a standalone, config-driven tool anyone can use to benchmark their own framework deployments.

What It Measures

• Runtime Performance — TTFB, FCP, LCP, CLS via Playwright; RPS via Autocannon; Lighthouse scores (performance, SEO, accessibility, best practices)
• Build Metrics — Client/server bundle sizes, dependency counts, build times
• Hosting Comparison — Cold start times, TTFB across providers (Vercel, Netlify, Cloudflare, Render, Railway, Coolify)
• LLM Understanding — How well AI models understand each framework's codebase (exploration time, token usage, answer accuracy)

Quick Start

git clone https://github.com/EmilMN/PickAFramework-Benchmark.git
cd PickAFramework-Benchmark
npm install

# Copy example config and add your URLs
cp bench.config.example.ts bench.config.ts
# Edit bench.config.ts with your deployed framework URLs

# Run benchmarks
npm run metrics:collect           # Runtime perf (all frameworks)
npm run metrics:collect -- nextjs # Single framework
npm run metrics:build             # Bundle sizes & deps
npm run hosting:collect           # Hosting provider comparison

Configuration

All framework URLs, build targets, and hosting providers are defined in bench.config.ts. Copy the example to get started:

cp bench.config.example.ts bench.config.ts

The example config shows the full structure with placeholder URLs. Replace example.com with your actual deployed framework URLs.

import { defineConfig } from './src/types/config.js'

export default defineConfig({
  frameworks: [
    {
      id: 'nextjs',
      baseUrl: 'https://nextjs.yourdomain.com',
      modes: { default: '/', ssg: '/ssg', ssr: '/ssr', isr: '/isr' },
    },
    // ...
  ],
  buildTargets: [/* ... */],
  hostingProviders: [/* ... */],
})

Available Scripts

Script	Description
npm run metrics:collect	Runtime performance (Playwright + Autocannon + Lighthouse)
npm run metrics:build	Bundle sizes and dependency counts
npm run metrics:build-times	Build time measurement
npm run build-times:quick	Quick JSON build time output
npm run hosting:collect	Hosting provider comparison
npm run llm:benchmark	LLM code understanding benchmark
npm run llm:evaluate	Review LLM answers
npm run llm:score	Score LLM answers (keyword or LLM-as-judge)
npm run metrics:split	Split combined metrics into per-framework files
npm run metrics:combine	Combine per-framework files back
npm run typecheck	TypeScript type checking

CLI Options

Runtime metrics:

npm run metrics:collect -- nextjs           # Single framework
npm run metrics:collect -- --only-autocannon # Server TTFB/RPS only
npm run metrics:collect -- --only-lighthouse # Lighthouse scores only
npm run metrics:collect -- --no-cdn          # Bypass CDN (origin only)

LLM benchmark (requires ANTHROPIC_API_KEY):

npm run llm:benchmark -- nextjs        # Single framework
npm run llm:benchmark -- --trials 5    # Multiple trials
npm run llm:score                      # Keyword-based scoring
npm run llm:score -- --use-judge       # LLM-as-judge scoring

Requirements

• Node.js 18+
• Deployed framework apps accessible via URL (for runtime metrics)
• A pnpm monorepo with framework apps (for build metrics)
• ANTHROPIC_API_KEY env var (only for LLM benchmarks)

License

MIT