System Design Fundamentals: Scaling Zero to Millions

Every successful application starts with zero users. The question is not whether your system will need to scale - it is when. And when that moment comes, the decisions you made early on will either accelerate your growth or become the ceiling you cannot break through.

I have scaled systems through every phase - from a monolith at Definitive Healthcare to event-driven microservices handling millions at BNP Paribas, and through my own products xThreads.app andPostQueue.app. This post maps the architecture decisions at each stage of growth, from zero users to millions.

What you will learn: The three phases of scaling: startup, growth, enterprise · Why monolithic is the right choice at phase 1 · When and how to introduce read replicas, caching, and message queues · Database scaling: connection pooling, replication, sharding · Microservices: decompose without downtime (Strangler Fig) · Fault tolerance: circuit breakers, retries, health checks · The one metric that matters at each phase

1. Phase 1: Zero to Thousands (The Startup Phase)

In the startup phase, your only real job is to ship something users want. Premature scaling is the most common mistake - you do not know which parts of your system will be bottlenecks until you have real traffic.

The right architecture for this phase is a monolith. A well-structured monolith lets you iterate fast, deploy in seconds, and debug easily. It is not a permanent state - it is the fastest path to product-market fit.

What Not to Do

I have seen startups burn months building a microservices infrastructure for an application with 100 users. Here is what you should not do at this stage:

• Do not set up Kubernetes. Use a single VM or a PaaS like Railway, Fly.io, or Heroku.
• Do not split into microservices. A monolith with clear module boundaries is faster to build and iterate.
• Do not introduce Kafka. A relational database with background jobs (Sidekiq, Celery) handles 99% of use cases.
• Do not premature-optimize database queries. Use an ORM, write clear models, and optimize only when you measure a real bottleneck.

2. Phase 2: Thousands to Hundreds of Thousands (The Growth Phase)

At this stage, you have proven product-market fit. Traffic is growing, and you are starting to feel the pressure. The database is the first bottleneck. Response times creep up. Background jobs pile up.

The goal of this phase is to buy time - extend the life of your monolithic architecture while adding the infrastructure you will need for the next phase. Three changes make the biggest impact:

Database: Read Replicas

Most applications are read-heavy. A single database handles both reads and writes. By adding read replicas, you offload read traffic and keep the primary database free for writes:

import psycopg2
from contextlib import contextmanager

class DatabaseRouter:
    """Route reads to replicas, writes to primary."""
    def __init__(self, primary_url: str, replica_urls: list[str]):
        self.primary = psycopg2.connect(primary_url)
        self.replicas = [psycopg2.connect(url) for url in replica_urls]
        self._replica_index = 0

    @contextmanager
    def read_connection(self):
        conn = self.replicas[self._replica_index % len(self.replicas)]
        self._replica_index += 1
        yield conn

    @contextmanager
    def write_connection(self):
        yield self.primary

# Usage
router = DatabaseRouter(
    primary_url="postgres://primary/db",
    replica_urls=["postgres://replica-1/db", "postgres://replica-2/db"],
)

# All reads go to replicas
with router.read_connection() as conn:
    posts = conn.execute("SELECT * FROM posts LIMIT 20")

# Writes go to primary
with router.write_connection() as conn:
    conn.execute("INSERT INTO posts ...")

Caching: The Multi-Layer Approach

Caching is the most cost-effective performance optimization. A well-cached system can handle 10x the traffic with the same database. Use multiple layers:

# Layer 1: CDN for static assets
# CloudFront / Cloudflare caches images, CSS, JS at the edge

# Layer 2: Redis for API responses and session data
import redis.asyncio as redis

cache = redis.Redis(host="localhost", port=6379)

async def get_user_feed(user_id: str) -> list[dict]:
    """Get user feed with Redis caching."""
    cached = await cache.get(f"feed:{user_id}")
    if cached:
        return json.loads(cached)

    feed = await db.query("SELECT * FROM posts WHERE user_id = ? ORDER BY created_at DESC LIMIT 50")
    await cache.setex(f"feed:{user_id}", 60, json.dumps(feed))  # 60-second TTL
    return feed

# Layer 3: Browser caching via HTTP headers
# Cache-Control: public, max-age=3600

3. Phase 3: Hundreds of Thousands to Millions (The Enterprise Phase)

At enterprise scale, the monolith becomes the bottleneck. Deployments are risky because a change in one area can bring down the entire application. The database is struggling despite replicas. Background jobs are competing for resources.

The solution is decomposition - splitting the monolith into focused services that can scale independently.

The Strangler Fig Pattern

You cannot rewrite the entire system at once. The Strangler Fig pattern lets you replace the monolith piece by piece, with zero downtime:

# Phase 1: Create a new microservice alongside the monolith
# Route NEW traffic from one feature to the new service

# Phase 2: Migrate existing data
async def migrate_users_to_new_service():
    """Copy users from monolith DB to new user service."""
    old_users = await monolith_db.query("SELECT * FROM users")
    for user in old_users:
        await user_service.create_user(user)

# Phase 3: Switch all traffic
# Update the routing layer to send ALL user traffic to the new service

# Phase 4: Remove the old code
# Delete the user module from the monolith
# monolith code deletion
# delete_module("user")  # Bye-bye, old code!

Asynchronous Processing with Message Queues

At this scale, synchronous request-response is no longer sufficient for everything. Heavy work moves to queues:

import asyncio
from dataclasses import dataclass

@dataclass
class Task:
    type: str         # "send_email", "generate_report", "process_image"
    payload: dict
    priority: int     # 1 (high) to 5 (low)

class AsyncWorker:
    def __init__(self):
        self.queue = asyncio.PriorityQueue()

    async def enqueue(self, task: Task):
        await self.queue.put((task.priority, task))

    async def process_loop(self):
        """Continuously process tasks."""
        while True:
            _, task = await self.queue.get()
            try:
                await self.execute(task)
            except Exception as e:
                print(f"Task failed: {task.type} - {e}")
                # Re-enqueue with backoff for retries

    async def execute(self, task: Task):
        if task.type == "send_email":
            await send_email(task.payload)
        elif task.type == "generate_report":
            await generate_report(task.payload)

# Usage
worker = AsyncWorker()

# Fire and forget - the user gets a response immediately
async def handle_user_signup(user: dict):
    await db.create_user(user)
    # Queue async work
    await worker.enqueue(Task("send_email", {"to": user["email"], "template": "welcome"}, 2))
    await worker.enqueue(Task("generate_report", {"user_id": user["id"]}, 4))
    return {"status": "created"}

4. Fault Tolerance: Design for Failure

At millions of users, failures are not rare - they are inevitable. Networks partition. Databases time out. Services crash. The question is not whether your system will fail, but how gracefully it degrades.

Circuit Breaker Pattern

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"       # Normal operation
    OPEN = "open"           # Failing - reject immediately
    HALF_OPEN = "half_open" # Testing if service recovered

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 30):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout

    def call(self, fn, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
            else:
                raise CircuitBreakerOpenError("Service temporarily unavailable")

        try:
            result = fn(*args, **kwargs)
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED  # Recovered!
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
                print(f"Circuit OPEN - {self.failure_count} failures")
            raise e

# Usage
breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30)

def get_user(user_id: str) -> dict:
    return breaker.call(user_service.get_user, user_id)

Retry with Exponential Backoff

import random
import time

def retry_with_backoff(fn, max_retries: int = 3, base_delay: float = 0.5):
    """Retry a function with exponential backoff and jitter."""
    for attempt in range(max_retries):
        try:
            return fn()
        except (ConnectionError, TimeoutError) as e:
            if attempt == max_retries - 1:
                raise  # Last attempt - let the error propagate
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            print(f"Attempt {attempt + 1} failed. Retrying in {delay:.1f}s...")
            time.sleep(delay)

5. The One Metric Per Phase

Different stages of growth demand focus on different metrics. Trying to optimize for everything at once is a recipe for nothing being optimized well:

6. Summary: What to Remember