Fine tuning Envoy proxy concurrency for the edge and mesh

How Envoy determines the number of worker threads, and what value to set for edge proxies, sidecars, and workloads with long-lived connections.

Threading model

---
config:
  flowchart:
    curve: linear
    nodeSpacing: 40
    rankSpacing: 60
---
flowchart TB
    subgraph KERNEL["Kernel space"]
        direction TB
        SOCK[("Listener socket<br/>accept queue")]
    end

    subgraph USER["User space"]
        direction TB

        subgraph MT["Main thread"]
            direction TB
            xDS["xDS"]
            RT["Runtime"]
            SF["Stat flush"]
            AD["Admin"]
        end

        subgraph W0["Worker thread 1 · event loop"]
            direction TB
            EP["epoll_wait()"]
            CN["Connections"]
            EP -->|"accept()"| CN
        end

        W1["Worker thread 2"]
        W2["Worker thread 3"]
        W3["Worker thread 4"]

        subgraph FF["File flush thread(s)"]
            direction TB
            F0["flush"]
        end
    end

    MT -->|"TLS update"| W0
    MT --> W1
    MT --> W2
    MT --> W3

    SOCK -.->|"socket readable"| EP
    SOCK -.-> W1
    SOCK -.-> W2
    SOCK -.-> W3

    W0 --> FF
    W1 --> FF
    W2 --> FF
    W3 --> FF

A single Envoy process runs a main thread alongside a pool of worker threads, sized by --concurrency. The main thread handles xDS updates, stats flushing, and the admin interface; it never handles a request. Each worker runs its own single-threaded, non-blocking event loop, and once a connection is accepted on a worker it belongs to that worker for its entire lifetime, never switching to another worker thread. Disk writes get their own threads too, so neither of the above ever blocks on I/O: every file Envoy writes, primarily access logs, has its own independent blocking flush thread.

Per-connection ownership on the workers is what keeps the hot path lock-free. Configuration that workers need on every request, such as cluster manager state and stats counters, lives in thread-local storage. The main thread computes an update once and posts it as a closure to every worker, and each worker installs its own copy into a thread-local slot. A worker never contends with another worker or with the main thread to read it. This synchronization between the main thread and worker threads is inspired by the Linux kernel’s read-copy-update (RCU) mechanism. More workers means more independent copies of that state, not more contention over one shared copy.

How many workers actually get spawned

An unset --concurrency doesn’t default to a single number. On Linux, Envoy resolves it as the minimum of three separate inputs: the hardware thread count, the CPU affinity (cpuset) size, and the cgroup CPU limit. On any other platform it’s just the hardware thread count. That minimum resolves differently depending on where the process is actually running:

• A physical laptop or bare-metal box: There’s normally no cgroup CPU quota and no cpuset restriction, so both of those terms are effectively “all cores” and the minimum collapses to the hardware thread count. One worker per core.
• A VM: Same logic, but “hardware threads” means whatever the hypervisor exposes as vCPUs, not the host’s physical core count. A 4-vCPU VM on a 64-core hypervisor gets 4 workers, not 64.
• A container in Kubernetes: This is where the third term stops being a formality. resources.limits.cpu becomes a cgroup CPU quota, and Envoy reads that quota back as if the machine only had that many cores, regardless of how many the node actually has. If the pod also pins to specific cores (a static CPU manager policy, or an explicit cpuset), that shows up as the cpuset term instead and can constrain it further.

# Explicit, bypassing all of the above.
envoy --concurrency 4

# Turn off just the cgroup component of autodetection.
ENVOY_CGROUP_CPU_DETECTION=false envoy

Passing --concurrency 0 explicitly still runs one worker. Envoy will not run with zero.

Picking a value for concurrency

Edge proxy: Envoy is typically the only tenant of the host machine, so matching worker count to physical core count maximizes throughput and minimizes context-switching overhead. Leaving --concurrency unset already gets you there on bare metal or a dedicated VM, since the hardware-thread term is the binding one.

Sidecar proxy: The opposite instinct is right here. A sidecar is one of two containers in a pod, sitting next to an application that usually needs most of the CPU itself, and it typically doesn’t push enough traffic to saturate even a couple of workers. Istio’s own sidecar injection defaulted concurrency to a flat 2 for years, independent of node size, precisely because most sidecars never need more than that regardless of how large the node underneath them happens to be. More recent versions leave it unset and let Envoy’s own cgroup-based detection match the sidecar’s own CPU limit instead, which lands on the same small number as long as that limit is actually set.

How much of that concurrency actually gets used also depends on what’s next to the sidecar. A single-threaded application (Python Flask, Node.js) can only ever have one request in flight on its own side at a time, so extra sidecar workers just sit there with nothing to hand connections off to. A multi-threaded application (Java, Go) can actually make use of a sidecar that’s able to hand it more than one connection at once. If traffic volume is genuinely high and the CPU and memory allocation can support it, it’s worth trying --concurrency somewhere in the 2-4 range and measuring the effect, rather than assuming higher is better.

Long-lived persistent connections (HTTP/2, gRPC-heavy): The right lever here depends on connection count, not just traffic volume. A connection sticks to whichever worker accepted it for its entire life, and accept-time distribution just relies on the kernel to balance across listener sockets. A handful of long-lived streams landing unevenly across a wide worker pool leaves some workers doing all the work and others doing none, no matter how high --concurrency goes:

# Force even distribution across workers instead of trusting accept-time balancing.
listener:
  connection_balance_config:
    exact_balance: {}

exact_balance adds a lock on the accept path to hand connections out round-robin, which is a real cost against raw throughput. It’s worth paying specifically when connections are long-lived and few relative to --concurrency.

With genuinely high volumes of concurrent long-lived connections instead of just a handful, more workers can actually help spread that load, but only if there’s enough CPU and memory allocated to cover each additional worker’s own connection pools; more workers without the CPU and memory to back them just trades one bottleneck for another.

What goes wrong at the extremes

If you leave --concurrency unset and don’t set a CPU limit on the container, the cgroup input has nothing to work with. Envoy then falls back to the cpuset size or the node’s total hardware thread count, whichever is smaller. On a large shared node, that number is not small. It’s the node’s full core count, even if the pod itself only needs a fraction of that. Envoy won’t warn you about it either.

WarningEvery worker is a standing cost, not just a scheduling slot

A worker thread that never gets its own CPU time doesn’t sit there for free. It holds its own thread-local copy of the cluster manager, stats, and (for TLS-terminating listeners) certificate context, and it runs its own idle event loop. Oversizing --concurrency on a small pod means paying that memory and scheduling cost N times over for CPU the container was never granted, and if the process ends up with more runnable threads than the cgroup’s actual quota allows, the extra threads compete for it and get throttled instead of simply sitting idle.

The fix in Kubernetes is to set --concurrency explicitly rather than relying on autodetection, sized to whichever case above actually applies. It also helps to set requests.cpu equal to limits.cpu, what Kubernetes calls Guaranteed QoS. Envoy’s autodetection reads the CPU limit, not the request, so if the two differ, autodetection can size workers to a number the pod isn’t actually guaranteed to get. Setting them equal keeps autodetection honest, in case it’s ever relied on again.

Either way, you can check what actually started. /server_info returns the running configuration as JSON, including a concurrency field. That’s the one place to confirm the worker count matches what was intended, not just what the node happened to offer.