Dynamic routing in OpenResty with Redis

Dynamic routing that scales to large route counts and rapid backend churn, with no config regeneration and no proxy reloads.

Tenants map to apps, apps map to addresses, and both tables live in Redis rather than nginx.conf. The config stays the same size at ten tenants or a hundred thousand, and moving a tenant between apps becomes one write.

Not every tenant belongs on the same application. Some are pinned to the version they were onboarded against, some are early on the next one, and a few enterprise accounts are still served by the system you are trying to retire. That is a routing table keyed by tenant, and the moment it lives in nginx.conf you are regenerating and reloading the proxy every time an account moves. So move the lookup onto the request path instead.

flowchart LR
    C["acme.example.com"] --> A["access_by_lua<br/>tenant = acme"]
    A -->|"1 · check"| L["lua_shared_dict<br/>route_cache"]
    L -->|"2 · miss"| R[("Redis<br/>tenant:acme → orders-v2<br/>app:orders-v2 → addr")]
    R -->|"3 · fill"| L
    L -->|"4 · peer"| B["balancer_by_lua<br/>set_current_peer"]
    B --> A1["orders-v2"]
    B --> A2["orders-v1"]
    B --> A3["legacy-billing"]

Two tables, not one

The obvious data model puts a backend address on each tenant, and it holds up until an app moves. Tenants are numerous and apps are few, so that address ends up duplicated across thousands of keys, and changing it means rewriting every one of them. Split the table in two instead:

# The apps. A handful of keys, changing rarely.
SET app:orders-v2      orders-v2.internal:8080
SET app:orders-v1      orders-v1.internal:8080
SET app:legacy-billing legacy-billing.internal:8080

# The tenants. Many keys, changing constantly.
HSET tenant:acme    app orders-v2      status active
HSET tenant:globex  app orders-v1      status active
HSET tenant:initech app legacy-billing status suspended

Migrating one tenant to a different app is HSET tenant:acme app orders-v1. Moving an app to a new address is a single SET, and every tenant on it follows without touching a tenant key. The status field gives you a suspension switch that takes effect at the edge, so a non-paying account never reaches the application at all.

Resolving both levels in one round trip

Two dependent reads would normally mean two round trips, since the second key is not known until the first returns. Push the indirection into Redis:

-- Redis-side script. Load once with SCRIPT LOAD and call it by SHA in
-- production; spelled out here for readability.
local app, status = unpack(redis.call("HMGET", KEYS[1], "app", "status"))
if not app then
    return {"", "", ""}                                  -- unknown tenant
end
if status ~= "active" then
    return {"", app, status}                             -- known, not serving
end
return {redis.call("GET", "app:" .. app) or "", app, status}

A missing hash field comes back as false rather than nil inside Redis Lua, which is why the unknown-tenant test reads the way it does. Returning a fixed three-element array avoids the other trap: a Lua table with a nil hole gets truncated on the way out.

The OpenResty side

The tenant comes from a server_name regex capture, the same wildcard-DNS shape as in tenant subdomains with wildcard DNS and NGINX, so adding an account needs no DNS change either:

http {
    lua_package_path "/etc/openresty/lua/?.lua;;";
    lua_shared_dict route_cache 10m;

    upstream tenant_app {
        server 0.0.0.1;          # placeholder, replaced on every request
        balancer_by_lua_block {
            local balancer = require "ngx.balancer"
            local peer = ngx.ctx.peer
            local ok, err = balancer.set_current_peer(peer.host, peer.port)
            if not ok then
                ngx.log(ngx.ERR, "failed to set peer: ", err)
                return ngx.exit(500)
            end
        }
        keepalive 64;
    }

    server {
        listen 443 ssl;
        server_name ~^(?<tenant>[^.]+)\.example\.com$;

        ssl_certificate     /etc/nginx/certs/wildcard.example.com.crt;
        ssl_certificate_key /etc/nginx/certs/wildcard.example.com.key;

        location / {
            access_by_lua_block {
                local peer, err = require("route").lookup(ngx.var.tenant)
                if not peer then
                    if err == "unknown tenant" then
                        return ngx.exit(ngx.HTTP_NOT_FOUND)
                    elseif err == "suspended" then
                        return ngx.exit(ngx.HTTP_FORBIDDEN)
                    end
                    ngx.log(ngx.ERR, "lookup failed for ", ngx.var.tenant,
                            ": ", err)
                    return ngx.exit(ngx.HTTP_BAD_GATEWAY)
                end

                ngx.ctx.peer = peer
                ngx.req.set_header("X-Tenant", ngx.var.tenant)
                ngx.req.set_header("X-App", peer.app)
            }

            proxy_pass http://tenant_app;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
        }
    }
}

ngx.req.set_header overwrites whatever the client sent, so a request arriving with its own X-Tenant cannot spoof one. Worth confirming rather than assuming.

The lookup itself caches in a shared dict, so the steady state costs no Redis traffic at all:

-- /etc/openresty/lua/route.lua
local redis = require "resty.redis"
local cache = ngx.shared.route_cache

local TTL  = 5        -- ceiling on how stale a routing decision can be
local MISS = "\0"     -- sentinel for a negatively cached tenant

local RESOLVE = [[ ... the Redis-side script from above ... ]]

local _M = {}

local function decode(blob)
    local addr, app, status = blob:match("^([^|]*)|([^|]*)|([^|]*)$")
    if status ~= "active" then
        return nil, status
    end
    if addr == "" then
        return nil, "app " .. app .. " has no address"
    end
    local host, port = addr:match("^(.+):(%d+)$")
    return { host = host, port = tonumber(port), app = app }
end

function _M.lookup(tenant)
    local hit = cache:get(tenant)
    if hit == MISS then
        return nil, "unknown tenant"
    elseif hit then
        return decode(hit)
    end

    local red = redis:new()
    red:set_timeouts(200, 200, 200)      -- connect, send, read

    local ok, err = red:connect("127.0.0.1", 6379)
    if not ok then
        return nil, err
    end

    local res, err = red:eval(RESOLVE, 1, "tenant:" .. tenant)
    if not res then
        return nil, err
    end

    -- Back to the pool rather than closed. Do not touch `red` after this.
    red:set_keepalive(60000, 100)

    local addr, app, status = res[1], res[2], res[3]
    if app == "" then
        cache:set(tenant, MISS, TTL)
        return nil, "unknown tenant"
    end

    local blob = addr .. "|" .. app .. "|" .. status
    cache:set(tenant, blob, TTL)
    return decode(blob)
end

return _M

The negative cache matters more than the positive one. Tenant subdomains are guessable and a wildcard record answers for all of them, so without MISS every request for a name that does not exist becomes a Redis query.

WarningThe lookup cannot live in the balancer phase

balancer_by_lua* runs without the cosocket API, so no ngx.socket.tcp, no Redis client, no ngx.sleep. set_by_lua* is blocking and has the same restriction. Any I/O has to happen in access_by_lua* or rewrite_by_lua* and be handed forward through ngx.ctx. Note also that balancer_by_lua* runs again on every retry, so keep it free of side effects.

NoteThe tenant is still only a routing hint

$tenant comes from the Host header, so it is attacker-controlled. It decides which app serves the request, never which tenant the caller is entitled to. The session check against it still belongs in the application.

Where this earns its keep

The interesting case is not a handful of tenants. It is the point where a Kubernetes ingress or gateway controller stops scaling, and both common architectures have a wall.

A template-and-reload controller such as ingress-nginx renders every Ingress into a single nginx.conf. At tens of thousands of tenants that file is enormous, reload time grows with it, and each reload spawns a new worker generation while the old one drains. Reload during churn means two worker sets resident at once and every keepalive connection on the old set eventually dropped. Tellingly, ingress-nginx already conceded half of this: endpoint changes are pushed into a Lua shared dict instead of triggering a reload, precisely because pod churn was reloading constantly. Putting the tenant table in Redis is the same move applied one level up.

An xDS controller such as Envoy Gateway or Istio avoids reloads, but the control plane still holds the full configuration, recomputes it on change, and pushes to every proxy that could be affected. Push amplification under churn is its own well-known scaling problem, and a snapshot carrying tens of thousands of per-tenant routes is expensive to build, serialize, and diff.

Pulling one route per request inverts the direction:

• Config size is O(1) in tenant count. The config above is the whole thing at ten tenants or a hundred thousand.
• Onboarding is a write. A new account is live within the cache TTL, with no reload and no push fan-out.
• Migrations are a write too. Moving a cohort onto the next app version is a loop over HSET, and rolling it back is the same loop.

The shape that fits is high tenant cardinality against low app cardinality, changing faster than a control plane converges.

What you give up

This is a real trade, not a free win.

Redis lands in the request path. The cache hides that for hits, but a cold worker with Redis down fails closed, so you want a replica, tight timeouts, and a deliberate decision about whether to keep serving stale entries through an outage rather than returning 502.

You also give up the control plane. No admission webhook rejecting a tenant pointed at an app that does not exist, no kubectl get showing which accounts are on which version, no controller reconciling certificates or per-tenant metrics. Something has to own those keys and stay authoritative about them, and that something is now yours to build and operate. The honest framing is that you are hand-rolling an xDS split: a control plane that writes routes, a data plane that reads them. Worth it when tenant cardinality or churn is genuinely past what an off-the-shelf controller handles, and hard to justify below that.