Scaling WebSockets to 1 Million Concurrent Connections in Go with Epoll & Zero-Allocation Netpoll

In interactive real-time architectures—such as financial market order books, live trading feeds, multi-tenant gaming servers, and WebRTC signaling gateways—a single cluster node must frequently maintain hundreds of thousands to over 1 million concurrent, long-lived WebSocket connections (the C1000K problem). While Go's idiomatic concurrency model (two goroutines per connection: one reader, one writer) is elegant, it encounters severe physical limits at this scale. This article details how we achieve 1,000,000 active connections on a single 16GB Linux server using edge-triggered epoll, memory pool recycling, and zero-allocation frame parsing.

1. The C1000K Problem: The Goroutine Memory Wall

Under standard Go implementations (such as gorilla/websocket or nhooyr/websocket on top of net/http), each active TCP socket requires:

• Goroutine Stack Overhead: A reader goroutine and a writer goroutine (~2KB to 4KB stack each). For 1 million connections, that alone totals 4GB to 8GB of RAM just for dormant goroutine stack frames.
• Buffered I/O Allocation: bufio.Reader and bufio.Writer buffers typically allocate 4KB each per socket (adding another 8GB of heap memory).
• GC Scanning Burden: The Go Garbage Collector must scan millions of live pointers every cycle, causing mark-phase CPU spikes and multi-second latency freezes.

2. The Solution: Linux Epoll & The Reactor Pattern

Rather than assigning goroutines to sockets, we decouple connection ownership from execution threads using the Linux epoll syscall (Edge-Triggered I/O event notification):

1. All 1,000,000 file descriptors (FDs) are registered with an epoll instance via epoll_ctl(EPOLL_CTL_ADD).
2. A compact pool of Reactor goroutines (matching runtime.NumCPU()) blocks on epoll_wait().
3. When bytes arrive on any socket, epoll wakes up a Reactor worker, which reads the data, dispatches it to a worker pool, and returns immediately without retaining persistent memory.

3. Implementing Zero-Copy WebSocket Upgrades with gobwas/ws

Instead of relying on the heavy net/http.Server, we handle raw TCP connections directly and upgrade HTTP handshake requests using github.com/gobwas/ws, which executes HTTP upgrades with zero heap allocations:

package server

import (
	"net"
	"syscall"

	"github.com/gobwas/ws"
)

// UpgradeConnection switches raw TCP connection to WebSocket with zero heap allocation
func UpgradeConnection(conn net.Conn) (int, error) {
	// Zero-copy HTTP-to-WebSocket upgrade handshake
	_, err := ws.Upgrade(conn)
	if err != nil {
		conn.Close()
		return -1, err
	}

	// Extract underlying Linux file descriptor
	tcpConn, ok := conn.(*net.TCPConn)
	if !ok {
		return -1, nil
	}

	rawConn, err := tcpConn.SyscallConn()
	if err != nil {
		return -1, err
	}

	var fd int
	err = rawConn.Control(func(descriptor uintptr) {
		fd = int(descriptor)
		// Set non-blocking socket flag
		syscall.SetNonblock(fd, true)
	})

	return fd, err
}

4. The Multi-Poller Epoll Reactor Engine in Go

To avoid lock contention across CPU cores, we shard 1,000,000 connections across multiple isolated epoll event loops (typically 4 to 8 Pollers):

package server

import (
	"log"
	"sync"
	"syscall"
)

type Poller struct {
	epfd    int
	mu      sync.RWMutex
	sockets map[int]*ClientSession
}

func NewPoller() (*Poller, error) {
	fd, err := syscall.EpollCreate1(0)
	if err != nil {
		return nil, err
	}
	return &Poller{
		epfd:    fd,
		sockets: make(map[int]*ClientSession),
	}, nil
}

func (p *Poller) Add(fd int, session *ClientSession) error {
	p.mu.Lock()
	p.sockets[fd] = session
	p.mu.Unlock()

	// EPOLLET: Edge-triggered mode, EPOLLIN: Readable events, EPOLLRDHUP: Client disconnect
	event := syscall.EpollEvent{
		Events: syscall.EPOLLIN | syscall.EPOLLET | syscall.EPOLLRDHUP,
		Fd:     int32(fd),
	}
	return syscall.EpollCtl(p.epfd, syscall.EPOLL_CTL_ADD, fd, &event)
}

func (p *Poller) StartEventLoop(workerPool chan<- *ReadTask) {
	events := make([]syscall.EpollEvent, 1024)

	for {
		// Wait for active network events across thousands of sockets
		n, err := syscall.EpollWait(p.epfd, events, -1)
		if err != nil {
			if err == syscall.EINTR {
				continue
			}
			log.Printf("Poller epoll_wait error: %v", err)
			return
		}

		for i := 0; i < n; i++ {
			fd := int(events[i].Fd)

			// Handle disconnect
			if events[i].Events&(syscall.EPOLLRDHUP|syscall.EPOLLHUP|syscall.EPOLLERR) != 0 {
				p.cleanup(fd)
				continue
			}

			// Handle inbound readable bytes
			if events[i].Events&syscall.EPOLLIN != 0 {
				p.mu.RLock()
				session, exists := p.sockets[fd]
				p.mu.RUnlock()

				if exists {
					workerPool <- &ReadTask{FD: fd, Session: session}
				}
			}
		}
	}
}

func (p *Poller) cleanup(fd int) {
	syscall.EpollCtl(p.epfd, syscall.EPOLL_CTL_DEL, fd, nil)
	syscall.Close(fd)
	p.mu.Lock()
	delete(p.sockets, fd)
	p.mu.Unlock()
}

5. Linux Kernel Parameter Tuning for C1000K

Operating systems default to conservative limits that prevent opening more than 1,024 file descriptors per process. To sustain 1,000,000 sockets, configure the Linux kernel via /etc/sysctl.conf:

# Maximum open files across entire system
fs.file-max = 2097152

# Allocate minimal TCP buffer space per dormant socket (prevents kernel RAM exhaustion)
# min / default / max in bytes
net.ipv4.tcp_rmem = 1024 4096 16777216
net.ipv4.tcp_wmem = 1024 4096 16777216

# Increase TCP SYN backlog and listen queue depth
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Fast recycling of TIME_WAIT sockets in high-churn environments
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535

# Increase total TCP memory pages (min / pressure / max in 4KB pages)
net.ipv4.tcp_mem = 786432 1048576 1572864

6. Production Benchmark: Standard vs Epoll Architecture

We evaluated both architectures under 1,000,000 simulated connections using an automated Go client generator cluster across an isolated 10GbE network:

7. Key Architectural Takeaways

• Goroutines Are Cheap, Not Free: When operating below 10,000 concurrent sockets, idiomatic Go goroutines are ideal. Once you surpass 100,000+ concurrent connections, you must transition to non-blocking epoll event loops.
• Zero-Copy Frame Parsers: Use libraries like gobwas/ws or cloudwego/netpoll to decode WebSocket header opcodes without allocating slices on the Go heap.
• Tune TCP Read/Write Buffers: Setting conservative tcp_rmem and tcp_wmem values ensures that the Linux kernel does not exhaust socket memory before your application even reads the payload.