HollowByte: The 11-Byte OpenSSL Attack That Freezes Your Server

Six weeks ago, OpenSSL shipped five patch releases — 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 — with no CVE attached, no security advisory, and no changelog entry that would tip off a server admin that anything serious had been fixed. The project called the underlying change a "hardening fix." That framing likely left thousands of servers unpatched.

This week, Okta's Red Team published the analysis they'd been sitting on, named the bug HollowByte, and showed exactly what an attacker can do with it: crash or permanently hobble a web server using a malicious payload of just 11 bytes. If your stack runs Apache, NGINX, Node.js, PHP, Python, Ruby, MySQL, or PostgreSQL — and your OpenSSL predates June 9, 2026 — read on.

How 11 Bytes Takes Down a Server

Every TLS handshake starts with a ClientHello message wrapped in a record with a 4-byte header. Three of those bytes declare the length of the message body that will follow. Vulnerable OpenSSL versions read that attacker-supplied length and immediately call malloc() to allocate a receive buffer — before a single byte of the promised payload has arrived. For a ClientHello, the spec allows up to 131 KB, so that is the ceiling the server may pre-allocate based entirely on what an untrusted connection claimed.

An attacker's client sends the 4-byte header (plus 7 bytes to form a minimally valid record), declares a maximum-size body, and then drops the connection without sending the rest. The OpenSSL worker thread blocks, waiting for data that will never arrive. The 131 KB allocation is freed when the connection is eventually cleaned up — but here is where it gets worse.

The Memory Doesn't Come Back

Linux's glibc memory allocator does not immediately return freed heap memory to the kernel. It holds freed chunks in internal bins for potential reuse. HollowByte weaponizes this by varying the declared payload length across waves of connections. Each slightly different allocation size ends up in a different bin. The freed buffers sit in glibc's heap, not available to the rest of the system, because no subsequent allocation matches their size exactly. The server's Resident Set Size climbs — and stays there — even after every attacker connection has closed.

In Okta's testing against NGINX on a 1 GB server, HollowByte locked 547 MB of memory in fragments and eventually triggered an out-of-memory kill. On a 16 GB server, the attack froze 25% of total RAM while keeping the incoming connection count safely below any threshold that would trigger a rate-limiting or alerting rule. Standard defenses — connection limits, fail2ban, DDoS-mitigation appliances counting packets — don't see anything unusual until the server starts misbehaving.

Only a process restart reclaims the memory. Until then, every new legitimate HTTPS connection is competing for a shrinking pool of available RAM.

What's Still Broken

The June 9 patches fix the TLS handshake path. They do not fix the equivalent code in OpenSSL's DTLS implementation (the UDP-based variant used in some VPN, QUIC-adjacent, and real-time protocols). Fixing DTLS would require more invasive changes; OpenSSL has deferred that work. If any component in your stack speaks DTLS, that attack surface remains open.

There is also no CVE, which means vulnerability scanners and patch-management tools that rely on CVE identifiers to flag outdated packages will not alert on this. The only way to know if you're covered is to check your installed OpenSSL version directly.

How to Check and Fix It

Run openssl version on each server. You need to be on one of the June 9 releases or newer:

• OpenSSL 4.0.x: 4.0.1 or later
• OpenSSL 3.6.x: 3.6.3 or later
• OpenSSL 3.5.x: 3.5.7 or later
• OpenSSL 3.4.x: 3.4.6 or later
• OpenSSL 3.0.x: 3.0.21 or later

Note that OpenSSL 1.1.1 and 1.0.2 are end-of-life and have not received this backport. If you are still on those branches, HollowByte is the least of your problems — you are exposed to a long list of unpatched issues and should be planning a migration.

After updating the package, restart every service that links OpenSSL. On a typical Linux server that means Apache or NGINX, your database daemon, and any application runtime that bundles its own TLS stack. A package update that doesn't trigger a service restart leaves the vulnerable binary running in memory.

If you manage upstream packages yourself, the fix landed in OpenSSL pull requests #30792, #30793, and #30794. Downstream Linux distributions — Debian, Ubuntu, RHEL, and their derivatives — have issued their own package updates; use your distribution's security advisories to confirm the exact package version that includes the backport for your OS release.

The Quiet Patch Problem

What makes HollowByte notable isn't just the attack itself — it's that a DoS affecting every major web server and runtime shipped as a silent hardening fix with no identifier to track. That gap between "OpenSSL fixed it" and "your stack is actually running the fix" is exactly where attackers operate. A 1 GB VPS with a shared hosting stack can be knocked offline with traffic a home broadband connection could sustain.

We keep restores in drills specifically because availability failures don't always come with warning labels. But an ounce of patching here is worth considerably more. Check your OpenSSL version today.