Google engineers have detailed a complex troubleshooting investigation that traced intermittent Google Kubernetes Engine (GKE) API timeouts to an interaction between Private Service Connect, Envoy proxy behavior, and a deep-seated bug in the Linux kernel TCP state machine.
Unraveling Intermittent GKE API Timeouts
In January, a customer utilizing multiple GKE clusters with Private Service Connect (PSC) for private control-plane access experienced frustrating intermittent connectivity failures. When pods attempted to reach the Kubernetes API server, about 800 developers were temporarily blocked from deploying to production. Packet captures uncovered that the control plane was responding to new TCP connection requests with stale ACK packets left over from previously closed connections.
Led by Akash Thakur, a senior technical solutions engineer at Google Cloud, a cross-functional team spanning GKE, Private Service Connect, networking, hostnet, kernel teams, and the Linux upstream community investigated the issue. They discovered that the problem did not stem from a single misconfiguration, but rather from three distinct root causes acting in concert:
- Packet reordering occurring within Private Service Connect.
- Envoy's half-close behavior, which prevented TCP connections from closing cleanly.
- A Linux kernel bug that improperly discarded certain reordered FIN packets, leaving sockets stuck indefinitely in the FIN_WAIT_2 state.
Coordinated Remediation and Upstream Kernel Patch
Resolving the cascading failure required a multi-layered approach carried out over several weeks. Google initially rolled back a Private Service Connect change to stabilize connectivity, subsequently adjusted proxy protocol handling, and introduced new regression tests. To solve the fundamental operating system flaw, an upstream Linux kernel patch authored by Eric Dumazet was integrated.
Although the underlying bug had already been fixed upstream in Linux 6.9.7, the issue affected GKE environments running an earlier Long Term Support (LTS) kernel branch. Google backported the fix to the 6.6 and 6.1 LTS branches, delivering the definitive resolution to GKE customers starting with version 1.33.5.

