Hybrid GPU Kubernetes with Cilium and Istio Networking

Status: Draft (2026-10-05), partially reflecting actual measurements. The sections "Measured: The Cost of the Istio Sidecar," "mTLS Between Services: Configuration and Actual Behavior," and "Measured: kube-proxy and Cilium When Services Grow to Ten Thousand" contain values directly measured by the author on a VM created in the experimental environment, and the "Reinforced Measurements" (2026-10-09) in those sections were measured on a kind cluster on the same host. The remaining descriptions are verified through official documentation and have not been measured yet. Please read "Verified Versions and Dates" and "Unverified Items" at the end of the article first. Writing Rule: Measured data is the default, and items that could not be measured are marked as "Unmeasured."

Table of Contents

• 1. Why Hybrid
• 2. Cluster Configuration Choices
• 3. Node Joining and Network Paths
• 4. The Role of Cilium
• 5. The Role of Istio
• Measured: The Cost of the Istio Sidecar
• mTLS Between Services: Configuration and Actual Behavior
• Measured: kube-proxy and Cilium When Services Grow to Ten Thousand
• 6. Cautions When Using Them Together
• 7. Crucial Considerations in LLM Serving
• 8. Service Discovery, Failure Domains, and Security
• Common Pitfalls
• Hands-On Practice
• Potential Interview Questions
• Common Misconceptions
• References
• External Evidence
• Verified Versions and Dates

The reasons commonly cited for choosing a hybrid approach are on-premises GPU unit costs, data location, and cloud supply and elasticity. Using them together requires designing cluster boundaries, paths for node joining, service discovery, and failure scopes entirely by yourself. This article summarizes the choices and constraints based on official documentation, while separately addressing long-lived connections and streaming, which are particularly problematic in LLM serving.

What You Can Explain in an Interview After Reading This Article

1. You can compare whether to combine or split clusters based on etcd latency, failure domains, and operational overhead, and explain where EKS Hybrid Nodes fit in.
2. You can explain the role boundaries between Cilium (Container Network Interface (CNI), policies, Gateway API, ClusterMesh) and Istio (mutual TLS (mTLS), L7 routing) and the configurations required when using them together.
3. You can explain with examples how streaming LLM responses break down due to proxy layer idle timeouts, HTTP/2 multiplexing, and shallow health checks.

1. Why Hybrid

There are three reasons. The first is cost. The logic that steady-state loads are handled by owned equipment and peaks are offloaded to the cloud is common, but this article has not verified cost figures. The second is GPU supply, and the third is data location. An AWS blog post (2025-03-19) lists retrieval-augmented generation (RAG) inference close to data, cloud utilization during peaks, and edge inference as use cases for Hybrid Nodes.

2. Cluster Configuration Choices

etcd documentation guides setting the heartbeat interval to 0.5–1.5 times the round-trip time (RTT) between members and the election timeout to at least 10 times the RTT. The defaults are 100ms and 1000ms, respectively. Kubernetes documentation states that if availability is critical, control plane components should be replicated across at least 3 failure domains.

The core conditions in the EKS Hybrid Nodes documentation are as follows:

• AWS manages the control plane while the user manages the on-premises nodes, with billing calculated per vCPU-hour.
• Connectivity must be stable; it is unsuitable for disconnected, degraded, intermittent, and limited (DDIL) environments.
• Running hybrid nodes on top of cloud infrastructure (including other clouds) is not supported.
• Node and pod CIDRs are passed as RemoteNodeNetworkand RemotePodNetwork, up to 15 each, and must be in RFC 1918 or Carrier-Grade NAT (CGNAT) ranges.
• Connectivity of at least 100Mbps and an RTT of 200ms or less is recommended, though not a strict requirement.
• In a mixed cluster, you use VPC CNI for cloud nodes and Cilium for hybrid nodes, and webhooks run on cloud nodes. For pods on both sides to communicate directly, the on-premises pod CIDR must be routable.

For GCP, we have only verified GKE Multi-cluster Services (for sharing services viaServiceExport) and multi-cluster Gateways (where a single configuration-only cluster manages Gateway API resources).

3. Node Joining and Network Paths

flowchart LR
  subgraph ONP["온프레미스"]
    G1["GPU 노드 A"] --- R["라우터·방화벽"]
    G2["GPU 노드 B"] --- R
  end
  R -- "Site-to-Site VPN 또는 Direct Connect" --> TGW["Transit Gateway 또는 가상 사설 게이트웨이"]
  TGW --> VPC["VPC 서브넷"]
  VPC --> CP["EKS 제어 평면"]
  VPC --> CN["클라우드 노드"]

AWS Site-to-Site VPN provides up to 1.25Gbps per tunnel, with a bandwidth-optimized option offering 5Gbps, aggregated via Equal-Cost Multi-Path (ECMP) when using dynamic routing on a Transit Gateway. The MTU is 1446 bytes and does not support Path MTU Discovery (PMTUD). Because overlay encapsulation further reduces pod MTU, fragmentation must be verified through measurement. Direct Connect is a private dedicated line, and MACsec is an optional feature on select dedicated connections. Rather than pulling large files like model weights across the WAN repeatedly (explanatory example: downloading 140GB at 1Gbps theoretically takes about 19 minutes), it is better to keep them in on-premises registries and caches.

The ports Cilium requires between nodes are VXLAN UDP 8472, WireGuard UDP 51871, and health check TCP 4240. In hybrid environments, intermediate firewalls frequently block these ports (see "Common Pitfalls" below).

4. The Role of Cilium

• kube-proxy replacement: Handles service load balancing with eBPF (Extended Berkeley Packet Filter). For internal cluster connections, sockets are assigned to backends at connect() time. Once a connection is long-lived, the backend does not change after being initially determined.
• Network policies: Evaluated based on identity rather than IP addresses.
• Gateway API: Supports Gateway API v1.6.1, requiring kubeProxyReplacement=trueand an L7 proxy (l7Proxy).
• ClusterMesh: Provides direct pod-to-pod communication across clusters (flat IP), global services, and policies. Conditions include identical datapath modes across all clusters, Cilium minor version differences of 1 or less, non-overlapping pod CIDRs, and IP connectivity between nodes. The default limit is up to 255 clusters, which can be increased to 511 using maxConnectedClusters, but this can only be set during installation. Connected clusters share a single trust domain, so the documentation warns to connect only clusters with equivalent security levels. Policies are not automatically propagated, and starting from version 1.19, policies select only local cluster endpoints by default.
• Service discovery: Supports both the service annotation-based global services and the standard MCS-API (ServiceExport, ServiceImport, clusterset.local). With service.cilium.io/affinity: local, traffic is not sent remotely as long as there is a healthy local backend.
• Mutual authentication (mTLS, based on SPIFFE/SPIRE) is in beta per documentation.

5. The Role of Istio

Istio handles service-to-service mTLS identities (SPIFFE-based certificates) and L7 routing (traffic splitting, retries, and timeouts viaVirtualService's weight).

Multi-cluster Ambient has beta support for multi-network configurations, and the undocumented or strict constraints include: not recommended for existing sidecar production, single-network untested, support for only multiple primary clusters, identical waypoint names across all clusters, and manual configuration synchronization. Version 1.31 (2026-08-31) added zone-aware load balancing (zoneAwareLbSetting) and waypoint canary.

Measured: The Cost of the Istio Sidecar

We quantified what the documentation refers to when it states that "sidecars add latency."

Environment. A single KubeVirt VM (8 vCPU, 7.9GiB, guest kernel 6.8.0-139), k3s v1.35.8, Istio 1.31.0 (profile=default, native sidecar), load testing tool fortio 1.75.3. Within the same VM, client and server pods were deployed to alternately measure plain(no injection) and mesh(sidecars on both sides, mTLS). Test conditions were 16 connections, a fixed 500 QPS for 25 seconds (5 runs), and saturation for 20 seconds (3 runs).

Variance across the 5 runs was within ±3% of the average latency, and there were no failed calls.

What these numbers mean. A single request passing through two sidecars (client-side and server-side) added about 2.2ms of latency in this environment. For LLM serving requests that take hundreds of milliseconds to the first token, this cost is minimal, but for responses streaming token by token or internal microservices where latency is around 1ms, it becomes significant. The drop in saturation throughput to 7.5% occurs because the proxies share the same CPU, which means that for sidecars, you must allocate a separate CPU budget in addition to latency .

What these numbers do not show.

• The comparison targets in this table are "no proxy" versus "two proxies and mTLS." The results isolating encryption and proxy hops can be found in the "mTLS Between Services" section below.
• Since this is communication within the same node, actual NIC and inter-node latencies are omitted. Interpret these values strictly as added overhead.
• The table above represents standard HTTP/1.1 with small payloads. gRPC, large payloads, and long-lived streaming are evaluated in a separate test at the end of the "mTLS Between Services" section below.
• Ambient mode (ztunnel) was not tested. Claims that "Ambient is better because sidecars are expensive" are not supported by this measurement.

Lessons learned from the measurement. The initial measurements were off due to fortio's default histogram resolution (1ms). When latency is under 1ms, percentiles are recorded as positions within the bucket (0.5, 0.9, 0.99ms) rather than actual values. I realized this after seeing a mismatch between the average latency of the same run and Little's Law (concurrent connections / throughput). This is a case where you must doubt the tool itself and cross-check the two metricsbefore trusting benchmark results, which will be covered again in Part 5.

mTLS Between Services: Configuration and Actual Behavior

These are the results of actually applying and verifying "how to establish mTLS between solutions" in Istio 1.31.0.

First, two layers must be distinguished. Cilium WireGuard encrypts pod traffic traveling between nodes using node-level keys (by default, this only covers traffic between pods managed by Cilium; node-to-host traffic requires enabling a beta option, and traffic within the same node is not encrypted), but it does not verify workload identities. Istio mTLS allows both sides to authenticate each otherusing certificates issued by service accounts (SPIFFE identity), and access can be allowed or denied based on that identity. If service-to-service authentication is the goal, Istio is the right tool.

The configuration consists of three pieces. All YAMLs below were applied to test VMs to verify their behavior. The API version is security.istio.io/v1 .

1. Label the namespace with istio-injection=enabled , and already running pods must be restarted for the sidecar to be injected.
2. Use PeerAuthentication to determine whether the server side will accept non-mTLS connections. Since the default when no policy exists is PERMISSIVE, the safe approach is to observe first and then elevate to STRICT.
3. Since identities can only be trusted once mTLS is established, on top of that, use AuthorizationPolicy to restrict which service accounts are allowed to call.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: <네임스페이스>}
spec: {mtls: {mode: STRICT}}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: only-good, namespace: <네임스페이스>}
spec:
  selector: {matchLabels: {app: <서버 앱>}}
  action: ALLOW
  rules:
  - from: [{source: {principals: ["cluster.local/ns/<네임스페이스>/sa/good"]}}]

The caller's DestinationRule(tls.mode: ISTIO_MUTUAL) is not mandatory because automatic mTLS is applied between meshes according to the documentation. In the experiment, it was specified to explicitly turn encryption on and off.

Actual Behavior. The client is a temporary curl pod, and the results are shown as the status code and curl exit code.

There are two important points in operations. First, the way STRICT rejects requests is a connection drop, not an HTTP error. Searching for 403 in server logs yields nothing, and on the client side, it only manifests as Connection reset or curl: (56) . When a failure occurs, your first suspicion should be: "Is it STRICT, but the peer is outside the mesh?" Second, authorization rejection is 403 . Because the mTLS connection was successfully established and the identity was simply not on the allowlist, the two symptoms differ, allowing you to distinguish the cause.

Separating the Costs of Encryption and Proxy Hops. This is a comparison with two sidecars left intact while only mTLS was toggled on and off (same VM, average latency is the median of 5 runs at 500 QPS, saturation throughput is the median of 3 runs).

Of the approximately 2.2ms added above, the two proxy hops account for about 2.0ms (approx. 92%) and encryption accounts for about 0.19ms (approx. 8%) . Contrary to the common belief that "mTLS is slow," in this environment most of the cost came from the proxy's HTTP processing rather than encryption. However, since this VM's CPU exposed AES hardware acceleration (aes, vaes), we cannot claim that the same ratio holds for CPUs without acceleration, large bodies, or long streaming sessions (unmeasured).

Additional test separating HTTP parsing from proxy hops. The same server was exposed via a port named http- (L7 parsing) and a port named tcp- (L4 byte relay) to compare sidecars (same VM, 500 QPS for 20 seconds, 5 cross runs, median average latency).

Of the 2.2ms added, HTTP parsing accounted for about 0.87ms (approx. 39%), while simply routing through L4 without parsing left +1.35ms (approx. 61%) incurred solely by traversing two sidecars. Therefore, "two proxy hops account for ~92%" is the sum of non-encryption costs, within which HTTP parsing's share is around 40%. External research ("Dissecting Service Mesh Overheads") reports that protocol parsing constitutes 60-70% of proxy costs, which is smaller in view of this environment's L4 versus L7 difference (we do not make definitive claims as conditions differ).

gRPC, Large Bodies, Long Streaming, and Plaintext Exception Ports.

• gRPC experienced a larger added value than HTTP/1.1 in the same environment (+2.2ms)(+3.4ms). We suspect this is because each call is a single HTTP/2 stream which incurs higher parsing costs, but we did not decompose it.
• Large bodies showed throughput about 3.5 to 4.5 times lower. Since it is within the same node, the plaintext side is closer to memory copy speeds. Forcing LLM weights or large responses through sidecars incurs this cost, so items to exclude must be determined.
• Token streaming intervals were unaffected. Events at 20ms intervals retained the same median even when passing through the sidecar, with p99 increasing by 0.1ms. In other words, sidecar latency does not accumulate per event, but reads as a fixed additional value along the request path (given this interval and event size).
• Exception ports could be opened with portLevelMtls . While maintaining STRICT, setting just one port to PERMISSIVE allowed plaintext clients outside the mesh to receive 200 .
• Limitations: All communications are within the same VM. The impact ratio of large bodies will vary if NICs and CPUs differ.

Precautions when applying to actual service clusters are inferences that have not yet been tested (unmeasured). Plaintext traffic coming from outside the mesh (e.g., Cilium Gateway) hits the connection drop mentioned above when reaching STRICT services, so the approach used is to set only the relevant port to PERMISSIVE using portLevelMtls (behavior verified in the table above). Databases, DaemonSets, and Jobs should be migrated last, and for Jobs, you must separately check for the issue where sidecars do not terminate.

Measured: kube-proxy and Cilium When Services Scale to Ten Thousand

We verified with numbers the claim that "if Cilium replaces kube-proxy, things remain fast even as services scale."

Environment. Two KubeVirt VMs (8 vCPUs each, 7.9GiB, guest kernel 6.8.0-139, k3s v1.36.5). One runs k3s built-in kube-proxy (iptables, nf_tables backend) and flannel, while the other runs Cilium 1.20.1 (KubeProxyReplacement=True, VXLAN). The two VMs ran on different physical hosts. Measurements were taken while scaling services to 0, 100, 1,000, 5,000, and 10,000.

How to Read.

• Rule counts scaled linearly with the number of services on both sides. The difference lies in the time it takes for new services to be reflected in the dataplane. The kube-proxy side took close to 1 second even with zero services (which aligns with documentation stating the minimum sync interval is 1s, though I couldn't verify the setting on this VM), increasing to about 2.3s at 10,000. Cilium remained around 0.3s regardless of the number of services.
• The gap widened further immediately after mass creation. Immediately after creating 10,000 services, the kube-proxy side took 45 seconds for the first new service to connect (appeared only once in two runs), while the Cilium side took 0.25 seconds. This is an important distinction when deploying many services simultaneously.
• New connection latency changed much less than expected. Even when rules reached 110,000 lines, the average latency only increased by about 0.05ms. The common belief that "iptables slows down the dataplane when there are many services" was not strongly verified in this environment (same node, 8 connections). The reflection delay had a greater practical impact.

What These Numbers Don't Tell You.

• Since the hosts are different, the absolute latency values of the two VMs cannot be compared; only the amount of increase within each VM is meaningful.
• This VM experiment did not measure inter-node communication or increasing backend counts per service. Values measured between kind nodes on the same host are provided in the augmentation below, and actual physical node-to-node communication remains unmeasured. Conntrack behavior under high connection counts is covered in a separate test below.
• We did not verify where the measured service was located within the KUBE-SERVICES chain. External measurements (Huawei 2017 presentation, external references below) report that first-packet latency can vary significantly depending on position within the chain, so the result that new connection latency changed very little may vary depending on service positioning.
• New connection latency in this VM experiment was measured only once per level (augmented measurements use 6,000 samples per condition).
• The VM experiment only compared iptables mode. ipvs·nftables mode was measured using kind on the same host in the augmentation below, and the 10,000-service condition was not remeasured with these two modes (unmeasured).

When the conntrack table fills up. On disposable VMs, nf_conntrack_max was lowered, and 4,000 concurrent loopback connections were opened simultaneously (connection timeout of 5 seconds, waiting before the test until existing entries dropped below 500).

• The number of successful connections was almost equal to the "limit minus existing entries"(2,048-140=1,908 vs 1,913, 1,024-59=965 vs 969). For new connections exceeding the limit, without any rejection response, packets were silently dropped, and the client waited until the timeout after retransmissions.
• The kernel log showed nf_conntrack: table full, dropping packet , but there were only 11 to 12 lines. Because the logs are rate-limited, this line count should not be read as the drop count.
• At the default limit (262,144 for this VM), the same load passed without issues. In other words, the risk comes from the product of the limit (proportional to node memory) and the concurrent connection count, and the moment the limit is exceeded, it manifests as some connections failing randomly. This is why the limit is monitored (nf_conntrack_count vs nf_conntrack_max).
• Limitations: This test used loopback connections on the same node at the exact moment all connections were open. The failure rate under short-lived connection frequencies and timeout distributions of real services was not measured (unmeasured).

Supplemental Practical Measurement: Four dataplaes in kind on the same host. The blanks left by the VM experiment above were filled using a kind cluster (Kubernetes v1.36.1, 1 control plane + 1 worker) inside Docker Desktop on a Mac (Apple M1 Max). We measured kube-proxy's iptables, ipvs, and nftables modes, as well as Cilium 1.20.1 (replacing kube-proxy, with VXLAN), in separate new clusters using the same script. Here, "between nodes" refers to between kind node containers on the same host, so physical NIC and line latency are not included. New connection latency is the median of 6,000 measurements per condition for a new TCP connection → HTTP request → one response. Since the client is written in Python, you should look at the differences between conditions rather than the absolute values.

• Datapath overhead was small across all four. On the same node, the median difference between routing via ClusterIP and connecting directly to the Pod IP was within ±0.02ms for all three kube-proxy modes, and +0.005ms for Cilium. When going to a different node, the increase was 0.02–0.04ms for kube-proxy and 0.06ms for Cilium. For Cilium, this value also bundles CNI differences due to the inclusion of VXLAN encapsulation.
• Even when increasing the number of backends from 10 to 1,000, the increase in the new connection median was under 0.06ms(iptables +0.026ms, ipvs +0.023ms, nftables +0.060ms, Cilium no change). While iptables sequentially steps through probabilistic rules within the service chain, at the scale of 1,000, the datapath difference was small, similar to the 10,000-service case in the VM experiment. The backends consist of 1,000 IP:ports created from 5 pods × 200 ports.
• The difference lay in propagation time. The iptables and nftables modes propagated changes made after a 2-second pause in just a few milliseconds, but subsequent changes made right after propagation had to wait about 1.02 seconds. Creating a new Service modifies the Service and EndpointSlice sequentially, taking 1.03–1.04 seconds even with zero services, which is the same phenomenon as the 0.96 seconds seen in the VM experiment. The ipvs mode, configured with a minimum sync interval of 0 seconds in kind defaults, took 0.12 seconds. Cilium took 0.24 seconds, and endpoint changes after the first run were consistent at about 0.47 seconds (0.50 seconds for reverts) regardless of the pause duration. It appears the agent batches changes at regular intervals, but the exact configuration value could not be verified.
• The number of entries scaled linearly for both: iptables had about 4 NAT rules per endpoint, and Cilium had 1 BPF LB entry per endpoint. The entry counts for ipvs and nftables could not be tallied this time due to incorrect counting commands.
• Limitations: This test used 2 nodes on a virtual bridge on the same host, sharing the same Docker instance with other clusters running different workloads. Runs during periods of high VM load were discarded and re-measured. Values between actual physical nodes, as well as conditions scaling both the number of services and endpoints simultaneously, were not measured (unmeasured).

Raw data and scripts: docs/experiments/2026-10-09-dataplane-scale/.

Lessons Learned from Measurement. In the initial measurements, the connection attempt limit (0.2s) caused Cilium's values to lock into two fixed figures: 206ms and 411ms. Any table where the same values repeat should raise suspicions about the tool's resolution. We reduced the attempt limit to 20ms and re-measured.

6. Considerations When Combining Tools

According to Cilium documentation, if you want to use Istio while Cilium replaces kube-proxy, you need socketLB.hostNamespaceOnly=trueand cni.exclusive=false. Keeping kube-proxy is recommended as the configuration with the fewest changes. For L7 HTTP policies, it is advised to use either Cilium or Istio, but not both. Ambient mTLS traffic is tunneled to port 15008, preventing Cilium policies from seeing the source and L7 information. In this case, Istio handles L4 and higher policies within the mesh.

7. Key Considerations for LLM Serving

클라이언트 -> LB/Gateway -> (Envoy 사이드카 or waypoint) -> 모델 서버(vLLM 등)
 각 홉마다: 요청 타임아웃 / 스트림 유휴 타임아웃 / 연결 유휴 타임아웃 / 드레인

• Long Streaming: Envoy's default stream idle timeout is 5 minutes; if it expires before response headers arrive, it results in a 408, and afterwards, a stream reset. This can be triggered if there is prolonged silence during the queuing phase of long prompts. Request VirtualServicein timeoutis disabled by default. Connection pool DestinationRulein idleTimeoutis based on the time with no active requests and defaults to 1 hour, and HTTP/2 PINGs cannot keep the connection alive.
• Retries: Istio defaults to 2 retries and applies only to connect-failure,refused-stream,unavailable,cancelled. You must verify that costly generation requests are not duplicated.
• Load Balancing: The default algorithm is least requests (LEAST_REQUEST). Istio Ambient documentation notes that when failing over to a remote network, HTTP multiplexing and connection pooling can cause requests to concentrate on a single remote endpoint. For LLMs with high variance in generation length, the limitations of load balancing based solely on request counts become more pronounced. The Gateway API Inference Extension addresses this problem with an Endpoint Picker that considers KV cache and request cost; the README states it is Generally Available (GA), and the picker implementation has been moved to the llm-d repository.
• Termination and Rollout: The default sidecar termination drain period is 5 seconds. For multi-minute streams, terminationDrainDurationand pod termination grace periods must be adjusted to match response lengths.
• Health Checks: A signal that the server process is alive is different from a signal that it can process requests. As in the Text-to-Speech (TTS) case under "Common Pitfalls" below, health checks can return 200 even when failure rates are high.
• Latency Distribution: Because prefill, token generation (decode), and queuing are mixed and lengthen tail latency, timeouts should be determined using p95/p99 rather than averages.

8. Service Discovery, Failure Domains, and Security

Failure domains should be evaluated by dividing them into clusters, network paths (VPN), and control planes. In mixed clusters, use Service trafficDistribution(PreferSameZone) to keep traffic within the same zone and place CoreDNS replicas on both sides (recommended by EKS documentation). For security, do not rely solely on perimeter firewalls; instead, establish inter-service mTLS, cluster-specific intermediate CAs under a common root, and automated certificate rotation tools like cert-manager or SPIRE. On-premise nodes authenticate to EKS using temporary credentials issued by SSM Hybrid Activations or IAM Roles Anywhere instead of long-term keys (EKS documentation). Because ClusterMesh and Istio multi-cluster merge trust domains the moment they connect, a compromise on one side can spread to the other.

Common Pitfalls

The author's experimental environment resides within a single private LAN, meaning there is no WAN, cloud, ClusterMesh, or ambient mesh. The items below summarize experiences from that environment as general cases.

• Tunnel Port Blocking: On a WSL2 node running on Windows, the Hyper-V firewall blocked UDP 8472, preventing pods on that node from communicating with pods on other nodes (verified via tcpdump). WireGuard UDP 51871 also failed to establish for the same reason.
• Kubelet Unable to Reach Local Pods: On the same WSL2 node, both HTTP and TCP probes timed out, so we switched to exec probes.
• Gateway Timeouts: Request timeouts were split by path: 120 seconds for general paths, 0s(unlimited) for Server-Sent Events (SSE) streaming responses, and 900 seconds for long-running session creations. When the gateway cut off connections, no errors were left in the server logs, making root cause analysis difficult.
• Minimal Istio Installation: Because Cilium was cni-exclusive=true, we ended up using istio-cni without istio-init, and lowered the Pod Security Admission (PSA) level to privileged for that specific namespace alone. The Cilium documentation mentioned above recommends cni.exclusive=false.
• Shallow Health Checks: When multiple TTS pods were placed behind a single service, about 18% of requests to pods sharing GPUs with other workloads failed due to Out-Of-Memory (OOM) errors, yet the pods remained in a Running state and health checks returned 200. Because a single output required dozens of calls, no completed output was produced for 74 minutes.

Try It Yourself

All examples below are read-only or executed within temporary namespaces. The commands were not executed and verified while writing this post.

# 읽기 전용: Cilium 설정 확인
kubectl -n kube-system get cm cilium-config -o yaml | grep -E "routing-mode|tunnel-protocol|enable-wireguard|kube-proxy-replacement|cni-exclusive|bpf-lb-sock-hostns-only"
kubectl get gateway,httproute -A

# 임시: 노드 사이 지연과 스트리밍 간격 (정리 포함)
kubectl create ns lab-net
kubectl -n lab-net run a --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"NODE_A"}}' -- sleep 3600
kubectl -n lab-net run b --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"NODE_B"}}' -- sleep 3600
kubectl -n lab-net exec a -- ping -c 10 $(kubectl -n lab-net get pod b -o jsonpath='{.status.podIP}')
kubectl delete ns lab-net

For ping results, check the maximum and standard deviation rather than just the average. In the same way, launch an SSE server in a temporary pod to send chunks at 10-second intervals and curl -Nreceive them to see if the service path cuts off the stream. Experiments to change the gateway timeouts.requesttouch the actual service route, so perform them only on a separate test gateway.

Likely Interview Questions

1. Should we use a single cluster or split them? Split them if you need to isolate blast radii and security boundaries; join nodes into a single control plane, like EKS Hybrid Nodes, if unifying management is crucial. Extending the control plane over a WAN requires matching the etcd timeout to the RTT and designing the failure domain for disconnects.
2. What network is required for EKS Hybrid Nodes? A private connection to the VPC via VPN or Direct Connect is needed. The node and pod CIDRs must not overlap with each other, the VPC, or the service CIDRs, and the pod CIDR must be routable for webhooks and direct node-to-node communication.
3. What happens when Cilium and Istio are used together? Cilium handles the CNI, L3/L4 policies, and ingress points, while Istio handles mTLS identities and L7 routing. In kube-proxy replacement mode, socket LB must be restricted to the host namespace and CNI exclusivity should be relaxed. L7 policies should be managed in only one place.
4. What if LLM streaming cuts off mid-way? Check request, stream idle, and connection idle timeouts at every hop, and identify the layers where logs are recorded. Separate timeouts along the streaming path, and match retries, draining, and health checks to the response length.
5. What about multi-cluster failures and security? Keep normal traffic local using region affinity, failing over to remote clusters only when all local backends are unhealthy. Because trust domains are merged, connect only clusters with equivalent security levels. Policies and configurations must be deployed per cluster to maintain consistency via GitOps.

Common Misconceptions

• ClusterMesh is not a feature that merges clusters into one. The API server and policies of each cluster remain independent.
• Hybrid Nodes are not a feature for attaching cloud nodes, but for attaching on-premises nodes to EKS.
• L7 features are not automatically enabled just because you use Ambient mode. A waypoint is required.
• A dedicated line does not automatically mean encryption. Encryption must be configured separately using MACsec or mTLS.

References

• https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-overview.html
• https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-prereqs.html
• https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html
• https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-webhooks.html
• https://aws.amazon.com/blogs/containers/run-genai-inference-across-environments-with-amazon-eks-hybrid-nodes/
• https://docs.aws.amazon.com/vpn/latest/s2svpn/vpn-limits.html
• https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html
• https://docs.cilium.io/en/stable/network/clustermesh/intro/
• https://docs.cilium.io/en/stable/network/clustermesh/setup/
• https://docs.cilium.io/en/stable/network/servicemesh/istio/
• https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/
• https://docs.cilium.io/en/stable/operations/system_requirements/
• https://istio.io/latest/docs/ambient/overview/
• https://github.com/istio/istio.io/blob/master/content/en/docs/ambient/install/multicluster/_index.md
• https://istio.io/latest/news/releases/1.31.x/announcing-1.31/
• https://istio.io/latest/docs/reference/config/networking/destination-rule/
• https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
• https://etcd.io/docs/v3.6/tuning/
• https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services
• https://github.com/kubernetes-sigs/gateway-api-inference-extension

External Evidence

For claims that could not be measured directly or where measurements were insufficient, the following materials were reviewed:

Verified Versions and Dates

As of 2026-10-05: Kubernetes 1.37.1 (2026-09-23), Cilium 1.20.2 (2026-09-16, 1.21.0-pre.3 exists), Istio 1.31.1 (2026-09-21, 1.31 supports Kubernetes 1.32–1.36), Gateway API v1.6.1 (per Cilium documentation). The actual versions used for measurements were Cilium 1.20.1 and Istio 1.31.0.

Unverified Items: Specific figures in the EKS Hybrid Nodes pricing table, latest supported Cilium version (AWS docs only mention 1.17 and 1.18), products for attaching on-premises nodes to GKE, default encryption policies for Direct Connect, exact behavior during ClusterMesh remote cluster disconnections, the impact of WAN latency on Ambient, and the actual output of the commands above.