Java Debugging: Stopping, Not Stopping, and Recording

Status: Draft (2026-10-26), measurement results reflected. There are two measurement environments. ① Five example programs were created and run directly in a container on a laptop (already pulled eclipse-temurin:21 image, JDK 21.0.12, no network). Nothing was newly downloaded or installed. ② Read-only commands (such as read-only commands(jcmd 1 VM.version ) were executed on a Keycloak pod (Red Hat build OpenJDK 21.0.11) running in the TT Lab cluster. Opening debug ports, starting JFR recording, taking heap dumps, or restarting was not done on production pods. `async-profiler` had to be downloaded, so it is unmeasured. Official documentation was checked by opening the original texts on the same day. Writing rules: Measured data is the basis. Unmeasured items are marked as "Unmeasured", unverified items as "Unverified", and guesses as "Estimated".

Table of Contents

• 1. Three Ways to Look
• 2. Two Paths to Look Inside the JVM
• 3. Stopping to Look with a Debugger: A Tour of jdb
  • Debug Ports Have No Authentication
• 4. How to Remotely Debug a Kubernetes Pod
• 5. Looking Without Stopping: jcmd
  • Measurement: How Much Memory Does the JVM in a Container Know It Has?
  • Measurement: Read-Only Check on Cluster Keycloak Pod
• 6. Case 1: Freezes and Deadlocks
• 7. Case 2: High CPU Usage
• 8. Case 3: Continuous Memory Growth
• 9. JFR: Flight Recorder That Is Always On
  • Environment Variables Are Contained in JFR Files
  • How JFR Is Used in Production
• 10. Order and Precautions in Production Pods
• 11. Conclusion
• References

1. Three Ways to Look

There are broadly three things you can do when a Java service behaves strangely.

• Stopping to look. Pause execution with a debugger and inspect variables step by step. It shows the most things, but while it is paused, the service is also paused. This is a tool for the development environment.
• Looking without stopping. Attach to a running JVM to take a snapshot of threads, heap, and settings at a specific moment (`jcmd`, `jstack`). This is the primary tool used in production. Since some commands briefly pause the JVM, you must know what they do.
• Recording and looking later. Read lightweight recordings that are always on (GC logs, JFR) afterwards. Past occurrences, such as "Why was it slow at 3 AM yesterday?", can only be investigated this way.

This post first illustrates the two paths through which the JVM communicates with external tools (Section 2), followed by debuggers (Sections 3–4), diagnostic commands (Section 5), three case studies (Sections 6–8), JFR (Section 9), and the operational sequence and precautions (Section 10).

2. Two Paths to Look Inside the JVM

The Debugger Path: JPDA. The Java Platform Debugger Architecture (JPDA) has three layers (official documentation).

-agentlib:jdwp=… loads the JDWP agent (native library) into the target JVM, invokes JVM TI, and speaks via JDWP. Expanding the commonly used line yields this (JDK 21 documentation).

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
               │                   │        │         └ 어디서 기다리나. 호스트를 비우면 루프백만, * 면 모든 인터페이스
               │                   │        └ 시작하자마자 멈추고 디버거를 기다릴까(기본 y!)
               │                   └ JVM 이 듣는 쪽(서버)이 될까(기본 n: JVM 이 디버거로 걸어 나간다)
               └ TCP 소켓

suspend has a default of y , meaning that if you forget this point, a service launched with debug options added will not start at all until a debugger attaches. In Kubernetes, readiness probes will continuously fail. The address changed in JDK 9. Previously, specifying just a port listened on all interfaces, but starting with JDK 9, it listens only on loopback. If you want all interfaces, you must attach *: . The JDK 9 release notes state that binding to * is "unsafe and not recommended".

The Diagnostic Command Path: Attach. jcmd·jstack·jmap does not use the network. It attaches via a Unix socket (.java_pid<PID>) in the target JVM's temporary directory, and upon receiving the first request, the JVM spawns a thread named Attach Listener to execute commands. This only works for the same user on the same host (or the same container). Therefore, in Kubernetes, you enter that container using kubectl exec to run it.

3. Stopping to Look with a Debugger: A Tour of jdb

A small program that scores points once every second (Rally.point(boolean leftWins, int rallyNo)) was compiled with -g , launched with JDWP enabled, and attached to using `jdb` within the same container (2026-10-26).

$ jdb -attach 5005 -sourcepath /w
> stop in Rally.point
Breakpoint hit: "thread=main", Rally.point(), line=6 bci=0
6            if (leftWins) left++; else right++;
main[1] where
  [1] Rally.point (Rally.java:6)
  [2] Rally.main (Rally.java:13)
main[1] locals
Method arguments:
leftWins = false
rallyNo = 6
main[1] print Rally.left           ← 정적 필드는 클래스 이름을 붙여야 한다(그냥 left 는 「Name unknown」)
 Rally.left = 4
main[1] set leftWins = !leftWins
 leftWins = !leftWins = true
main[1] clear Rally.point
main[1] cont

Looking at the program's output alongside timestamps reveals what the debugger did.

A debugger not only pauses execution but also allows modifying values. Furthermore, even after the debugger disconnects, the port waits for the next client. These two facts lead to the security issues discussed in the next section.

IDE remote debugging works on the same principle. IntelliJ's "Remote JVM Debug" and VS Code's "request": "attach" settings connect via JDI to the same JDWP port in place of the `jdb` position above.

Debug Ports Have No Authentication

A JDWP connection begins with both sides exchanging JDWP-Handshake 14 bytes. That is all. I verified this directly by connecting to the port, sending 14 bytes, and sending a single command packet (11 bytes) asking to "tell me the VM version".

JDWP-Handshake
…Java Debug Wire Protocol (Reference Implementation) version 21.0
JVM Debug Interface version 21.0
JVM version 21.0.12 (OpenJDK 64-Bit Server VM, mixed mode, sharing)…

It asked for neither passwords nor certificates. This means anyone who can reach the port can do anything a debugger can do (invoking arbitrary methods, changing values). Although I could not find an official documentation statement explicitly declaring that JDWP lacks authentication (unverified), the measurements above, the warning in the JDK 9 release notes, and the explanation on the OpenJDK mailing list that it "allows remote code execution" point in the same direction. In the measurement container, the address=*:5005 socket was listening on all addresses ( /proc/net/tcp6 ) at port 5005.::What about the cost?

When no debugger is attached, the cost of simply loading the JDWP agent was too small to measure (Section 9 table: −0.4% compared to baseline, within noise range). Therefore, it easily lingers in production images because "it is tedious to turn off", making it dangerous.4. How to Remotely Debug a Kubernetes Pod

As a rule, you should not do this in production pods. If a breakpoint pauses service threads, requests pile up during that time, liveness probes fail, and kubelet kills the container. A safe setup for development or staging looks like this.

Deploy a separate copy for debugging.

1. Do not modify the production Deployment; create a pod dedicated to debugging using the same image ( or a separate manifest). Adding JDWP options to the environment variablekubectl debug <파드> --copy-to=<사본> allows the JVM to read them without modifying the image's startup script. JAVA_TOOL_OPTIONS Bind only to loopback.
2. Ensure it listens only on the loopback inside the pod like address=127.0.0.1:5005 , and do not create a Service outside the pod.
3. Open to your PC only via port-forward. kubectl port-forward pod/<사본> 5005:5005 opens by default only to localhost on your PC (official documentation: --address default localhost) and supports TCP only. Since it passes through the API server's authentication and authorization (RBAC's pods/portforward), Kubernetes effectively provides authentication that JDWP itself lacks.
4. Turn off or lengthen probes. Remove liveness probes in the copy, and configure IDE breakpoints to pause "only that thread" rather than "all threads".
5. Delete the copy when finished.

The procedures in this section are unmeasuredin accordance with this post's rule not to open debug ports on production pods. The fact that `port-forward` reaches the loopback address inside the pod was verified only through Kubernetes documentation examples (forwarding to pod ports).

5. Looking Without Stopping: jcmd

jcmd <PID> help lists all commands accepted by that JVM. Frequently used commands are listed along with their impact on the JVM(impact determined by measurements and documentation).

Measurement: How Much Memory Does the JVM in a Container Know It Has?

The JVM reads the container's memory limit (cgroup) to determine the default heap size. The default is 25% of that limit (-XX:MaxRAMPercentage default, official documentation). Memory limits were given to a laptop container, and the maximum heap was read using -XX:+PrintFlagsFinal .

If you run an application in a 2GiB pod without options, the heap will be only 512MiB, leaving the remaining 1.5GiB idle. Conversely, if you set the ratio too high, off-heap memory (thread stacks, metaspace, direct buffers) will run short, causing the container to be OOMKilled. In this case, the pod disappears without any Java OutOfMemoryError or heap dump.

Measurement: Read-Only Check on Cluster Keycloak Pod

I entered the Keycloak pod in the TT Lab cluster using kubectl exec and executed `jcmd` (2026-10-26). This image uses Red Hat's headless JRE, which also contained jcmd (`jstack·jmap ` was absent). Since a single `jcmd` performs both jobs, there is no issue. The JVM was PID 1.

A few things can be read. Since the image has ExitOnOutOfMemoryError enabled, the JVM terminates immediately and Kubernetes restarts it if the heap overflows. Instead, there are no heap dump options to leave the cause. Compared to the maximum heap of 1,434MiB, the actual usage is 157MiB, so there is plenty of room. You can also see that the 2GiB and 70% calculation (1,434MiB) in the table above matches the production pod value exactly.

Keep in mind that an Attach Listener thread is spawned every time a command is executed. Even a "read command" creates a thread inside the target JVM.

6. Case 1: Freezes and Deadlocks

After launching a classic example where two threads acquire two locks in reverse order (rally-A is table → net, rally-B is net → table) and waiting 2 seconds, jstack 1 was captured.

Found one Java-level deadlock:
=============================
"rally-A":
  waiting to lock monitor 0x0000ffff18001d30 (object 0x000000008bb13c80, a java.lang.Object),
  which is held by "rally-B"
"rally-B":
  waiting to lock monitor 0x0000ffff1c000f00 (object 0x000000008bb13c70, a java.lang.Object),
  which is held by "rally-A"
…
"rally-A":
	at Deadlock.grab(Deadlock.java:10)
	- waiting to lock <0x000000008bb13c80> (a java.lang.Object)
	- locked <0x000000008bb13c70> (a java.lang.Object)
…
Found 1 deadlock.

The JVM finds the cycle on its own (same shape as the example in the official troubleshooting guide). The CPU times for the two threads are 1.01ms and 0.60ms respectively, which also shows that a frozen service does not consume CPU. In reality, freezes are more commonly caused by waiting for something slow(e.g., DB connection pool exhaustion, external HTTP calls without timeout) rather than deadlocks. Therefore, dumps should be taken three times at 5–10 second intervals to see if the same thread remains on the same line. Looking at just one instance does not tell you if it was just coincidentally there at that moment.

7. Case 2: High CPU

This is an example where one thread (referee-worker) runs calculations and four threads simply sleep. The order is Which thread is it? → What is that thread doing?.

1. Check the CPU usage per thread inside the container using top -H -p 1 . Actual measurement: TID 24 is at 90.9%, and the name is referee+(truncated).
2. jstack 1 Find that TID in nid . The

"referee-worker" #20 [24] … cpu=3922.52ms elapsed=3.96s … nid=24 runnable
   java.lang.Thread.State: RUNNABLE
	at Hot.scoreRally(Hot.java:7)

in the thread line is the OS thread number. nid=24 The measured JDK 21 outputs the OS thread number in decimaland it also displays [24] . Guides from the JDK 8 and 11 eras told us to convert TID to hex because it was shown in hex like nid=0x18 , so you have to read according to the context.

3. A single stack is just a moment. We use JFR to see "where time is mostly spent." We started recording on the running JVM without restarting .

$ jcmd 1 JFR.start name=hot duration=10s filename=/w/hot.jfr
$ jfr view hot-methods /w/hot.jfr
Method                  Samples Percent
Hot.scoreRally(long)        460 100.00%

For 10 seconds, all 460 execution samples (default setting is about 20ms interval) were in a single method. The jfr view cpu-load of the same recording showed a JVM user CPU average of 9.02%, which was relative to the entire machine (10 cores) , meaning "one core was fully maxed out." Always check what the percentage is a ratio of.

async-profiler(latest v4.6) is a tool that uses JVM's internal API (AsyncGetCallTrace) and Linux perf_events to capture both Java and native stacks without safepoint bias (official README). It is especially great for CPU issues because it generates flame graphs immediately, but it wasn't measured this time because it requires downloading and putting it into the pod, unmeasured.

8. Case 3: Memory keeps increasing

We ran a "cache" that simply puts 64KB frames into a list without removing them, with a maximum heap of 64MB. The options are the combination recommended for production.

java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/w/heap.hprof \
     -XX:+ExitOnOutOfMemoryError "-Xlog:gc*:file=/w/gc.log:uptime,level,tags" -cp classes Leak

-Xlog:gc* is a format created by JDK 9's unified logging (JEP 158) and unified GC logging (JEP 271). The results were as follows.

There is only one way to read this: If usage does not come down even after a Full GC, something alive is holding onto the objects. This is the criterion that distinguishes between simply high allocation and a leak. The class histogram at the 2-second mark even showed the shape of the culprit. [B(byte arrays) ranked first with 9,728 instances totaling 26,419,600 bytes. However, it was recorded in the measurements that the command itself triggered a Full GC (line at 2.3s). This is not a command you should casually run to "take a quick look" in a production service with a large heap.

Finding "who is holding those arrays" in a heap dump (dominator tree) requires analysis tools like Eclipse MAT or VisualVM, so it is unmeasured. Two operational facts were confirmed through actual measurements: the dump file is as large as the heap size(65MB for a 64MB heap), meaning a 1.4GB heap pod requires an equivalent amount of writable volume. Furthermore, the dump contains everything that was in memory—user information, tokens, and passwords as they are—so it must be handled like a secret value.

9. JFR: Always-On Flight Recording

JDK Flight Recorder (JFR) became open source in JDK 11 (JEP 328), and the target overhead for its default configuration is 1% or less. JDK 21's default.jfc describes itself as "safe for continuous production use, typically under 1%," and profile.jfc as "typically around 2%." To verify this, we ran and measured a task mixing hash calculations and short-lived object allocations a fixed number of times (4 million times). Four configurations were alternated 7 times each, pinned to 2 container CPUs.

Since the baseline's 7-run range was already fluctuating by about ±2% at 1,713–1,784ms, the above differences can only be stated to be "within the document's 1% and 2%." Because overhead varies depending on the nature of the task (e.g., heavy allocations or intense lock contention), we recommend measuring it in your own service.

JFR Files Contain Environment Variables

In the actual measurement records above, jfr summary had jdk.InitialEnvironmentVariable instances and jdk.InitialSystemProperty had 15 instances, and opening them with jfr print revealed keys and values as they were . Because Kubernetes pods frequently receive database passwords or API keys as environment variables, JFR files must be sanitized before attaching them to tickets or sending them externally. JDK 21's jfr scrub handles this task.

$ jfr scrub --exclude-events jdk.InitialEnvironmentVariable,jdk.InitialSystemProperty hot.jfr hot-clean.jfr

Actual Measurement: In the cleaned file, both events dropped to 0, while the 460 execution samples remained intact (211,772 → 201,397 bytes).

How to Use JFR in Production

• Always On: Configure a rotating duration in the startup options like -XX:StartFlightRecording=settings=default,maxage=6h,disk=true,filename=/tmp/jfr/ . After an issue occurs, extract the past few hours using jcmd 1 JFR.dump . This is the only way to inspect "what happened in the past."
• On Demand Only: Turn it on without restarting using jcmd 1 JFR.start duration=60s , as shown in sections 6–7.
• Read: jfr summary, jfr view hot-methods·gc·allocation-by-class, or open it with JDK Mission Control.

10. Sequence and Precautions in Production Pods

Figure 2 summarizes what to check first for each symptom. The principle of the sequence is starting with the lowest overhead.

1. What is Already Available: Application logs, GC logs, metrics, and pre-enabled JFR. Do not touch anything else.
2. Lightweight Reads: VM.flags, GC.heap_info, VM.uptime. Filter out incorrect heap size configurations (Section 5) here first.
3. Short Pauses: Thread.print three times with intervals.
4. Record: JFR.start duration=….
5. Heavy Operations: Heap dumps, class histograms. Route traffic away from the pod (remove it from readiness status) or use -XX:+HeapDumpOnOutOfMemoryError to automatically trigger them only at the moment of an OOM.

And Avoid These.

• Do not leave JDWP options in production images and manifests. In particular, address=*:… and Service exposure are remote code execution vectors.
• Treat heap dump and JFR files as files containing secret values. When sending them externally, sanitize JFR using scrub, and as a rule, do not send heap dumps.
• Configure container memory limits alongside MaxRAMPercentage . The default 25% is too little, and 90% starves off-heap memory.
• Verify what a figure like "CPU 9%" is a percentage of. 9% of a 10-core machine is an entire single core.

11. Conclusion

• The debugger (JPDA) consists of three layers: JVM TI, JDWP, and JDI, with -agentlib:jdwp opening the gateway. There is no authentication at this gateway. Actual measurements showed that VM information was obtained with just a 14-byte handshake, and the port reopened even after the debugger disconnected. In Kubernetes, place it behind the API server's authentication using loopback + port-forward.
• The primary operational tool is jcmd. It connects via a Unix socket within the same container, and each command carries different overhead. The class histogram triggered a Full GC in actual measurements.
• First moves for the three cases: pauses mean Thread.print three times, CPU means top -H → nid → JFR, and memory means "Does it not decrease even after a Full GC?"
• JFR default configuration had less than 1% measured overhead and can be enabled without a restart. Since environment variables remain in the file, export it through jfr scrub .

A word from TT. If you halt a rally with a debugger, the ball, the opponent, and the spectators all freeze. During a match, only take snapshots with jcmd, and it is best to keep the video (JFR) always running. Oh, and since the locker room password (environment variables) is captured in the video, make sure to erase it before sharing!

References

• JPDA Architecture https://docs.oracle.com/en/java/javase/21/docs/specs/jpda/architecture.html · JDWP Connection and Invocation Options (JDK 21) https://docs.oracle.com/en/java/javase/21/docs/specs/jpda/conninv.html
• JDK 9 Release Notes (JDWP default binding to localhost, * is "unsafe and not recommended", JDK-8041435) https://www.oracle.com/java/technologies/javase/9-notes.html
• OpenJDK discuss mailing list (2025-01-20, mentioning remote code execution risk of JDWP, not an official document) https://mail.openjdk.org/pipermail/discuss/2025-January/006502.html
• jcmd manual JDK 21 https://docs.oracle.com/en/java/javase/21/docs/specs/man/jcmd.html · JDK 25 https://docs.oracle.com/en/java/javase/25/docs/specs/man/jcmd.html · JEP 444 (Virtual Threads, Thread.dump_to_file) https://openjdk.org/jeps/444
• java command manual JDK 21 (MaxRAMPercentage default 25%, HeapDumpOnOutOfMemoryError, StartFlightRecording) https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html · 8u92 Release Notes (ExitOnOutOfMemoryError) https://www.oracle.com/java/technologies/javase/8u92-relnotes.html
• JEP 158 Unified JVM Logging https://openjdk.org/jeps/158 · JEP 271 Unified GC Logging https://openjdk.org/jeps/271 · JEP 328 Flight Recorder https://openjdk.org/jeps/328
• jfr tool manual https://docs.oracle.com/en/java/javase/21/docs/specs/man/jfr.html · default.jfc (JDK 21u) https://raw.githubusercontent.com/openjdk/jdk21u/master/src/jdk.jfr/share/conf/jfr/default.jfc
• Troubleshooting Guide: Hangs and Loops (Deadlock report example, JDK 17) https://docs.oracle.com/en/java/javase/17/troubleshoot/troubleshoot-process-hangs-and-loops.html
• async-profiler https://github.com/async-profiler/async-profiler
• Keycloak Container Dockerfile (UBI9-micro, java-21-openjdk-headless) https://github.com/keycloak/keycloak/blob/main/quarkus/container/Dockerfile
• Kubernetes port-forward https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/ · https://kubernetes.io/docs/reference/kubectl/generated/kubectl_port-forward/
• All original sources were accessed and verified on 2026-10-26.