The heap had room, but the service stopped

When the pool runs dry, the service stands still

Goal

Create pool exhaustion yourself and turn it into a fast failure with a timeout, then put Tomcat's thread pool and DB connection pool settings into a separate CATALINA_BASE instance and actually start it.

Why it matters

A service stops at its smallest pool. More important than the pool size is the wait upper bound — without one, a single slow downstream swallows all worker threads. You will confirm by hand the same principle from both the Java code (Semaphore) and the Tomcat configuration (connector, DataSource). The material PoolDemo.java runs on 127.0.0.1:8086 (4 worker threads, 2 connections in the pool, 3-second queries) with a management port 8087 (/stats, a separate thread). Tomcat is installed at $CATALINA_HOME.

Steps

Compile PoolDemo.java in /root/jvm/pool, start it with nohup java -cp /root/jvm/pool PoolDemo > /root/jvm/pool/server.log 2>&1 &, and write the pid to /root/jvm/pool/server.pid. curl -s http://127.0.0.1:8086/health returns ok, and curl -s http://127.0.0.1:8087/stats returns a single line starting with served=.
Send 6 /query requests at once (curl -s -m 30 http://127.0.0.1:8086/query & six times), and after 1 second run jcmd <pid> Thread.print > /root/jvm/pool/dump-pool.txt. It must contain parking to wait for and Semaphore, and /stats must show a max_waiting of 2 or more.
Kill the process in server.pid, restart it with -Ddb.pool.timeout.ms=500, and update server.pid. Send 6 /query requests at once again, but collect the result of one of them with curl -s -m 10 -o /dev/null -w '%{http_code}\n' … >> /root/jvm/pool/after-timeout.txt — a 503 must be mixed in.
Create a CATALINA_BASE: under /root/jvm/pool/tc, the directories conf (copy $CATALINA_HOME/conf), logs, temp, webapps, and work.
In /root/jvm/pool/tc/conf/server.xml, on the 8080 <Connector>, set maxThreads="50", acceptCount="20", and connectionTimeout="5000" (change the existing connectionTimeout value). The XML must be valid.
In /root/jvm/pool/tc/conf/context.xml, inside <Context>, add <Resource name="jdbc/app" auth="Container" type="javax.sql.DataSource" maxTotal="20" maxIdle="5" maxWaitMillis="2000" driverClassName="org.h2.Driver" url="jdbc:h2:mem:app" username="sa" password=""/>.
Start the instance with CATALINA_BASE=/root/jvm/pool/tc JAVA_OPTS=-Xmx256m catalina.sh start. 8080 must return an HTTP status line, and the command line of that JVM must contain -Dcatalina.base=/root/jvm/pool/tc.
Save a dump of the Tomcat JVM to /root/jvm/pool/dump-tomcat.txt, read from the dump the number of threads starting with "http-nio-8080-exec- and the maxThreads of the server.xml 8080 connector, and write them to /root/jvm/pool/sizing.txt as two lines, exec_threads=<n> and max_threads=<n>.

Notes

Tomcat pid: jcmd -l | grep Bootstrap. Dump: jcmd <pid> Thread.print.
Connector attribute documentation: https://tomcat.apache.org/tomcat-10.1-doc/config/http.html , DataSource: https://tomcat.apache.org/tomcat-10.1-doc/jndi-datasource-examples-howto.html
Common mistakes: only increasing the number of worker threads, letting the health check go through the DB, and starting BASE without copying conf (it dies because there are no configuration files).

Start PoolDemo

Compile PoolDemo.java in /root/jvm/pool, start it with nohup, and write the pid to /root/jvm/pool/server.pid. 8086 /health returns ok, and 8087 /stats starts with served=.

After javac -d /root/jvm/pool /opt/lab/fixtures/jvm/PoolDemo.java, run nohup java -cp /root/jvm/pool PoolDemo > server.log 2>&1 & and echo $! > server.pid.

Drain the connection pool

Send 6 /query requests at once and after 1 second run jcmd Thread.print > /root/jvm/pool/dump-pool.txt. It must contain parking to wait for and Semaphore, and max_waiting in /stats must be 2 or more.

for i in 1 2 3 4 5 6; do curl -s -m 30 http://127.0.0.1:8086/query & done; sleep 1; jcmd $(cat server.pid) Thread.print > dump-pool.txt. With 2 connections and 6 requests, 4 wait.

Put an upper bound on the wait

Kill the process in server.pid, restart it with -Ddb.pool.timeout.ms=500, and update server.pid. Send 6 /query requests at once and collect the result codes in /root/jvm/pool/after-timeout.txt (a 503 must be mixed in).

Give each curl -o /dev/null -w '%{http_code}\n' and collect the output with >> after-timeout.txt. A request that could not get a connection within 500ms gets a 503. When waiting for background curls, a bare wait also waits for the java started with nohup, so wait only for the curl pids.

Create CATALINA_BASE

Under /root/jvm/pool/tc, create conf (a copy of $CATALINA_HOME/conf), logs, temp, webapps, and work.

mkdir -p tc/logs tc/temp tc/webapps tc/work && cp -r $CATALINA_HOME/conf tc/conf. Do not touch the installation (HOME).

The connector's three numbers

On the 8080 Connector in /root/jvm/pool/tc/conf/server.xml, set maxThreads="50", acceptCount="20", and connectionTimeout="5000". The XML must be valid.

The 8080 connector in the distribution's server.xml already has connectionTimeout="20000" — change the value and add the other two attributes. The attributes can be anywhere inside the opening tag.

An upper bound on waiting in the connection pool

Inside in /root/jvm/pool/tc/conf/context.xml, add a name="jdbc/app" DataSource Resource with maxTotal="20" maxIdle="5" maxWaitMillis="2000".

Put a single line right before . If maxWaitMillis is -1, it waits forever — that is the cause of the incident.

Start the instance

Start it with CATALINA_BASE=/root/jvm/pool/tc JAVA_OPTS=-Xmx256m catalina.sh start. 8080 must return an HTTP status line, and the JVM command line must contain -Dcatalina.base=/root/jvm/pool/tc.

catalina.sh is in $CATALINA_HOME/bin. Startup takes a few seconds, so repeat curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ (even a 404 means a status line came back).

Count pool threads in the dump

Save a dump of the Tomcat JVM to /root/jvm/pool/dump-tomcat.txt, and write the number of threads starting with "http-nio-8080-exec- and the maxThreads of the server.xml 8080 connector to /root/jvm/pool/sizing.txt as exec_threads= and max_threads=.

The pid comes from jcmd -l | awk '/Bootstrap/{print $1}'. To count: grep -c '^"http-nio-8080-exec-' dump-tomcat.txt. Since minSpareThreads defaults to 10, there are around 10 right after startup. max_threads is the value on the 8080 connector — do not pick up the Executor example in a comment in the distribution's server.xml (maxThreads=150).