TT Lab
Get started
Learn Learning paths Courses

Tomcat & nginx Operations

Configuring the Thread Pool, Heap and GC Logs With Reasons

Continue in TT Lab

Goal

You set up a shared Executor, acceptCount, Keep-Alive, heap and GC logs yourself, and calculate maxThreads with Little's law and leave a document of the evidence.

Why it matters

When a performance issue arises, someone always says "let's raise maxThreads". But if you don't know at what point that value operates, most of the time nothing changes or it gets worse. maxThreads is 'requests being processed at once', maxConnections is 'sockets being held', and acceptCount is 'the OS backlog when even that overflows', and when the backlog overflows too, the client gets not a 503 but Connection refused. The person who knows this difference ends an outage quickly.

Steps

  1. After starting Tomcat, add, in server.xml, inside <Service>, a shared Executor. name="tomcatThreadPool", namePrefix="labhub-exec-", maxThreads="150", minSpareThreads="20".
  2. Give the 8080 connector the attribute executor="tomcatThreadPool" so that it uses this Executor, and restart. After sending a few requests, take a thread dump and save it to /root/tc/threads.txt. labhub-exec- threads must be visible in the dump.
  3. Set acceptCount="50", maxConnections="2000" and connectionTimeout="20000" on the 8080 connector.
  4. Set keepAliveTimeout="5000" and maxKeepAliveRequests="100" on the 8080 connector, and after restarting, send two requests on the same connection with a single curl and save its verbose output (-v) to /root/tc/keepalive.txt. The file must show a trace of reuse: Re-using existing connection or Connection #0 to host.
  5. In /opt/tomcat/bin/setenv.sh, in CATALINA_OPTS, put -Xms512m -Xmx512m and restart. Both values must be visible in the arguments of the running process.
  6. Add GC log options to the same CATALINA_OPTS and restart. Set the log file to /root/tc/gc.log and file rotation to 5 files of 10MB each. After the restart, /root/tc/gc.log must be created and must not be empty.
  7. Create /root/tc/load.sh. It takes two arguments (URL 동시요청수, the URL and the number of concurrent requests), sends the requests concurrently, and prints ok=<성공수> fail=<실패수> on the last line. HTTP 200 counts as success, anything else (including 404) counts as failure.
  8. Write /root/tc/sizing.md. Apply Little's law with the scenario below to calculate the maxThreads value, and leave the calculation process and result.
    • Target throughput: 300 TPS
    • Average response time: 200ms
    • Peak headroom: 30% The document must have exactly one line of the form maxThreads=<계산값>, and all three words TPS, ms and 여유율 (headroom) must appear.

Notes

Define a shared Executor

After starting Tomcat, add, in server.xml, inside <Service>, a shared Executor. name="tomcatThreadPool", namePrefix="labhub-exec-", maxThreads="150", minSpareThreads="20".

The Executor is a child of Service and must be declared before the Connector. If you give namePrefix a meaningful value, you can tell them apart right away later in a thread dump.

Connect the connector to the Executor

Give the 8080 connector the attribute executor="tomcatThreadPool" so that it uses this Executor, and restart. After sending a few requests, take a thread dump and save it to /root/tc/threads.txt. labhub-exec- threads must be visible in the dump.

If you give a connector the executor attribute, that connector's maxThreads is ignored. You can confirm whether it was actually applied by looking at the thread names in the thread dump. Use jcmd or jstack.

Connection acceptance parameters

Set acceptCount="50", maxConnections="2000" and connectionTimeout="20000" on the 8080 connector.

acceptCount is the OS backlog after maxConnections is used up. When this overflows too, the client gets not a 503 but a connection refusal.

Keep-Alive settings and verification

Set keepAliveTimeout="5000" and maxKeepAliveRequests="100" on the 8080 connector, and after restarting, send two requests on the same connection with a single curl and save its verbose output (-v) to /root/tc/keepalive.txt. The file must show a trace of reuse: Re-using existing connection or Connection #0 to host.

With curl -v you can see the Connection value in the response headers and whether the connection was reused. If you send the two requests in one curl, reuse can be observed.

Fix the heap

In /opt/tomcat/bin/setenv.sh, in CATALINA_OPTS, put -Xms512m -Xmx512m and restart. Both values must be visible in the arguments of the running process.

For server applications, the convention is to set the minimum heap and the maximum heap equal. It is to eliminate the GC that occurs as the heap grows and shrinks.

Enable the GC log

Add GC log options to the same CATALINA_OPTS and restart. Set the log file to /root/tc/gc.log and file rotation to 5 files of 10MB each. After the restart, /root/tc/gc.log must be created and must not be empty.

Since JDK 9 you use -Xlog unified logging. If you don't give the file rotation options (count and size) together, the GC log fills the disk.

Concurrent load script

Create /root/tc/load.sh. It takes two arguments (URL 동시요청수, the URL and the number of concurrent requests), sends the requests concurrently, and prints ok=<성공수> fail=<실패수> on the last line. HTTP 200 counts as success, anything else (including 404) counts as failure.

Starting several curls in the background and gathering them with wait is enough. Judge the success of each request by the HTTP status code.

Write the thread pool sizing evidence

Write /root/tc/sizing.md. Apply Little's law with the scenario below to calculate the maxThreads value, and leave the calculation process and result.

Little's law: concurrent work = target TPS × average response time (in seconds). Multiply by the headroom and round up. You must leave the calculation process in the document so that when someone changes this value six months from now, it serves as the basis.