What Is Inside When You Open Tomcat

In one line

If you know the roles of just seven directories, most operational situations with Tomcat can be explained, and the places where incidents actually happen are these three: bin/setenv.sh, work/ and logs/catalina.out.

Why still Tomcat?

If you use Spring Boot, an embedded Tomcat is included, so java -jar is all you need. But when you go out to Korean SI sites, there are still far more systems that deploy a WAR to a standalone Tomcat. The reason is not technical preference.

• The operations organization manages by WAS unit. Several business WARs are deployed to one server, and the procedures for start/stop/monitoring are standardized around the WAS.
• Standard frameworks (including the eGovFrame) and deployment procedure documents are written around WARs.
• When an organization that used WebLogic/JEUS moved down to Tomcat to cut costs, the way of operating stays the same.

That is why the task that lands on a newcomer in the first week is "please bring up a Tomcat". And at that moment you meet a server where systemctl start tomcat doesn't work.

A directory map

/opt/tomcat
├── bin/       기동·정지 스크립트. catalina.sh, startup.sh, shutdown.sh, setenv.sh
├── conf/      설정. server.xml, web.xml, context.xml, tomcat-users.xml, logging.properties
├── lib/       톰캣 자신과 모든 웹앱이 공유하는 JAR (JDBC 드라이버를 여기 두는 관행)
├── logs/      catalina.out, localhost_access_log.*.txt, 각종 *.log
├── webapps/   배포 대상. WAR 를 두면 자동으로 같은 이름 디렉터리로 전개된다
├── work/      JSP 를 컴파일한 .java/.class 캐시
└── temp/      java.io.tmpdir

Let's pick out only the points that actually cause trouble in the field.

• bin/setenv.sh — Tomcat does not ship this file by default, but if it exists catalina.sh reads it automatically. Put JVM options (CATALINA_OPTS) here. If you edit catalina.sh directly, it disappears when you upgrade Tomcat. Always put them in setenv.sh.
• work/ — If the screen is still the old one after deployment, nine times out of ten it is the cache here. This is why "delete the work directory" is in the deployment procedure document.
• logs/catalina.out — It is not rotated. If left alone it grows to tens of GB. This file is the number one cause of outages from a full disk.
• lib/ vs WEB-INF/lib/ — If the same library exists in both with different versions, classloader hell opens up. When a NoSuchMethodError appears, look here first.

The hierarchy of server.xml

server.xml is a set of Russian dolls. It is quicker to understand if you read from the outside in.

<Server port="8005" shutdown="SHUTDOWN">      <!-- JVM 하나 -->
  <Service name="Catalina">                   <!-- 커넥터들 + 엔진 하나 -->
    <Connector port="8080" protocol="HTTP/1.1" ... />   <!-- 바깥과 만나는 문 -->
    <Engine name="Catalina" defaultHost="localhost">    <!-- 요청 처리 엔진 -->
      <Host name="localhost" appBase="webapps"          <!-- 가상 호스트 -->
            unpackWARs="true" autoDeploy="true">
        <Valve className="...AccessLogValve" pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

• The Server's port="8005" is not a service port but the port that receives the shutdown command. shutdown.sh connects to it over TCP and sends the string SHUTDOWN. In other words, if this port is open and the string is the default, anyone can bring down the WAS with telnet. It is an item always flagged in security audits, and there are two remedies: make the shutdown string long, or disable it entirely with port="-1". Note in the procedure document that with -1, shutdown.sh no longer works and you have to bring it down with kill.
• The Connector is covered fully in a later module. Remember just one thing here — you can have several connectors, each with a different port, protocol and thread pool.
• The Host's appBase is webapps and autoDeploy="true", so copying a WAR deploys it. In production many organizations set autoDeploy="false" and have the deployment script deploy explicitly, to prevent the accident of a file getting in by mistake.
• In the pattern of the AccessLogValve, many systems still don't put %D (processing time in ms). Without response time in the access log, you have no evidence at all when you get a report that "it's slow". Be sure to add this before go-live.

context.xml — per-webapp configuration

conf/context.xml is global, and conf/Catalina/localhost/<앱이름>.xml is per-app configuration (the file is named after the app). With the latter, you can serve a directory that lies outside webapps (docBase). This is where the operating style of putting the deployment artifact in /app/deploy/order and never touching Tomcat comes from.

The DB connection pool (<Resource>) is usually here too. So the answer to "where is the DB connection information?" is usually one of two: context.xml or the application's properties file.

Start and stop — when there is no systemd

# 기동 (백그라운드, catalina.out 으로 로그)
/opt/tomcat/bin/catalina.sh start

# 포그라운드 (컨테이너에서 쓰는 방식. 로그가 터미널로)
/opt/tomcat/bin/catalina.sh run

# 정지 (8005 포트로 SHUTDOWN 전송, 종료 대기)
/opt/tomcat/bin/catalina.sh stop 30 -force

After stop, the 30 -force means "wait 30 seconds and kill if it hasn't died". It is good to put this in operations scripts. If there are threads in the middle of handling sessions, Tomcat sometimes doesn't die quietly.

The most reliable way to confirm it has started is not the process but the log and the port.

grep 'Server startup in' /opt/tomcat/logs/catalina.out | tail -1
ss -ltn | grep 8080

A "state where the process is up but the port is not open" really exists. It is the case where only the connector died because of a port conflict or an initialization failure. If you look only at the process and report "it's up", you get a phone call again five minutes later.

What you see in the field

What you go through in the first week of touching Tomcat at an SI site is usually one of these three.

• "I added the JVM options but they don't take effect." Either catalina.sh was edited directly, or setenv.sh was created and it wasn't restarted. Whether it took effect must be asked of the process, not the file — look at the actual launch arguments through /proc/<pid>/cmdline.
• "I deployed but the screen is unchanged." It is the JSP compile cache left in work/. This is why "delete the work directory" is in the deployment procedure document, and burning hours on a procedure document missing that one line really happens.
• "The disk is full." catalina.out has no rotation. It must be cut by logrotate, not by Tomcat, and if left alone it grows to tens of GB.

All three are problems that arise not from not knowing Tomcat but from not knowing which file does what. That is why the directory map comes first.