enhancement
major
minor
defect
critical
major
minor
minor
#30191
Module shutdown stops no module at all if the class of its InContext job cannot be loaded (database stays open in the container's JVM)
Problem
When an application is stopped in Tomcat after its unpacked jars were deleted (found in E-Home's Debian package, haumacher/ehome#81: prerm removed the context descriptor and dpkg/postrm deleted the war and the unpacked directory before Tomcat had undeployed the app), the module shutdown aborts before it stops a single module:
SEVERE [Catalina-utility-1] StandardContext.listenerStop Exception sending context destroyed event to listener instance of class [com.top_logic.element.ElementStartStop] java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /var/lib/tomcat10/webapps/ehome/WEB-INF/lib/tl-xio.jar at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:100) ... at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1186) at com.top_logic.basic.module.ModuleUtil.stopInThreadContext(ModuleUtil.java:759) at com.top_logic.basic.module.ModuleUtil.shutDown(ModuleUtil.java:736) at com.top_logic.basic.module.ModuleUtil$GlobalContext.shutDown(ModuleUtil.java:102) at com.top_logic.basic.module.ModuleUtil$ModuleContext.close(ModuleUtil.java:69) at com.top_logic.util.AbstractStartStopListener.shutDownModuleSystem(AbstractStartStopListener.java:650) at com.top_logic.util.AbstractStartStopListener.contextDestroyed(AbstractStartStopListener.java:494)
Line 759 is the anonymous new InContext() {...} in stopInThreadContext, a class that is loaded only at shutdown. Its loading fails, the exception escapes the whole shutdown, and no module is stopped: the connection pool and the embedded H2 stay open in Tomcat's JVM (threads MVStore background writer, H2-save, Timer-n of commons-pool's eviction, the scheduler's DefaultScheduler). The next deployment of the same application fails with MVStoreException: The file is locked (OverlappingFileLockException) until Tomcat is restarted.
Deleting a running application's jars is the deployment's fault (E-Home's package now waits for the undeploy), but the consequence is out of proportion: one class that cannot be loaded means that nothing is shut down, including the database.
Expectation
The shutdown makes progress module by module even if something in it fails: the job class of stopInThreadContext (and whatever else the stop path loads lazily) is loaded with the module system at start (a named class referenced at startup, or the class loaded eagerly), and a Throwable escaping the stop of one module is logged and the shutdown goes on with the next one (today BasicRuntimeModule.shutDown catches Exception only).
Reproduction
Tomcat 10.1, an application with an embedded H2 database deployed by a context descriptor in conf/Catalina/localhost/: start it, then delete the descriptor and the unpacked webapps/<app>/WEB-INF/lib within the few seconds before Tomcat's auto-deployer notices the descriptor is gone. The log shows the trace above, /proc/<tomcat>/fd still holds h2.mv.db, and deploying the app again fails with "The file is locked".
Solution
- The stop path of ModuleUtil has no class of its own that is loaded only at shutdown: the job of stopInThreadContext is a lambda calling a method of ModuleUtil instead of an anonymous InContext class. The surrounding ThreadContext.inSystemContext path is the one the startup already uses.
- BasicRuntimeModule.shutDown logs every Throwable escaping the stop of the service (not only Exception); the module is inactive afterwards and the shutdown continues with the next module, so a failing module no longer keeps its dependencies (e.g. the database) running.