MQSeries.net :: View topic

EricL · Posted: Wed Apr 25, 2018 4:23 pm Post subject: stopServer.sh script didn't work

Hi,

We noticed a strange thing on our WAS Prod environment.

1. We have to recycle one JVM by cronjob daily

2. JVM recycle scripts - stop/start script working well all the time, but got issue today, "stopServer.sh serverName" didn't stop fully and properly.

3. "stopServer.sh serverName" started at 2:11 AM, OutOfMemoryError occurred at this time:

[4/25/18 2:11:43:657 EDT] 00000107 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service Uncaught service() exception thrown by servlet action: java.lang.OutOfMemoryError: Java heap space
at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3468)
â€¦.
t com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1731)

4. But log did show JVM was going to stop:

[4/25/18 2:11:43:665 EDT] 000000f3 AlarmThreadMo W UTLS0008W: The return of alarm thread "Deferrable Alarm : 0" (00000003) to the alarm thread pool has been delayed for 18286 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
at com.ibm.ws.util.ThreadPool$Worker.disableHungThreadDetection(ThreadPool.java:1800)
at com.ibm.ws.util.ThreadPool$Worker.threadReturned(ThreadPool.java:1775)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1746).
[4/25/18 2:11:43:668 EDT] 00000003 AlarmThreadMo W UTLS0009W: Alarm Thread "Deferrable Alarm : 0" (00000003) previously reported to be delayed has now completed. It was active for approximately 21065 milliseconds.
[4/25/18 2:11:45:619 EDT] 00000008 ServerCollabo A WSVR0023I: Server XYZ_JVM1 is stopping

5. Strangely, after point 4, the listener from XYZ_JVM1 keep on working, keep on receiving messages from MQ, it didn't stop as usual.

6. The error message followed point 5:
[4/25/18 2:16:05:393 EDT] 0000000a ApplicationMo W DCSV0004W: Did not receive adequate CPU time slice. Last known CPU usage time at 03:15:18:869 EDT. Inactivity duration was 16 seconds.

7. step 5 repeating, e.g: the listener from XYZ_JVM1 keep on receiving and processing message

Just would like to know anybody experienced same thing? how to make sure "stopServer.sh serverName" stop JVM surely?

Thanks