SwiftMQ 9.4.2 Router does not terminate

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

SwiftMQ 9.4.2 Router does not terminate

sanjan
Hi,

I am trying to shutdown a swiftmq router (v 9.4.2) by issuing kill command. ie: kill <pid>

However the process is not terminating.

Console log:
Shutdown SwiftMQ 9.4.2 Production ...
... shutdown: FileCache Swiftlet
... shutdown: Monitor Swiftlet
... shutdown: Scheduler Swiftlet
... shutdown: JMS Application Container Swiftlet
... shutdown: Deploy Swiftlet
... shutdown: JMS Swiftlet
+++ waiting for connection shutdown ...
... shutdown: JNDI Swiftlet
... shutdown: Routing Swiftlet (Unlimited Connections)
+++ Waiting for Connection Termination ...


PS output:

  PID %CPU %MEM   RSZ    VSZ CMD
22564  0.5 11.1 452076 1522624 java -server -Xmx1024M -cp ../../jars/swiftmq.jar:../../jars/jndi.jar:../../jars/jms.jar:../../jars/dom4j-full.jar:../../jars/xstream.jar com.swiftmq.Router ../../config/us2appstage002_core/routerconfig.xml


Router config:

]$ grep listener ../../config/us2appstage002_core/routerconfig.xml
      <listener name="plainsocket" connectaddress="us2appstage002" port="4001">
<listener name="plainsocket" port="5001">

cli port:
]$ netstat -an | grep :4001 | wc -l
0

router port:
]$ netstat -an | grep :5001 | wc -l
0


Can somebody explain why?

Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

IIT Software
Administrator
You may shoot a thread dump with kill -3 <pid> to see why it hangs.
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

sanjan
Hi.

so here is the result. from the thread dump.

2014-05-28 13:08:58
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):

"SIGTERM handler" daemon prio=10 tid=0x0000000045839000 nid=0x15ee waiting for monitor entry [0x00000000469e6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045752800 nid=0x702e waiting for monitor entry [0x00000000417dc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045bc4800 nid=0x6f99 waiting for monitor entry [0x0000000045389000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x00000000457a2000 nid=0x2dd waiting for monitor entry [0x00000000474f1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x000000004568e000 nid=0x274 waiting for monitor entry [0x00000000473f0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x000000004568c000 nid=0x25c waiting for monitor entry [0x00000000472ef000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045689800 nid=0x247 waiting for monitor entry [0x00000000471ee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045687800 nid=0x22f waiting for monitor entry [0x00000000470ed000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045685800 nid=0x219 waiting for monitor entry [0x0000000046fec000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045683000 nid=0x206 waiting for monitor entry [0x0000000046eeb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045681000 nid=0x1f2 waiting for monitor entry [0x0000000046dea000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x000000004567f000 nid=0x1df waiting for monitor entry [0x0000000046ce9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045c6b800 nid=0x1cc waiting for monitor entry [0x0000000046be8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"SIGTERM handler" daemon prio=10 tid=0x0000000045c69800 nid=0x18c waiting for monitor entry [0x0000000046ae7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - waiting to lock <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"Thread-8" prio=10 tid=0x00000000456d1000 nid=0x7e85 waiting on condition [0x00000000467e4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0e33e40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at com.swiftmq.tools.concurrent.Semaphore.waitHere(Unknown Source)
        at com.swiftmq.impl.routing.single.RoutingSwiftletImpl.shutdown(Unknown Source)
        at com.swiftmq.swiftlet.SwiftletManager$SwiftletShutdown.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"Thread-0" prio=10 tid=0x000000004566e800 nid=0x7e2f in Object.wait() [0x0000000044177000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000ef84c1a0> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1186)
        - locked <0x00000000ef84c1a0> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1239)
        at com.swiftmq.swiftlet.SwiftletManager.shutdownSwiftlet(Unknown Source)
        at com.swiftmq.swiftlet.SwiftletManager.stopKernelSwiftlets(Unknown Source)
        at com.swiftmq.swiftlet.SwiftletManager.shutdown(Unknown Source)
        - locked <0x00000000c002d390> (a com.swiftmq.swiftlet.SwiftletManager)
        at com.swiftmq.swiftlet.SwiftletManager$1.run(Unknown Source)

"SIGTERM handler" daemon prio=10 tid=0x000000004566e000 nid=0x7e2e in Object.wait() [0x000000004548a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c0363878> (a com.swiftmq.swiftlet.SwiftletManager$1)
        at java.lang.Thread.join(Thread.java:1186)
        - locked <0x00000000c0363878> (a com.swiftmq.swiftlet.SwiftletManager$1)
        at java.lang.Thread.join(Thread.java:1239)
        at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
        at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
        at java.lang.Shutdown.runHooks(Shutdown.java:79)
        at java.lang.Shutdown.sequence(Shutdown.java:123)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - locked <0x00000000bbc00a28> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=10 tid=0x00002aaab45c6800 nid=0x5828 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"SwiftMQ-timer.tasks-3" prio=10 tid=0x00002aaab4242800 nid=0x584d waiting on condition [0x0000000043c72000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0000148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-timer.tasks-2" prio=10 tid=0x00002aaab4240800 nid=0x584c waiting on condition [0x0000000043b71000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0000148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-timer.tasks-1" prio=10 tid=0x00002aaab423e800 nid=0x584b waiting on condition [0x0000000043a70000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0000148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-timer.dispatcher-1" prio=10 tid=0x00002aaab423c800 nid=0x584a in Object.wait() [0x000000004396f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at com.swiftmq.impl.timer.standard.c.run(Unknown Source)
        - locked <0x00000000c00003c0> (a java.util.LinkedList)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-store.log-1" prio=10 tid=0x00002aaab423a800 nid=0x5849 waiting on condition [0x000000004386e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0000478> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.throttle-3" prio=10 tid=0x00002aaab4238800 nid=0x5848 waiting on condition [0x000000004376d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0074888> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.throttle-2" prio=10 tid=0x00002aaab4236800 nid=0x5847 waiting on condition [0x000000004366c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0074888> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.throttle-1" prio=10 tid=0x00002aaab4234800 nid=0x5846 waiting on condition [0x000000004356b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0074888> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.service-3" prio=10 tid=0x00002aaab4232800 nid=0x5845 waiting on condition [0x000000004346a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c00856d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.service-2" prio=10 tid=0x00002aaab4230800 nid=0x5844 waiting on condition [0x0000000043369000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c00856d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.service-1" prio=10 tid=0x00002aaab422e800 nid=0x5843 waiting on condition [0x0000000043268000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c00856d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.scheduler-3" prio=10 tid=0x00002aaab422c800 nid=0x5842 waiting on condition [0x0000000043167000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0086278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.scheduler-2" prio=10 tid=0x00002aaab422a800 nid=0x5841 waiting on condition [0x0000000043066000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0086278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-routing.scheduler-1" prio=10 tid=0x00002aaab4228800 nid=0x5840 waiting on condition [0x0000000042f65000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0086278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-queue.timeout-1" prio=10 tid=0x00002aaab4226800 nid=0x583f waiting on condition [0x0000000042e64000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0086dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jndi-1" prio=10 tid=0x00002aaab4224800 nid=0x583e waiting on condition [0x0000000042d63000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0087550> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.session-1" prio=10 tid=0x00002aaab4222800 nid=0x583d waiting on condition [0x0000000042c62000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0087ca0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.connection-5" prio=10 tid=0x00002aaab4220800 nid=0x583c waiting on condition [0x0000000042b61000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008c6c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.connection-4" prio=10 tid=0x00002aaab421e800 nid=0x583b waiting on condition [0x0000000042a60000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008c6c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.connection-3" prio=10 tid=0x00002aaab421c800 nid=0x583a waiting on condition [0x000000004295f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008c6c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.connection-2" prio=10 tid=0x00002aaab4218800 nid=0x5839 waiting on condition [0x000000004285e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008c6c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-jms.connection-1" prio=10 tid=0x00002aaab4216800 nid=0x5838 waiting on condition [0x000000004275d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008c6c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-filecache.session-1" prio=10 tid=0x00002aaab4215000 nid=0x5837 waiting on condition [0x000000004265c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008d5b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-filecache.request-1" prio=10 tid=0x00002aaab4211800 nid=0x5836 waiting on condition [0x000000004255b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c008dd50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-amqp.session-1" prio=10 tid=0x00002aaab4210800 nid=0x5835 waiting on condition [0x000000004245a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c00906d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"SwiftMQ-amqp.connection-1" prio=10 tid=0x00002aaab421b000 nid=0x5834 waiting on condition [0x0000000042359000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0090e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1923)
        at com.swiftmq.impl.threadpool.standard.PoolDispatcher.a(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

"Low Memory Detector" daemon prio=10 tid=0x0000000045600800 nid=0x5832 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00000000455fe000 nid=0x5831 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00000000455f8800 nid=0x5830 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00000000455f6800 nid=0x582f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00000000455d4000 nid=0x582e in Object.wait() [0x0000000041ee1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c0091930> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00000000c0091930> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00000000455d2000 nid=0x582d in Object.wait() [0x0000000041de0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c0091ac0> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00000000c0091ac0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00000000455cb800 nid=0x582c runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000045584800 nid=0x582a runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000045586800 nid=0x582b runnable

"VM Periodic Task Thread" prio=10 tid=0x000000004560b000 nid=0x5833 waiting on condition

JNI global references: 1571

Heap
 PSYoungGen      total 325824K, used 121677K [0x00000000eaab0000, 0x00000000ffac0000, 0x0000000100000000)
  eden space 323200K, 36% used [0x00000000eaab0000,0x00000000f1efb590,0x00000000fe650000)
  from space 2624K, 98% used [0x00000000ff620000,0x00000000ff8a8090,0x00000000ff8b0000)
  to   space 2112K, 0% used [0x00000000ff8b0000,0x00000000ff8b0000,0x00000000ffac0000)
 PSOldGen        total 42176K, used 13128K [0x00000000c0000000, 0x00000000c2930000, 0x00000000eaab0000)
  object space 42176K, 31% used [0x00000000c0000000,0x00000000c0cd2308,0x00000000c2930000)
 PSPermGen       total 21248K, used 16755K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
  object space 21248K, 78% used [0x00000000bae00000,0x00000000bbe5ce58,0x00000000bc2c0000)


Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

IIT Software
Administrator
Thank you. Never seen that before. It waits for connections to be closed although no connection weren't active anymore. Seems to be a timing problem. Will be fixed. Using kill -9 <pid> is the only way to end it but should be save now.
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

sanjan
If I use kill -9 what will happen to the persistent queues in the routers.
Will the messages get lost?
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

IIT Software
Administrator
On Linux you may run "sync" from the command line to ensure everything is properly written to disk and then use the kill command. On startup of the router it will run through the recovery and everything should be consistent of no error is reported.
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

sanjan
Hi,

I did some tests with killing a router by SIGKILL (Kill -9) .

And I noticed that the persistent queue's content is retained. when I brought up the router back online.
However, I could notice 2 scenarios in the console log.

1. No logging in when killed. No special logging when brought back online.
======================================================

Starting router at:  Thu May 29 10:25:30 CEST 2014
Booting SwiftMQ 9.4.2 Production ...
... startup: Trace Swiftlet
... startup: Log Swiftlet
... startup: Authentication Swiftlet
... startup: Threadpool Swiftlet
... startup: Timer Swiftlet
... startup: Network Swiftlet
... startup: Store Swiftlet
... startup: Queue Manager Swiftlet
... startup: Topic Manager Swiftlet
... startup: Accounting Swiftlet
... startup: Management Swiftlet
... startup: XA Resource Manager Swiftlet
... startup: Routing Swiftlet (Unlimited Connections)
... startup: JNDI Swiftlet
... startup: JMS Swiftlet
... startup: Deploy Swiftlet
... startup: JMS Application Container Swiftlet
... startup: Scheduler Swiftlet
... startup: Monitor Swiftlet
... startup: FileCache Swiftlet
SwiftMQ 9.4.2 Production is ready.
====KILLED WITH KILL -9======================
Starting router at:  Thu May 29 11:33:27 CEST 2014
Booting SwiftMQ 9.4.2 Production ...
... startup: Trace Swiftlet
... startup: Log Swiftlet
... startup: Authentication Swiftlet
... startup: Threadpool Swiftlet
... startup: Timer Swiftlet
... startup: Network Swiftlet
... startup: Store Swiftlet
... startup: Queue Manager Swiftlet
... startup: Topic Manager Swiftlet
... startup: Accounting Swiftlet
... startup: Management Swiftlet
... startup: XA Resource Manager Swiftlet
... startup: Routing Swiftlet (Unlimited Connections)
... startup: JNDI Swiftlet
... startup: JMS Swiftlet
... startup: Deploy Swiftlet
... startup: JMS Application Container Swiftlet
... startup: Scheduler Swiftlet
... startup: Monitor Swiftlet
... startup: FileCache Swiftlet
SwiftMQ 9.4.2 Production is ready.


02. No logging when killed with kill -9. but there is some recovery logging in the console when bringing router back online.
==========================================================================

Starting router at:  Thu May 29 11:29:15 CEST 2014
Booting SwiftMQ 9.4.2 Production ...
... startup: Trace Swiftlet
... startup: Log Swiftlet
... startup: Authentication Swiftlet
... startup: Threadpool Swiftlet
... startup: Timer Swiftlet
... startup: Network Swiftlet
... startup: Store Swiftlet
... startup: Queue Manager Swiftlet
... startup: Topic Manager Swiftlet
... startup: Accounting Swiftlet
... startup: Management Swiftlet
... startup: XA Resource Manager Swiftlet
... startup: Routing Swiftlet (Unlimited Connections)
... startup: JNDI Swiftlet
... startup: JMS Swiftlet
... startup: Deploy Swiftlet
... startup: JMS Application Container Swiftlet
... startup: Scheduler Swiftlet
... startup: Monitor Swiftlet
... startup: FileCache Swiftlet
SwiftMQ 9.4.2 Production is ready.
====KILLED WITH KILL -9======================
Starting router at:  Thu May 29 11:31:02 CEST 2014
Booting SwiftMQ 9.4.2 Production ...
... startup: Trace Swiftlet
... startup: Log Swiftlet
... startup: Authentication Swiftlet
... startup: Threadpool Swiftlet
... startup: Timer Swiftlet
... startup: Network Swiftlet
... startup: Store Swiftlet
+++ RecoveryManager/restarting, processing transaction log...
+++ RecoveryManager/restart done.
+++ consistency check in progress ...
    Checking Queue Store: DB_INSERTMT
    Checking Queue Store: DB_INSERTMT_ERROR
    Checking Queue Store: DB_INSERTMT_RETRY
    Checking Queue Store: DB_MAILORDER
    Checking Queue Store: DB_MAILORDER_ERROR
    Checking Queue Store: DB_MAILORDER_RETRY
    Checking Queue Store: routerdlq
    Checking Queue Store: swiftmqmgmt
    Checking Queue Store: swiftmqscheduler
    Checking Queue Store: sys$jndi
    Checking Queue Store: sys$prepared
    Checking Queue Store: sys$scheduler
    Checking Queue Store: sys$topic
    Checking Queue Store: sys$xa_heuristic
    Checking Queue Store: testqueue
    Checking Queue Store: unroutable
+++ consistency check done.
... startup: Queue Manager Swiftlet
... startup: Topic Manager Swiftlet
... startup: Accounting Swiftlet
... startup: Management Swiftlet
... startup: XA Resource Manager Swiftlet
... startup: Routing Swiftlet (Unlimited Connections)
... startup: JNDI Swiftlet
... startup: JMS Swiftlet
... startup: Deploy Swiftlet
... startup: JMS Application Container Swiftlet
... startup: Scheduler Swiftlet
... startup: Monitor Swiftlet
... startup: FileCache Swiftlet
SwiftMQ 9.4.2 Production is ready.


May I know what are the scenarios that make this "recovery manager and consistency check" kick in during a router start up. what does it actually do? as messages were retained when this recovery manager didn't kick in.
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

IIT Software
Administrator
The router performs a checkpoint on a regular shutdown. The transaction log will then be empty. If you hard kill a router may left the transaction log with log records. If the router detects that on startup, it starts the recovery manager which performs a consistency check on the store and does a rolling foward of all tx contained in the transaction log.
Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

sanjan
Thanks for the explanation, so does this mean that using kill -9 to kill a router process will not destroy any persistent message that is already stored in the queue?

Also I want to know what are the purposes of these 2 files.

1. transaction.log
2. page.db



Reply | Threaded
Open this post in threaded view
|

Re: SwiftMQ 9.4.2 Router does not terminate

IIT Software
Administrator
This is all documented.