For some reason we have a few locked messages on one queue. How can I delete them? The application instance that locked them was stopped/restarted since the messages got stuck on the queue. Is the only way to stop the router and delete the store file?
In that case there are consumers (applications) that locked these messages (the messages are delivered to the consumer caches and locked for other consumers). There is no other reason for a lock. You need to check which application helds the locks.
This is what happened as far as we can tell: there is an application which uses Apache Camel to manage the JMS connection and receive messages. Camel was configured to wait for messages for 1 second. This made Camel to connect, wait 1 second, disconnect if no messages were received and reconnect immediately. The application has been restarted several times since we discovered these stuck messages, the queue itself is defined to be persistent. So why were the messages not re-delivered/unlocked once the messages were delivered, locked, and the connection broken to SwiftMQ? The messages are not re-delivered even after a restart nor unlocked.
If they messages are locked, they stay in some consumer-side cache. This is for fact. May be Camel etc doesn't close a consumer? You can try to delete the connection by SwiftMQ Explorer. This will unlock all messages held by that connection.
If the messages are locked after a restart, either this is an XA-based lock (which you proofed is not) or a connection was made and a consumer locks it without processing (e.g. transacted session, no commit). You have to check your applications. You may restart and make sure no connection is active. The messages should be unlocked.
To give you an update after restarting the routers the messages either disappeared or were unlocked and could be deleted. So the only way to unlock the messages was to restart the routers making sure no instance was running.
You suggested that already, but shouldn't the lock be released when the application that receives (and thus locks) the message disconnects? If it should, it did not happen! I stopped the component that receives these messages several times and the messages remained locked until I stopped the routers themselves. Although I never checked if the messages were unlocked while the component was not connected. Again, this is with version 9.2.5, so this problem may be fixed in later releases.
When messages are transferred to a client's consumer cache (on client side), they will be locked. If the client's consumer closes, they will be unlocked. The unlock takes place at latest when the corresponding connection disconnects. So if you kill a client without properly closing the connection (calling connection.close()) then there is a little change that you get into a so-called half-open TCP socket and the connection is forced to close when the keep alive mechanism jumps in.
This works since years without any problems. I can ensure you there is no issue with locked messages.
I'm sure the TCP connection was closed correctly, yet messages were stuck. I still have the production system with stuck messages. After business hours I will stop the component and make sure no TCP connection was left behind in any state and check the messages. I'll report back with the result.
I stopped the only component that reads from this queue, but the messages remained locked. I made sure even the TIMEWAIT status from gone on the sockets before checking. Sending application can't lock messages, right?
I'm not sure if I was looking at the JMS Swiftleft/Usage or Network Swiftlet/Usage, but the connection was gone. As I said in a previous post this connection tore down and was rebuilt every second, so what I saw in the list is that the list changes every second until I stopped the component. I'm sure SwiftMQ saw the connection terminated, that is how I discovered this problem, the info.log was full of new connection and EOFException entries.
1) Under Router Environment disable the Smart Tree attribute. This requires a reboot of the router. Thereafter you have much more depth in the management tree in SwiftMQ Explorer.
2) Under Queue Manager / Usage you will see the receivers that are connected to the respective queue. These are holding the locks.
3) Under JMS Swiftlet / Usage you the connections, broken down to sessions and receivers.
4) Stop your component and check whether the corresponding connection disappears. If not, force a disconnect via SwiftMQ Explorer. The locks should disappear when all receivers on this queue under Queue Manager / Usage disappear too.
I disabled the Smart Tree attribute, rebooted the routers but under Queue manager / Usage I only see senders and receives when they connect while SwiftMQ Explorer is running. If I start SwiftMQ Explorer after the components have connected I see nothing under either receiver or sender, there are no entries.
Also I see not Active Topic Publishers, there's no such tree branch at all. How could someone determine the IP/port of a topic publisher?
Forget about locked messages, that is not a problem any more. I only commented into this thread because you gave me this Smart Tree parameter in this thread.
So why I don't see queue senders and receivers when I connected after the JMS clients have connected?
I downloaded the latest (9.7.1) JMS client ZIP to make sure it's not a bug in 9.2.5 client, but that did not help. Same problem. If a JMS client connects while Explorer is running it shows up correctly as a sender or receiver.
You see them if they have created a QueueSender/MessageProducer resp. a Receiver/Consumer. If they use an anonymous Sender/Producer (created with null destination), you don't see them. Check under JMSSwiftlet / Usage and drill down the sessions to see the resources that are created from this connection.