How to delete locked messages

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

How to delete locked messages

Morc
  Hi,

  For some reason we have a few locked messages on one queue. How can I delete them? The application instance that locked them was stopped/restarted since the messages got stuck on the queue. Is the only way to stop the router and delete the store file?

  Thanks,

  Morc
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
In this case it seems these messages are locked by an XA transaction (routing connection). Look here. A commit or rollback unlocks these messages.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
Thanks for the reply, but that is not it. We have two instances of SwiftMQ 9.2.5 in replicated HA. No routing connections and there are no entries under XA usage. Any ideas now? :)
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
In that case there are consumers (applications) that locked these messages (the messages are delivered to the consumer caches and locked for other consumers). There is no other reason for a lock. You need to check which application helds the locks.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
This is what happened as far as we can tell: there is an application which uses Apache Camel to manage the JMS connection and receive messages. Camel was configured to wait for messages for 1 second. This made Camel to connect, wait 1 second, disconnect if no messages were received and reconnect immediately. The application has been restarted several times since we discovered these stuck messages, the queue itself is defined to be persistent. So why were the messages not re-delivered/unlocked once the messages were delivered, locked, and the connection broken to SwiftMQ? The messages are not re-delivered even after a restart nor unlocked.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
If they messages are locked, they stay in some consumer-side cache. This is for fact. May be Camel etc doesn't close a consumer? You can try to delete the connection by SwiftMQ Explorer. This will unlock all messages held by that connection.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
In reply to this post by Morc
If the messages are locked after a restart, either this is an XA-based lock (which you proofed is not) or a connection was made and a consumer locks it without processing (e.g. transacted session, no commit). You have to check your applications. You may restart and make sure no connection is active. The messages should be unlocked.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
To give you an update after restarting the routers the messages either disappeared or were unlocked and could be deleted. So the only way to unlock the messages was to restart the routers making sure no instance was running.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
In that case the messages were locked by some consumer.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
You suggested that already, but shouldn't the lock be released when the application that receives (and thus locks) the message disconnects? If it should, it did not happen! I stopped the component that receives these messages several times and the messages remained locked until I stopped the routers themselves. Although I never checked if the messages were unlocked while the component was not connected. Again, this is with version 9.2.5, so this problem may be fixed in later releases.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
When messages are transferred to a client's consumer cache (on client side), they will be locked. If the client's consumer closes, they will be unlocked. The unlock takes place at latest when the corresponding connection disconnects. So if you kill a client without properly closing the connection (calling connection.close()) then there is a little change that you get into a so-called half-open TCP socket and the connection is forced to close when the keep alive mechanism jumps in.

This works since years without any problems. I can ensure you there is no issue with locked messages.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
I'm sure the TCP connection was closed correctly, yet messages were stuck. I still have the production system with stuck messages. After business hours I will stop the component and make sure no TCP connection was left behind in any state and check the messages. I'll report back with the result.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
I stopped the only component that reads from this queue, but the messages remained locked. I made sure even the TIMEWAIT status from gone on the sockets before checking. Sending application can't lock messages, right?
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
Right. The connection is gone in JMS Swiftlet / Usage?
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
I'm not sure if I was looking at the JMS Swiftleft/Usage or Network Swiftlet/Usage, but the connection was gone. As I said in a previous post this connection tore down and was rebuilt every second, so what I saw in the list is that the list changes every second until I stopped the component. I'm sure SwiftMQ saw the connection terminated, that is how I discovered this problem, the info.log was full of new connection and EOFException entries.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
Do the following:

1) Under Router Environment disable the Smart Tree attribute. This requires a reboot of the router. Thereafter you have much more depth in the management tree in SwiftMQ Explorer.

2) Under Queue Manager / Usage you will see the receivers that are connected to the respective queue. These are holding the locks.

3) Under JMS Swiftlet / Usage you the connections, broken down to sessions and receivers.

4) Stop your component and check whether the corresponding connection disappears. If not, force a disconnect via SwiftMQ Explorer. The locks should disappear when all receivers on this queue under Queue Manager / Usage disappear too.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
  I disabled the Smart Tree attribute, rebooted the routers but under Queue manager / Usage I only see senders and receives when they connect while SwiftMQ Explorer is running. If I start SwiftMQ Explorer after the components have connected I see nothing under either receiver or sender, there are no entries.

  Also I see not Active Topic Publishers, there's no such tree branch at all. How could someone determine the IP/port of a topic publisher?
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
Well, as I said, messages are either being locked by a consumer or by XA. You need to refresh the message viewer to see messages being unlocked.

Topic publishers are not shown under Topic Manager. If (you have to check) then you see it as a sender on queue tpc$<roottopic> under Queue Manager / Usage.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

Morc
  Forget about locked messages, that is not a problem any more. I only commented into this thread because you gave me this Smart Tree parameter in this thread.

  So why I don't see queue senders and receivers when I connected after the JMS clients have connected?

  I downloaded the latest (9.7.1) JMS client ZIP to make sure it's not a bug in 9.2.5 client, but that did not help. Same problem. If a JMS client connects while Explorer is running it shows up correctly as a sender or receiver.
Reply | Threaded
Open this post in threaded view
|

Re: How to delete locked messages

IIT Software
Administrator
You see them if they have created a QueueSender/MessageProducer resp. a Receiver/Consumer. If they use an anonymous Sender/Producer (created with null destination), you don't see them. Check under JMSSwiftlet / Usage and drill down the sessions to see the resources that are created from this connection.
12