Locked messages with an HA Router

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Locked messages with an HA Router

Matthew
We're still getting locked messages (albeit very rarely) and there is no obvious reason for them (we have read all previous forum messages about locked messages, restarted all consumers and checked the XA Resource Manager).  

Restarting the router is not an activity we take lightly but this does remove the locked messages.  However, I am keen to find out exactly why they are locked (without the restart if possible).

The HA Router comes with the JMS XA/ASF Swiftlet - does this mean that by default we will be using XA or have any other impact on locked messages? If so, would we not expect to see any locked messages in the XA Resource Manager?

Only one queue currently appears to suffer from this issue and the consumer is a Apache Camel application using the following spring configuration:

 
    <bean id="swiftJndiTemplate"
          class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">com.swiftmq.jndi.InitialContextFactoryImpl</prop>
                <prop key="java.naming.provider.url">${java.naming.provider.url}</prop>
            </props>
        </property>
    </bean>

    <bean id="JMSConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" lazy-init="true">
        <property name="jndiTemplate" ref="swiftJndiTemplate"/>
        <property name="jndiName" value="${swift.connectionFactory.jndiName:QueueConnectionFactory}"/>
    </bean>

    <bean id="singleSharedConnectionFactory"
          class="com.swiftmq.jms.springsupport.SingleSharedConnectionFactory"
          destroy-method="destroy">
        <property name="targetConnectionFactory" ref="JMSConnectionFactory"/>
        <property name="poolExpiration" value="${swift.connectionFactory.poolExpiration:10000}"/>
    </bean>

    <bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
        <property name="connectionFactory" ref="singleSharedConnectionFactory"/>
        <property name="transacted" value="true"/>
        <property name="transactionManager" ref="txManager"/>
    </bean>

    <bean id="txManager" class="org.springframework.jms.connection.JmsTransactionManager">
        <property name="connectionFactory" ref="singleSharedConnectionFactory"/>
    </bean>

Any help/thoughts would be appreciated.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
The JMS XA/ASF Swiftlet has only the XA feature but only uses it if you use it from your app.

Concerning locked messages:

When a consumer is created (a JMS MessageConsumer etc) and either the first time a receive() is called or a MessageListener is set, a request is sent to the router to fill this consumer's client side cache. Further receives or listener invocations are then served out of this client side cache. Default size is 500 messages or 2 MB. If the cache is near empty, it is filled in the background.

All messages in the client side cache are covered by an internal read transaction which locks the messages until either these are consumed or the transaction is rolled back. The latter unlocks the messages and other consumers can have them.

If MessageConsumer.close() is called, the read transaction which hold the locks is rolled back and thus, unlocks all messages.

Concerning our spring support library:

It adds a layer for pooling sessions, producers and consumers from a single shared connection. If a consumer from a pooled session is created, it either checks its internal consumer pool and returns a free one which has the same queue name or it creates a new pooled consumer. The app uses it just like a normal MessageConsumer. The difference is in the close() method which just adds it to the consumer pool.

There is a clean up thread in the lib which checks all pools (session, producer, consumer) on expiration which is the case after 60 secs non-use. The consumer will then be deleted from the pool and the regular close() method of the consumer is called which then leads to rollback of the read transaction and unlock of all messages of this particular consumer.

I assume that Camel or Spring uses a scheme to create a consumer, read one message and closes it. Something like this. If multiple threads are used it may lead to multiple consumers for a single queue which are checked in and out of the pool without ever reaching the expiration time. However, if multiple threads are being used, a dedicated JMS session must be used for each thread and thus only a single consumer can be active for each dedicated queue.

So actually I'm not sure what's going on there. Under standard JMS spec usage (1 thread, dedicated session), you will always use the same consumer for a queue/topic and the messages must be unlocked 60 secs (expiration time) after the consumer's close method has been called. So either the close() method isn't called or multiple threads use the same session which is forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Hi - thanks for the detailed response - I am attempting to reproduce this issue so that I can prove it one way or the other.  The Camel documentation states that the default configuration is to use a single consumer.

Putting the root cause aside for a moment - is there any way that a feature could be added to the message router to allow us to manually either commit or rollback the locked messages so that they can be either cleared or made available for consumption again?


Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
You can do that already by deleting the respective connection via SwiftMQ Explorer/CLi. Under JMS Swiftlet / Usage.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
In reply to this post by Matthew
If you have transparent reconnect enabled (default), the client should then reconnect.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
In reply to this post by IIT Software
Does it make any difference if I kill the application consuming the messages rather than 'delete entity' in the explorer?

There is only one consumer on the queue and the messages remain on the queue after the consumer is killed...

During testing I can see that multiple (3) sessions are in use but only a single connection for the consumer.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Just to be clear - when I used the word 'consumer' in my last message, I'm referring to the application as opposed to a JMSConsumer..

So there is only one application consuming messages from the queue in question and I have stopped it and restarted but the messages remain locked on the queue.

Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
In reply to this post by Matthew
You can kill it or delete the connection. It's up to you.

1 connection, multiple sessions is typical for SingleSharedConnection.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
After you have stopped it, refresh the Message Viewer in Explorer and check whether the corresponding JMS connection is gone.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
The connection is gone in the Explorer and I also confirmed that the socket was closed and cleaned up properly using netstat output
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
And you have still these locks? Actually there can only be 2 reasons for this: another consumer locks them or they are locked by the Routing Swiftlet (XA).
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
In reply to this post by Matthew
One way to check whether there is a consumer on this queue is via Explorer. You need to disable "smart tree" under Router Environment. Unfortunately this requires a restart. Thereafter you can drill down Queue Manager / Usage and see each consumer on this queue.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Hi there, fortunately we know for sure that there is no other consumer - it is not possible.

With regard to XA, we do not specifically use this in our application.  

In Routing Swiftlet (unlimited)/Listeners there is a single listener defined (plainsocket) with the 'Use XA' option selected... could this have any bearing? There are no entities in the 'Usage' node.

There are no entities in XA Resource Manager Swiftlet/Usage
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
No.

Then it's caused by Camel and Spring in combination with our spring support lib. Can you give me the Camel config for this queue. I need to test that.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Sure - the camel route is nice and simple - it consumes the message from the JMS queue and places a file in a local directory:

    from("jms:queue:MyTestQueue@router1")
             .transacted()
             .to("log:JMSInbound?level=INFO&showBody=false&showHeaders=true")
             .to("file:{{data.path}}/queued");

the SMQP URL looks like this:

smqp://smqhost1:5004/host2=smqhost2;port2=5004;timeout=10000;reconnect=true;retrydelay=1000;maxretries=0

and the ConnectionFactory has the default settings.

We do not see the log line that should be observed immediately upon receipt of the message.  I wondered if the 10,000 second timeout on the SMQP url may be causing an issue but have not been able to replicate the issue in the test environment as yet.
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Typo - that was 10,000ms timeout... not seconds :)
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
Ok, I'll test it. May take a couple of days...
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
In reply to this post by Matthew
I've never used Camel. Can you please send a stripped Spring config with everything I need incl. the Camel stuff so that I can use it out of the box for my test?
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

Matthew
Sure - I've attached a tarball containing a gradle project that just needs the swift libs in the lib folder and the queue name and jndi settings configured in config.properties.  It should consume messages from the queue and create files in the data/queued folder.

swifttest.gz
Reply | Threaded
Open this post in threaded view
|

Re: Locked messages with an HA Router

IIT Software
Administrator
Thanks, I've tested it. It works as expected. It creates 1 session with 1 receiver on the queue and consumes all messages without remaining locks.

Do you have a way to reproduce it?
12