ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » Multithreading using MQCTL with operation MQOP_START

Post new topic  Reply to topic
 Multithreading using MQCTL with operation MQOP_START « View previous topic :: View next topic » 
Author Message
rahulhegde
PostPosted: Mon Jan 06, 2014 9:37 pm    Post subject: Multithreading using MQCTL with operation MQOP_START Reply with quote

Novice

Joined: 06 Jan 2014
Posts: 19

Hello,

I am trying to use asynchronous support using MQCB, MQCTL and MQOP_START to spawn thread for each message received on the local queue. The scenario is that there are multiple messages that will be put at the same time and I would like to see multiple threads spawned for each message consumer.

My application does the following:

1. Creates '2' Queue Manager Connection Handle using MQCNO_HANDLE_SHARE_BLOCK mode
2. Create '2' Object Descriptor to be opened for the same local queue.
3. Maps/Registers each Connection Handle and Object Descriptor to separate consumer callback function.
4. Starts the Message Consumption function for each Connection Handle.

What I see:

As i put bulk messages on to the local queue, there is serialized/sequential processing where

1. Last Registered Consumer Callback is invoked with a thread Id
2. The same thread Id is used, and the next registered callback is invoked.

This is all sequential.

But as per "infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzak.doc%2Ffr41340_.htm"

"Start the consuming of messages for all defined message consumer functions for the specified connection handle. Callbacks run on a thread started by the system, which is different from any of the application threads."

I was expecting each connection handle to spawn separate thread at a time but this was not expected. Could you please help me to know how do I acheive multi-thread spawing?

Thanks.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jan 07, 2014 5:12 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You have a choice about whether you let the MQ api create the thread that runs the call back or not.

It's not clear if you have actually demonstrated that you are seeing synchronous behavior or not.

I would suggest having each callback implement a sleep function to wait, so that you can make sure you have sent two messages that will be processed at the same time. That is, prevent MQ from processing the messages faster than you expect.

You also need to confirm that the connection handles are not using connection sharing. If the SHARECNV on the channel is greater than 1, then you will still be subject to serialization of the MQ conversation across a single network socket.

Paul Clarke will be along shortly to correct me on the above.
Back to top
View user's profile Send private message
rahulhegde
PostPosted: Tue Jan 07, 2014 5:47 am    Post subject: Reply with quote

Novice

Joined: 06 Jan 2014
Posts: 19

Did try the same way where the consumer callback was put to a staggered sleep of 5 secs for total of 50secs.

I am using a local queue. SHARECNV is set to 10, tried using 1 too.

Have tried these options, considering bulk messages available on a single local queue:

1. Single App Thread - Multi QM Handle with QMOP_START Mode
QM uses APP Thread but message consumer callback is processed in a sequential way.

2. Multiple App Thread - Multi QM handle with QMOP_START Mode
QM spawns new Thread for consumer callback processing but still this is in a sequential way.

3. Multiple App Thread - Multi QM handle with QMOP_START_WAIT Mode
QM uses APP Thread for consumer callback processing and acheives parallel processing = total number of app threads.

Thanks.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Tue Jan 07, 2014 8:00 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

@mqjeff - I am a little confused why you think SHARECNV has anything to do with Aysync thread processing. SHARECNV is all about how the connections are shared across TCP sockets and does not really affect the way threads are dispatched in the client.

@rahulhedge - I think we'd need to know a bit more about your application to be able to comment. Firstly, are you using local or client bindings because the threading models and environments are different. In local bindings there is a dedicated thread per HCONN for the entire life of the connection. However, on the client the threads are far more fluid. A single thread can serve multiple connections or indeed you could have no thread currently associated with the consumer depending on workload. As I recall you can adjust this level of thread sharing. In your case it sounds as though you want no sharing ever, I think you can achieve this by saying MQCTLO_THREAD_AFFINITY on your QMOP_START call (although my memory could well be dubious). The other 'issues' with the client are things like read ahead. You may need to suppress this if you want to force equal sharing.

However, bear in mind that forcing sharing can actually inhibit your efficiency. Consider the case where you just have a simple MQGET wait loop in a thread. Now suppose you start 10 threads each with that simple MQGET loop. Many people think that as you put messages to the queue that each of the threads would get an equal share of the messages. However, MQ will try to deliver the message to a 'hot' thread. ie. it will try to use as few threads as possible. This is because it is far more efficient to re-use a thread which is already swapped in than wake up a dormant thread which may be swapped out.

Anyway, hope this helps,

Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
mqjeff
PostPosted: Tue Jan 07, 2014 8:16 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

PaulClarke wrote:
@mqjeff - I am a little confused why you think SHARECNV has anything to do with Aysync thread processing. SHARECNV is all about how the connections are shared across TCP sockets and does not really affect the way threads are dispatched in the client.


My point is that even if the app gets multiple threads consuming messages individually, they still have to come across the channel.

So it's a factor that can affect how many callbacks are dispatched, because the callback can't be dispatched until the message moves across the channel.

That is, with SHARECNV > 1 then messages that might be dispatched across individual sockets could get sequenced down the same socket, and thus have callbacks dispatched in sequence rather than in parallel.

Yes?
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Tue Jan 07, 2014 8:42 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Well, yes and no. Certainly the value of SHARECNV affects the traffic on the socket. However, so does just having multiple sockets. It is very difficult to predict the exact effect of all these shared resources. That is why performance tuning is so hard.

So, it's easier to concentrate on the actual blockers. Does having a value of SHARECNV > 1 inhibit parallel processing in the client. No it doesn't. If, as you say, messages get dispatched down the same socket then they will still be targeted at individual HOBJs and therefore HCONNs and therefore threads. Although as I said before it is possible that the two HCONNs are associated with the same thread. However, given that the messages are coming down a single piece of wire there are already serialised to some extent which is why adding a 5 second sleep was a good suggestion. Adding an artificial sleep should guarantee that the threads will be parallel if they can be.

I'm not sure we know whether they are running as a client yet but if they are then I suspect the 'problem' is either READAHEAD or thread sharing rather than SHARECNV.

Of course I could well be wrong, all of this is only a guess,

Cheers,
Paul
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
rahulhegde
PostPosted: Wed Jan 08, 2014 4:53 am    Post subject: Reply with quote

Novice

Joined: 06 Jan 2014
Posts: 19

First, thanks all for the support.

MQCTLO_THREAD_AFFINITY use has resolved my issue. Now, it processes multiple messages at the same time. This parallel processing count is equal to number of connections to queue manager.

Below quote was bit hard to disect and it really means dedicated thread per consumer callback as i understand.

[quote]5.A callback routine should not, in general, rely on being invoked from the same thread each time. If required, use the MQCTLO_THREAD_AFFINITY when the connection is started.[/quote]

I have few other questions around MQCB:

1. I did see from [url]http://www.mqseries.net/phpBB2/viewtopic.php?t=65879&sid=c609252af7e4722aa922e587e31e93c8[/url] from descripton -

Paul has mentioned that we should not use MQ spawned threads to process business logic on the messages that are received. I feel since MQ uses system threads for executing consumer callback, there should be no harm unless MQ specifies thread limit usage count for its whole system?

2. Since there are multiple connections to the Queue Manager for the same local queue, will there be a commit/rollback impact across different threads or this would still remain specific to Queue Manager?

3. Is there any other dis-advantage of using MQCTLO_THREAD_AFFINITY, this is for case where we create more than 1 queue manager connection and each queue manager is registered with a consumer callback? Is this a good practice.

4. As an alternate approach to acheive parallel processing of messages from the same local queue, is using MQOP_START_WAIT, a advisable approach as this would result in application creating threads rather MQ spawing system threads?

Note: Parallel processing definition -
@time t1 = 2 Messages are available on the local queue
@time t2 = These 2 messages are fetched from the local queue and its processing is started in parallel.

Thanks a lot.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Wed Jan 08, 2014 5:31 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Great, glad that MQCTLO_THREAD_AFFINITY did the trick. If there is anyone from Hursley listening perhaps it would be worth updating the manuals.

Quote:
Paul has mentioned that we should not use MQ spawned threads to process business logic on the messages that are received. I feel since MQ uses system threads for executing consumer callback


to be honest I don't remember saying this. It is perfectly reasonable to process business logic on the consumer thread. What is perhaps not advisable is doing something which will take a very long time. Having said that I would say there are two exceptions:


  1. If you need to guarantee ordering.
    As soon as you return from a consumer thread you may be invoked with the next message. Therefore it is reasonable, in my opinion, to delay return until you are ready for the next message.
  2. If you are the only one using the thread
    When you use MQCTLO_THREAD_AFFINITY you are effectively telling MQ to create a thread for the lifetime of the connection for this, and only this, connection to use. When using such this option I don't think it too unreasonable to do long running processing in the thread,


Now, having said all that I think it is still the application responsibility to ensure that it behaves well on shut down. So, if the administrator tries to quiesce the Queue Manager the application should ensure that at all times it is fairly responsive and will end within a few seconds. There could be many techniques so how this is achieved would depend on the application design.

As for commit/rollback. Each HCONN would have it's own transaction which must be commited or rolledback accordingly. Bear in mind that there is huge scope for reordering the messages here. However, I assume, since you want to process the messages in parallel, that ordering is not a concern.

I can't think of any disadvantages other than slightly higher thread usage when using MQCTLO_THREAD_AFFINITY. If you are only having a few connections then it is of little concern.

Personally I would use MQOP_START rather than MQOP_START_WAIT. MQOP_START_WAIT was mainly provided for environments where you need control over the processing thread and therefore wanted to start it yourself rather than have MQ magic one up for you.


Anyway, just my two cents worth....

Hope that helps,

Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
rahulhegde
PostPosted: Wed Jan 08, 2014 5:55 am    Post subject: Reply with quote

Novice

Joined: 06 Jan 2014
Posts: 19

Thanks Paul.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ API Support » Multithreading using MQCTL with operation MQOP_START
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.