| Author |
Message
|
| vivekkooks |
 Posted: Sun Apr 03, 2005 10:49 pm Post subject: Multi-threaded Queue application |
 |
|
Voyager
Joined: 11 Jun 2003
Posts: 91
|
Hello All,
I am using WebSphere MQ-5.3 on Linux. I am using C++ MQ APIs in my applications.
I am using single phase commit mechanism when retrieving the messages from the queue.
The pseudo-code for the operations is as follows (single-threaded):
1. connect to Qmgr and open the queue
loop{
1. Get Message Options:
Set Options: MQGMO_WAIT + MQGMO_FAIL_IF_QUIESCING
Set Sync Point Participation to TRUE
2. Get Message
3. Parse the message and insert the records in the database table
4. Queue Manager.commit() to erase the message from the queue.
}
I want to make this application multi-threaded so that to increase the performance.
Not my doubts are as follows:
1. Can I "Get" multiple message if I am using single phase commit? (Set Sync Point Participation to TRUE)
2. I "commit()" the message once it is processed successfully, but can I commit a specific message?
Here is the pseudo code for multi-threaded application:
1. connect to Qmgr and open the queue
loop{
1. Get Message Options:
Set Options: MQGMO_WAIT + MQGMO_FAIL_IF_QUIESCING
Set Sync Point Participation to TRUE
2. Get Message (Get multiple messages)
---> use the thread pool to process the messages
3. Threaded--->Parse the message and insert the records in the database table
4. I want to have a message based commit. How should I have it? ---->Queue Manager.commit() to erase the message from the queue.
}
Your help is highly appreciated. |
|
| Back to top |
|
 |
| JLRowe |
| Posted: Mon Apr 04, 2005 1:46 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
| The MQ connection is not threadsafe, you need 1 connection per thread. |
|
| Back to top |
|
|
| vivekkooks |
| Posted: Mon Apr 04, 2005 2:10 am Post subject: |
|
|
Voyager
Joined: 11 Jun 2003
Posts: 91
|
It is correct that I need separate connection for each thread.
But, how should I use multi-threading with queue having single phase commit.?
An alternative I have is as follows :
1. Main Thread: Get messages
2. Main Thread: Assign one message for each thread
3. Thread: No need to connect to the queue manager. Just parse the message (it is in the form of string) and insert the data in the database table. Notify the main thread about success/failure
Now Main Thread has to perform QManager.commit() so that the messages are removed from the queue. But the problem is if one threads fails to process the message successfully, how to backout() that perticular message?
[/b] |
|
| Back to top |
|
|
| malammik |
| Posted: Mon Apr 04, 2005 4:27 am Post subject: |
|
|
Partisan
Joined: 27 Jan 2005
Posts: 397
Location: Philadelphia, PA
|
The problem is in your design. You have one process that reads messages ONE a time and commits them ONE a time, yet you are trying to process them in a multithreaded fashion commit pending successful completion of the threads. There is no good solution here because design is bad. Either do not use multithreading or have each thread make its own connection.
_________________
Mikhail Malamud
http://www.netflexity.com
http://groups.google.com/group/qflex |
|
| Back to top |
|
|
| vivekkooks |
| Posted: Mon Apr 04, 2005 4:59 am Post subject: |
|
|
Voyager
Joined: 11 Jun 2003
Posts: 91
|
The pseudo-code is bad and not the design.
The design offers the flexibity to have a separate connection to the queue.
In the above case(separate connection for each thread), does MQ allows synchronization for message subscription?
e.g. Each thread will connect to the queue manager and start subscribing the messages from the SAME queue. Now, does MQ synchronize messages subscription on its own so that every thread gets the next message and no two threads get the same copy of the message(I am using single phase commit) ? or I have to write code for it? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Apr 04, 2005 5:10 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| vivekkooks wrote: |
| e.g. Each thread will connect to the queue manager and start subscribing the messages from the SAME queue. Now, does MQ synchronize messages subscription on its own so that every thread gets the next message and no two threads get the same copy of the message(I am using single phase commit) ? or I have to write code for it? |
MQ will handle it. The same message will never be delivered to more then one thread. Unless the original thread backs it out back to the queue, at which point its up for grabs again - to ONE thread only.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| vivekkooks |
| Posted: Thu Apr 07, 2005 9:07 am Post subject: |
|
|
Voyager
Joined: 11 Jun 2003
Posts: 91
|
Hello,
I am getting an error when multiple threads are trying to connect to the Queue
ImqQueue::open ended with reason code 2219
So, this is MQRC_CALL_IN_PROGRESS means only one thread will be able to open a queue? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Apr 07, 2005 9:09 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| vivekkooks wrote: |
| So, this is MQRC_CALL_IN_PROGRESS means only one thread will be able to open a queue? |
Depends on the open options.
And the shareability of the QCONN.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 07, 2005 1:38 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Read the manuals for your programming language.
Check out whether the connection is threadsafe in your language.
If not you need to change your code to ensure that each thread is using it's own connection.
Enjoy  |
|
| Back to top |
|
|
| vivekkooks |
| Posted: Fri Apr 08, 2005 2:19 am Post subject: |
|
|
Voyager
Joined: 11 Jun 2003
Posts: 91
|
| jefflowrey wrote: |
Depends on the open options.
And the shareability of the QCONN. |
The open options are (MQOO_INPUT_SHARED + MQOO_FAIL_IF_QUIESCING)
Reason code 2219 also describes that the MQI call entered before previous call complete.
The queue is defined as (define qlocal) SHARE.
I create 5 threads. Here is the thread_function pseudo code
1. connect to Queue Manager (# is successful for all the threads)
2. queue.setConnectionReference( mgr ) (#successful for all threads)
3. open the queue with the above options (Only one thread is able to open the queue, rest are erroring with reason code:2219)
4. close all the connections
So, essentially each thread is using its own connection.
What's missing in this? |
|
| Back to top |
|
|
| clindsey |
| Posted: Fri Apr 08, 2005 4:52 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
Take a look at the MQCONNX call in the WMQ Prog. Ref. Specifically look at the MQCNO connect options. (field Options). Your choices are:
MQCNO_HANDLE_SHARE_NONE,
MQCNO_HANDLE_SHARE_BLOCK,
MQCNO_HANDLE_SHARE_NO_BLOCK
These control how the connection may be shared across threads and the detail is discussed more here than with the C++ classes. For ImqQueueManager, set this in the connect options.
Charlie |
|
| Back to top |
|
|
| prithun |
| Posted: Fri Apr 08, 2005 7:18 am Post subject: |
|
|
Apprentice
Joined: 08 Apr 2005
Posts: 33
|
I am also facing similar problems.
I am not getting any error when multiple threads are trying to connect to the Queue, but only the first thread is getting connected to the queue and the program terminates without showing any error.
This is what each thread does.
mgr.setConnectOptions(MQCNO_HANDLE_SHARE_BLOCK);
mgr.connect( );
queue.setConnectionReference( mgr );
queue.setOpenOptions(MQOO_INPUT_SHARED+MQOO_FAIL_IF_QUIESCING);
queue.open( );
print the messages and close the connections.
Can anybody help me in this regard.
Is mq with multithreading documented clearly any where? |
|
| Back to top |
|
|
| clindsey |
| Posted: Fri Apr 08, 2005 7:56 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
What platform are you trying this on. The thread connect options are valid on AIX, HP-UX, OS/400, Solaris, Linux, Windows.
Be sure also that any libs you link with are thread safe, e.g. pthreads, c runtime, etc. as it fits for your platform. On some of the unix platforms, the MQ libs have _r at the end for the thread safe versions (for example, imqb23gl_r and imqs23gl_r on linux)
Charlie |
|
| Back to top |
|
|
| prithun |
| Posted: Fri Apr 08, 2005 9:21 am Post subject: |
|
|
Apprentice
Joined: 08 Apr 2005
Posts: 33
|
| The platform is LINUX and I am using pthreads. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Apr 08, 2005 9:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| prithun wrote: |
| The platform is LINUX and I am using pthreads. |
Read clindsey's post again.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
