| Author |
Message
|
| vsathyan |
 Posted: Wed Apr 15, 2015 9:23 am Post subject: Multiple Readers with MQOO_BROWSE |
 |
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
Hi,
I want to search for a specific message in a queue. If i get the messages using a for loop in a single thread using MQOO_BROWSE, and the number of messages present in the queue is more than 50000 - if it takes 1 second to browse a message and search the data content, searching all the messages takes more than 17 hours.
I would like to create multiple threads using MQOO_BROWSE open option without doing a destructive get.
The challenge i'm facing is - how to prevent threads from picking up a message which have been searched already by another thread.
In other words, lets say two threads open the queue at the same time.
If thread 1 browses the first message, thread two should browse and search the second message. Next, if the second thread wants to browse the third message, how can it validate that it is not already browsed by thread 1.
I do not want to open the queue with exclusive open option as i cannot create more than 1 thread.
Or is there a better way to loop through the messages using browse option? Please advise.
Thank you in advance. |
|
| Back to top |
|
 |
| Andyh |
| Posted: Wed Apr 15, 2015 9:30 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
"if it takes 1 second", I'd be horrified to think that a simple browse would take any thing like a second.
That aside, doesn't MQGMO_MARK_BROWSE_CO_OP satisfy your requirement ? |
|
| Back to top |
|
|
| vsathyan |
| Posted: Wed Apr 15, 2015 9:50 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
| Andyh wrote: |
"if it takes 1 second", I'd be horrified to think that a simple browse would take any thing like a second.
|
The size of each message is greater than 1.5 MB. I have to retrieve the complete message payload data to a buffer and search for specific data to validate. Hence it takes around a second.
Regarding MQGMO_MARK_BROWSE_CO_OP, this makes perfect match for my requirement. Let me try it out and update this thread.
Thanks for your advise.  |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Apr 15, 2015 10:15 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you can put the relevant data into MQ message properties - the built in SQL search might be faster.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Wed Apr 15, 2015 10:48 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
You are searching through 50,000 records, each one over 1.5MB......
Are you sure you are not treating the queue as a database? That is, after all, rule number one of 'no-no's
Having said that even at 1.5MB ...1 second per message is still an enormous time. On my relatively modest development machine I can browse 2MB persistent messages at a rate of about 400 per second. You're not doing this over a client connection are you?
Regards,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| vsathyan |
| Posted: Tue Jul 07, 2015 10:06 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
Yes Paul, we are doing it over a MQ client connection over a network. Hence the delay.
We have a centralized web-service which connects to requested queue managers, opens the queue which has messages and browses for specific strings.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 07, 2015 10:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vsathyan wrote: |
| We have a centralized web-service which connects to requested queue managers, opens the queue which has messages and browses for specific strings. |
That's a horrible design and very anti-pattern; as a previous poster eloquently put it:
| PaulClarke wrote: |
| That is, after all, rule number one of 'no-no's |
There's no way you should design a system that requires you to browse through messages looking for "the right one" according to content. For no other reason (as you've discovered) that it doesn't scale. Hence it's position at the top of the "don't do this" list.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| vsathyan |
| Posted: Tue Jul 07, 2015 10:58 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
@Vitor,
Thanks for the prompt response.
This is not used on a regular basis. The web service will be enabled only when required, configured to connect to the required queue manager and browse the message content for specific data.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 07, 2015 11:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vsathyan wrote: |
| This is not used on a regular basis. The web service will be enabled only when required, configured to connect to the required queue manager and browse the message content for specific data. |
At 17 hours (your estimate) I hope it's not for an urgent requirement..... 
I look forward to your update on what improvements MQGMO_MARK_BROWSE_CO_OP makes to response time.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Tue Jul 07, 2015 11:17 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
It is asking a lot that MQGMO_MARK_BROWSE_CO_OP is going to fix the performance problem. Issuing 50,000 MQGets and sending 75 GB over a client connection is going to take some time
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 07, 2015 11:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PaulClarke wrote: |
| It is asking a lot that MQGMO_MARK_BROWSE_CO_OP is going to fix the performance problem. Issuing 50,000 MQGets and sending 75 GB over a client connection is going to take some time |
Hence my interest, and knowledge of the OP's Plan B.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
