ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » Multiple Readers with MQOO_BROWSE

Post new topic  Reply to topic
 Multiple Readers with MQOO_BROWSE « View previous topic :: View next topic » 
Author Message
vsathyan
PostPosted: Wed Apr 15, 2015 9:23 am    Post subject: Multiple Readers with MQOO_BROWSE Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

Hi,

I want to search for a specific message in a queue. If i get the messages using a for loop in a single thread using MQOO_BROWSE, and the number of messages present in the queue is more than 50000 - if it takes 1 second to browse a message and search the data content, searching all the messages takes more than 17 hours.

I would like to create multiple threads using MQOO_BROWSE open option without doing a destructive get.

The challenge i'm facing is - how to prevent threads from picking up a message which have been searched already by another thread.

In other words, lets say two threads open the queue at the same time.
If thread 1 browses the first message, thread two should browse and search the second message. Next, if the second thread wants to browse the third message, how can it validate that it is not already browsed by thread 1.

I do not want to open the queue with exclusive open option as i cannot create more than 1 thread.

Or is there a better way to loop through the messages using browse option? Please advise.

Thank you in advance.
Back to top
View user's profile Send private message
Andyh
PostPosted: Wed Apr 15, 2015 9:30 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

"if it takes 1 second", I'd be horrified to think that a simple browse would take any thing like a second.

That aside, doesn't MQGMO_MARK_BROWSE_CO_OP satisfy your requirement ?
Back to top
View user's profile Send private message
vsathyan
PostPosted: Wed Apr 15, 2015 9:50 am    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

Andyh wrote:
"if it takes 1 second", I'd be horrified to think that a simple browse would take any thing like a second.


The size of each message is greater than 1.5 MB. I have to retrieve the complete message payload data to a buffer and search for specific data to validate. Hence it takes around a second.

Regarding MQGMO_MARK_BROWSE_CO_OP, this makes perfect match for my requirement. Let me try it out and update this thread.

Thanks for your advise.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Apr 15, 2015 10:15 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

If you can put the relevant data into MQ message properties - the built in SQL search might be faster.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Wed Apr 15, 2015 10:48 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

You are searching through 50,000 records, each one over 1.5MB......

Are you sure you are not treating the queue as a database? That is, after all, rule number one of 'no-no's

Having said that even at 1.5MB ...1 second per message is still an enormous time. On my relatively modest development machine I can browse 2MB persistent messages at a rate of about 400 per second. You're not doing this over a client connection are you?

Regards,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
vsathyan
PostPosted: Tue Jul 07, 2015 10:06 am    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

Yes Paul, we are doing it over a MQ client connection over a network. Hence the delay.

We have a centralized web-service which connects to requested queue managers, opens the queue which has messages and browses for specific strings.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Jul 07, 2015 10:25 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vsathyan wrote:
We have a centralized web-service which connects to requested queue managers, opens the queue which has messages and browses for specific strings.


That's a horrible design and very anti-pattern; as a previous poster eloquently put it:

PaulClarke wrote:
That is, after all, rule number one of 'no-no's


There's no way you should design a system that requires you to browse through messages looking for "the right one" according to content. For no other reason (as you've discovered) that it doesn't scale. Hence it's position at the top of the "don't do this" list.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
vsathyan
PostPosted: Tue Jul 07, 2015 10:58 am    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

@Vitor,

Thanks for the prompt response.

This is not used on a regular basis. The web service will be enabled only when required, configured to connect to the required queue manager and browse the message content for specific data.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Jul 07, 2015 11:04 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vsathyan wrote:
This is not used on a regular basis. The web service will be enabled only when required, configured to connect to the required queue manager and browse the message content for specific data.


At 17 hours (your estimate) I hope it's not for an urgent requirement.....

I look forward to your update on what improvements MQGMO_MARK_BROWSE_CO_OP makes to response time.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Tue Jul 07, 2015 11:17 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

It is asking a lot that MQGMO_MARK_BROWSE_CO_OP is going to fix the performance problem. Issuing 50,000 MQGets and sending 75 GB over a client connection is going to take some time

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Tue Jul 07, 2015 11:19 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

PaulClarke wrote:
It is asking a lot that MQGMO_MARK_BROWSE_CO_OP is going to fix the performance problem. Issuing 50,000 MQGets and sending 75 GB over a client connection is going to take some time


Hence my interest, and knowledge of the OP's Plan B.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ API Support » Multiple Readers with MQOO_BROWSE
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.