ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » search for an MQ message in a queue

Post new topic  Reply to topic Goto page 1, 2  Next
 search for an MQ message in a queue « View previous topic :: View next topic » 
Author Message
sharonvk
PostPosted: Tue Nov 15, 2016 6:09 am    Post subject: search for an MQ message in a queue Reply with quote

Newbie

Joined: 21 Sep 2016
Posts: 8

Hi
we are working with a trade system , from time to time we have a failure and we need to retransmit messages from a certain block id( which is applicative identifier in the data field)
For this we browse the queue and search each message for the specific application id ( block id) , and from this point we retransmit all the messages

the problem is that sometimes we need to search like a million messages before we find it , and this take time sometimes 40 seconds

is there a better solution ? I thought about writing the application id the correlation id field and using MQGET to search for it .

does any one have a better solution ? ( quicker )

can you send a simple example how to search for correlation id using the MQ get ,
i assume that after finding the message the cursor is set to this message and all the messages i read are then after this message .

Thanks ( i'm using MQ 75
Back to top
View user's profile Send private message
zpat
PostPosted: Tue Nov 15, 2016 6:23 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

Operating system?

Programming language?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
sharonvk
PostPosted: Tue Nov 15, 2016 6:28 am    Post subject: Reply with quote

Newbie

Joined: 21 Sep 2016
Posts: 8

HI

MQ 75 on Linux red hot enterprise 64 bit
programming language C

Thanks
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Nov 15, 2016 6:38 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9392
Location: US: west coast, almost. Otherwise, enroute.

Search google for 'mq match options'
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 15, 2016 6:42 am    Post subject: Re: search for an MQ message in a queue Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

sharonvk wrote:
I thought about writing the application id the correlation id field and using MQGET to search for it .


You can do a get by correl id.

sharonvk wrote:
does any one have a better solution ? ( quicker )


Scale your system so you don't have a million messages in a queue. Seriously. You're using MQ like a database and if you really need to keep a million messages around, load them into a real database like DB2 or Oracle. Both of those can search a million rows in a lot less than 40 seconds.

sharonvk wrote:
can you send a simple example how to search for correlation id using the MQ get ,


There's one included in the samples that come with the software, and other examples anywhere MQ samples are found on the web.

sharonvk wrote:
i assume that after finding the message the cursor is set to this message and all the messages i read are then after this message .


Wrong assumption. Each MQGet starts from the top of the queue and looks for the first message that matches the criteria specified on the get. The only way to get the behavior you describe is to explicitly set the Browse option on the MQGet (i.e. do a non-destructive read) then a MQGet message under cursor followed by a browse next, get message under cursor, browse next, get message under cursor and so forth.

If you think 40 seconds is a long time, you'll hate how slowly a browse cursor works. You'll be able to drink a cup of coffee using the programming model I describe on a queue of a million messages.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
sharonvk
PostPosted: Tue Nov 15, 2016 7:13 am    Post subject: Reply with quote

Newbie

Joined: 21 Sep 2016
Posts: 8

Vitor

We currently use the browse next in a loop till we find the requested message , then we continue with browse next and write the messages to our trade queues

The question i ask is about the search loop , does the search by correlation id will actually drop the time of the first loop we do to search the message

regarding sizes of queues , usually we don't get millions messages in our queues , so we don't use a DB ,,,, ( it was mainly because of Donald Trump .....)
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 15, 2016 7:31 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

sharonvk wrote:
We currently use the browse next in a loop till we find the requested message , then we continue with browse next and write the messages to our trade queues


And you get a 40 second response on a queue of a million messages? I'm impressed. I bet your IBM sales rep is also impressed by the amount of resource you're spending...

sharonvk wrote:
The question i ask is about the search loop , does the search by correlation id will actually drop the time of the first loop we do to search the message


Ok, if I understand your question (and I don't think it survived translation into English unscathed!) a get by correlation id will outperform a browse by correlation id but won't allow you to get the next message after the one you've just got.

sharonvk wrote:
regarding sizes of queues , usually we don't get millions messages in our queues , so we don't use a DB ,,,, ( it was mainly because of Donald Trump .....)


If you have any depth of messages there's something fishy with your set up. You clearly have devoted a lot of resources to this, but I put it to you that holding any non-trivial number of messages is done better in a DB.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Tue Nov 15, 2016 8:05 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

If you are using browse in an application BEWARE - you are in danger of duplicating production messages if not careful.

Ensure you get the chosen message destructively before putting to another queue, you can make the get and put under the same syncpoint unit of work, then commit both operations at the same time.

This ensures no other copy of your application will do the same thing to the same message and if a crash happens then the message will be on one queue only (not lost and not duplicated).

If you don't understand this please do not proceed further. Call in someone experienced to do the job.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
Andyh
PostPosted: Tue Nov 15, 2016 9:27 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

I presume that what's being suggested here is setting the initial browse cursor by combining MQMO_MATCH_CORREL_ID in conjunction with MQGMO_BROWSE_FIRST, and then getting a succession of further messages with MQGMO_BROWSE_NEXT.

If there is a destructive MQGET between each browse then you want to be sure that you've got the latest service applied (there was a recent performance APAR in this area).

If you only want to re-process subsequent messages for the same application identifier then you can continue to specify MQMO_MATCH_CORREL_ID with MQGMO_BROWSE_NEXT.

I wouldn't expect any particular performance issues with such a design, and I'm a little concerned at the negative comments here with regard to browse performance.

I don't mean to appear to condone using MQ like a DB, for longer term storage and re-reading of messages, but I'm not sure what is the perceived performance problem with browse (unless it's the APAR referred to above).
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Nov 15, 2016 11:30 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9392
Location: US: west coast, almost. Otherwise, enroute.

An application that browses every message in a queue causes the MQMD and data payload for every message to be moved from queue disk to the app, only then for the app to discover it doesn't want/need this particular message.

Using MQ to match on MsgId or CorrelId or GroupId has MQ internals only pass a matching message. If memory serves (waiting here to be corrected), MQ will search through MQMD indices looking for a matching message, and pass nothing to the app if there's no match.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Tue Nov 15, 2016 3:41 pm    Post subject: Re: search for an MQ message in a queue Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3251
Location: London, ON Canada

sharonvk wrote:
can you send a simple example how to search for correlation id using the MQ get

Here you go:
Code:
MQGMO          gmo = {MQGMO_DEFAULT};
MQMD           md  = {MQMD_DEFAULT};

/* Set default value for MsgId */
memcpy(md.MsgId, MQMI_NONE, MQ_MSG_ID_LENGTH);

/* 'something' is what you want to search for. */
memcpy(md.CorrelId, something, MQ_CORREL_ID_LENGTH);

/* Get the message by CorrelID */
gmo.MatchOptions = MQMO_MATCH_CORREL_ID;

/* Set the options for the get calls         */
gmo.Options = MQGMO_WAIT | MQGMO_FAIL_IF_QUIESCING | MQGMO_NO_SYNCPOINT;
gmo.WaitInterval = 15000; /* 15 seconds */

MQGET(Hconn, HobjQ, &md, &gmo, bufSize, buffer, &msglen, &mqCC, &mqRC);

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
sharonvk
PostPosted: Tue Nov 15, 2016 11:50 pm    Post subject: Reply with quote

Newbie

Joined: 21 Sep 2016
Posts: 8

Thank you All

I will try searching with the browse by correlation id and then browse next
and see how it effects the performance .
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Nov 16, 2016 12:06 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

If you locate a message with get browse, remember to do a proper syncpointed get (under the browse cursor to avoid another search) before processing the data.

Bear in mind that the message may have been removed by another instance of your application before your proper get. Always code to allow for multiple instances of your application (or any other) to be running.

As I mentioned, do NOT propagate data obtained with browse or you risk duplication. There are ways to lock messages etc - but just using the API normally is best. Browsing is generally for utilities to use.

There is actually no need to use browse at all - you can just issue MQGET for the matching message. If it is there you will get it, if not then you get a RC 2033. Simple, and safer in terms of data integrity.

Browse is not some kind of searching function. It just means the get is non-destructive. Useful for tools like MQ explorer or MO71 to use, and I believe some versions of JMS use it for selector purposes - but there are other ways to do that as well.

You don't mention if your application is client or bindings, it can be very inefficient to examine each message with browse over a client channel.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Nov 16, 2016 12:31 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

sharonvk wrote:
Linux red hot enterprise 64 bit


Are your servers getting over-heated? Too much message searching?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Nov 16, 2016 8:26 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

zpat wrote:
sharonvk wrote:
Linux red hot enterprise 64 bit


Are your servers getting over-heated? Too much message searching?


Too much cinnamon in the AC unit.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » IBM MQ API Support » search for an MQ message in a queue
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.