| Author |
Message
|
| sharonvk |
 Posted: Tue Nov 15, 2016 6:09 am Post subject: search for an MQ message in a queue |
 |
|
Newbie
Joined: 21 Sep 2016
Posts: 8
|
Hi
we are working with a trade system , from time to time we have a failure and we need to retransmit messages from a certain block id( which is applicative identifier in the data field)
For this we browse the queue and search each message for the specific application id ( block id) , and from this point we retransmit all the messages
the problem is that sometimes we need to search like a million messages before we find it , and this take time sometimes 40 seconds
is there a better solution ? I thought about writing the application id the correlation id field and using MQGET to search for it .
does any one have a better solution ? ( quicker )
can you send a simple example how to search for correlation id using the MQ get ,
i assume that after finding the message the cursor is set to this message and all the messages i read are then after this message .
Thanks ( i'm using MQ 75 |
|
| Back to top |
|
 |
| zpat |
| Posted: Tue Nov 15, 2016 6:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Operating system?
Programming language?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| sharonvk |
| Posted: Tue Nov 15, 2016 6:28 am Post subject: |
|
|
Newbie
Joined: 21 Sep 2016
Posts: 8
|
HI
MQ 75 on Linux red hot enterprise 64 bit
programming language C
Thanks |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 15, 2016 6:38 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Search google for 'mq match options'
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 15, 2016 6:42 am Post subject: Re: search for an MQ message in a queue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sharonvk wrote: |
| I thought about writing the application id the correlation id field and using MQGET to search for it . |
You can do a get by correl id.
| sharonvk wrote: |
| does any one have a better solution ? ( quicker ) |
Scale your system so you don't have a million messages in a queue. Seriously. You're using MQ like a database and if you really need to keep a million messages around, load them into a real database like DB2 or Oracle. Both of those can search a million rows in a lot less than 40 seconds.
| sharonvk wrote: |
| can you send a simple example how to search for correlation id using the MQ get , |
There's one included in the samples that come with the software, and other examples anywhere MQ samples are found on the web.
| sharonvk wrote: |
| i assume that after finding the message the cursor is set to this message and all the messages i read are then after this message . |
Wrong assumption. Each MQGet starts from the top of the queue and looks for the first message that matches the criteria specified on the get. The only way to get the behavior you describe is to explicitly set the Browse option on the MQGet (i.e. do a non-destructive read) then a MQGet message under cursor followed by a browse next, get message under cursor, browse next, get message under cursor and so forth.
If you think 40 seconds is a long time, you'll hate how slowly a browse cursor works. You'll be able to drink a cup of coffee using the programming model I describe on a queue of a million messages.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sharonvk |
| Posted: Tue Nov 15, 2016 7:13 am Post subject: |
|
|
Newbie
Joined: 21 Sep 2016
Posts: 8
|
Vitor
We currently use the browse next in a loop till we find the requested message , then we continue with browse next and write the messages to our trade queues
The question i ask is about the search loop , does the search by correlation id will actually drop the time of the first loop we do to search the message
regarding sizes of queues , usually we don't get millions messages in our queues , so we don't use a DB ,,,, ( it was mainly because of Donald Trump .....) |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 15, 2016 7:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sharonvk wrote: |
| We currently use the browse next in a loop till we find the requested message , then we continue with browse next and write the messages to our trade queues |
And you get a 40 second response on a queue of a million messages? I'm impressed. I bet your IBM sales rep is also impressed by the amount of resource you're spending...
| sharonvk wrote: |
| The question i ask is about the search loop , does the search by correlation id will actually drop the time of the first loop we do to search the message |
Ok, if I understand your question (and I don't think it survived translation into English unscathed!) a get by correlation id will outperform a browse by correlation id but won't allow you to get the next message after the one you've just got.
| sharonvk wrote: |
| regarding sizes of queues , usually we don't get millions messages in our queues , so we don't use a DB ,,,, ( it was mainly because of Donald Trump .....) |
If you have any depth of messages there's something fishy with your set up. You clearly have devoted a lot of resources to this, but I put it to you that holding any non-trivial number of messages is done better in a DB.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 15, 2016 8:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you are using browse in an application BEWARE - you are in danger of duplicating production messages if not careful.
Ensure you get the chosen message destructively before putting to another queue, you can make the get and put under the same syncpoint unit of work, then commit both operations at the same time.
This ensures no other copy of your application will do the same thing to the same message and if a crash happens then the message will be on one queue only (not lost and not duplicated).
If you don't understand this please do not proceed further. Call in someone experienced to do the job.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Nov 15, 2016 9:27 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
I presume that what's being suggested here is setting the initial browse cursor by combining MQMO_MATCH_CORREL_ID in conjunction with MQGMO_BROWSE_FIRST, and then getting a succession of further messages with MQGMO_BROWSE_NEXT.
If there is a destructive MQGET between each browse then you want to be sure that you've got the latest service applied (there was a recent performance APAR in this area).
If you only want to re-process subsequent messages for the same application identifier then you can continue to specify MQMO_MATCH_CORREL_ID with MQGMO_BROWSE_NEXT.
I wouldn't expect any particular performance issues with such a design, and I'm a little concerned at the negative comments here with regard to browse performance.
I don't mean to appear to condone using MQ like a DB, for longer term storage and re-reading of messages, but I'm not sure what is the perceived performance problem with browse (unless it's the APAR referred to above). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 15, 2016 11:30 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
An application that browses every message in a queue causes the MQMD and data payload for every message to be moved from queue disk to the app, only then for the app to discover it doesn't want/need this particular message.
Using MQ to match on MsgId or CorrelId or GroupId has MQ internals only pass a matching message. If memory serves (waiting here to be corrected), MQ will search through MQMD indices looking for a matching message, and pass nothing to the app if there's no match.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Nov 15, 2016 3:41 pm Post subject: Re: search for an MQ message in a queue |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
| sharonvk wrote: |
| can you send a simple example how to search for correlation id using the MQ get |
Here you go:
| Code: |
MQGMO gmo = {MQGMO_DEFAULT};
MQMD md = {MQMD_DEFAULT};
/* Set default value for MsgId */
memcpy(md.MsgId, MQMI_NONE, MQ_MSG_ID_LENGTH);
/* 'something' is what you want to search for. */
memcpy(md.CorrelId, something, MQ_CORREL_ID_LENGTH);
/* Get the message by CorrelID */
gmo.MatchOptions = MQMO_MATCH_CORREL_ID;
/* Set the options for the get calls */
gmo.Options = MQGMO_WAIT | MQGMO_FAIL_IF_QUIESCING | MQGMO_NO_SYNCPOINT;
gmo.WaitInterval = 15000; /* 15 seconds */
MQGET(Hconn, HobjQ, &md, &gmo, bufSize, buffer, &msglen, &mqCC, &mqRC); |
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| sharonvk |
| Posted: Tue Nov 15, 2016 11:50 pm Post subject: |
|
|
Newbie
Joined: 21 Sep 2016
Posts: 8
|
Thank you All
I will try searching with the browse by correlation id and then browse next
and see how it effects the performance . |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 16, 2016 12:06 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you locate a message with get browse, remember to do a proper syncpointed get (under the browse cursor to avoid another search) before processing the data.
Bear in mind that the message may have been removed by another instance of your application before your proper get. Always code to allow for multiple instances of your application (or any other) to be running.
As I mentioned, do NOT propagate data obtained with browse or you risk duplication. There are ways to lock messages etc - but just using the API normally is best. Browsing is generally for utilities to use.
There is actually no need to use browse at all - you can just issue MQGET for the matching message. If it is there you will get it, if not then you get a RC 2033. Simple, and safer in terms of data integrity.
Browse is not some kind of searching function. It just means the get is non-destructive. Useful for tools like MQ explorer or MO71 to use, and I believe some versions of JMS use it for selector purposes - but there are other ways to do that as well.
You don't mention if your application is client or bindings, it can be very inefficient to examine each message with browse over a client channel.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 16, 2016 12:31 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| sharonvk wrote: |
Linux red hot enterprise 64 bit
|
Are your servers getting over-heated? Too much message searching? 
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 16, 2016 8:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
| sharonvk wrote: |
Linux red hot enterprise 64 bit
|
Are your servers getting over-heated? Too much message searching? |
Too much cinnamon in the AC unit.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
