| Author |
Message
|
| kraf |
 Posted: Wed Sep 12, 2012 2:47 am Post subject: Getting a (too) large message with two MQGET calls |
 |
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hi all,
We are trying to implement a procedure that will allow us to get a message from MQ, and if it is larger than the buffer size, do a MQGET again with the correct buffer size (as read from the data length returned from the first call).
The MQ is opened with the following options:
| Code: |
compute ws-mqm-cr-options = mqoo-input-shared
call 'MQOPEN' using ws-mqm-cr-hconn
mqod
ws-mqm-cr-options
ws-mqm-cr-hobj
ws-mqm-cr-compcode
ws-mqm-cr-reason |
Then in the first MQGET call we do as follows:
| Code: |
compute mqgmo-options = mqgmo-wait
+ mqgmo-convert
+ mqgmo-syncpoint
call 'MQGET' using ws-mqm-cr-hconn
ws-mqm-cr-hobj
mqmd
mqgmo
ws-mqm-cr-length
message-inputarea
ws-mqm-cr-data-length
ws-mqm-cr-compcode
ws-mqm-cr-reason |
If the message is larger than the buffer size in “ws-mqm-cr-length” then the the call fails with a reason code of 2080 (MQRC_TRUNCATED_MSG_FAILED). However the actual size of the message is returned in the response (ws-mqm-cr-data-length). The message in the “message-inputarea” contains garbage. Also the compcode is returned as 1.
The idea is now to resize the buffer with the size obtained from the first call (ws-mqm-cr-length) and then call MQGET again. We use the same options as above.
Now the message still contains garbage, however the reason code nor the compcode contain any errors.
Now the question is why this happens and how to remedy it?
We are aware of the BROWSE_FIRST option, but why does the second MQGET give us a reason code OK but garbage in the buffer?
I would highly appreciate any ideas, comments, and solutions!
Kind regards
Kraf |
|
| Back to top |
|
 |
| exerk |
| Posted: Wed Sep 12, 2012 3:24 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Getting 'garbage' in the buffer both times suggests to me that the convert isn't happening, so, the question is, is the message content being sent as MQFMT_STRING?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 12, 2012 4:17 am Post subject: Re: Getting a (too) large message with two MQGET calls |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| kraf wrote: |
Hi all,
We are trying to implement a procedure that will allow us to get a message from MQ, and if it is larger than the buffer size, do a MQGET again with the correct buffer size (as read from the data length returned from the first call).
|
It would be far simpler to make the buffer larger. This will accomplish the MQGET in one MQI call, rather than two MQI calls for each message.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| kraf |
| Posted: Wed Sep 12, 2012 4:18 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hi Exerk,
The content of the message is being sent as MQFMT_STRING.
Futhermore, if the message size is smaller than the buffer size when calling MQGET the first time, then the content of the message is correct (in our case human readable), and we can proceed as planned.
Best regards
Kraf |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 12, 2012 4:20 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| kraf wrote: |
Hi Exerk,
The content of the message is being sent as MQFMT_STRING.
Futhermore, if the message size is smaller than the buffer size when calling MQGET the first time, then the content of the message is correct (in our case human readable), and we can proceed as planned.
Best regards
Kraf |
Then go with bruce2359's suggestion and set your buffer to be a little larger than the MAXMSGL of the queue - which should be set at that of the largest message you're expecting.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 12, 2012 4:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
How do you resize the message-inputarea?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| kraf |
| Posted: Wed Sep 12, 2012 4:49 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hi Exerk and Bruce2359,
Unfortunately I am working in an environment where it is not allowed to initialize a large area each time a message is received, which is why we started out on this quest.
The message-input area is Getmain’ed. So for the first call we getmain an area with the same size as the buffer length. This size has been set to accommodate 99.9% of the incoming messages (32kb).
If the first call then fails, we getmain a new area again with the same size as the new buffer length(>32kb).
Have you got other suggestions to what could be wrong?
I appreciate your time.
Best regards
Kraf |
|
| Back to top |
|
|
| nathanw |
| Posted: Wed Sep 12, 2012 4:58 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
I would ask why you cannot
| Code: |
| initialize a large area each time a message is received |
This is the first time I have heard of this
_________________
Who is General Failure and why is he reading my hard drive?
Artificial Intelligence stands no chance against Natural Stupidity.
Only the User Trace Speaks The Truth  |
|
| Back to top |
|
|
| kraf |
| Posted: Wed Sep 12, 2012 5:01 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hmm I just wondered, when you write:
| exerk wrote: |
...set your buffer to be a little larger than the MAXMSGL of the queue... |
Would it matter if the buffer is excatly the same size of the MAXMSGL?
Or, stated differently; Why should it be a little larger?
Are there something that requires some sort of overhead?
And if yes, what and how much?
Best regards
Kraf |
|
| Back to top |
|
|
| kraf |
| Posted: Wed Sep 12, 2012 5:11 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hi nathanw,
| Quote: |
I would ask why you cannot
...
This is the first time I have heard of this
|
It is possible, no doubt about that. However I am operating under a development policy that dissallows it. Which is unfurtunate, since is seems like the easiest way of doing it.
Best regards
Kraf |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 12, 2012 5:12 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| kraf wrote: |
Hmm I just wondered, when you write:
| exerk wrote: |
...set your buffer to be a little larger than the MAXMSGL of the queue... |
Would it matter if the buffer is excatly the same size of the MAXMSGL?
Or, stated differently; Why should it be a little larger?
Are there something that requires some sort of overhead?
And if yes, what and how much?
Best regards
Kraf |
No, it wouldn't matter if your buffer size was greater than MAXMSGL, and if you expect messages > 32KB you should know what is the maximum size of message you will receive. Generally I set MAXMSGL on queues to be that plus 10% and suggest to the relevant developers to set that as their buffer size, so if your 'large' messages are 34KB for example your buffer size would be 37.5KB, which would allow for all the messages you are likely to receive without you having to do two calls.
I suspect that the MAXMSGL for your queue manager and queues has been left at the initial value set at creation of the queue manager etc.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 12, 2012 5:19 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| kraf wrote: |
Unfortunately I am working in an environment where it is not allowed to initialize a large area each time a message is received, which is why we started out on this quest.
The message-input area is Getmain’ed. So for the first call we getmain an area with the same size as the buffer length. This size has been set to accommodate 99.9% of the incoming messages (32kb).
If the first call then fails, we getmain a new area again with the same size as the new buffer length(>32kb).
Have you got other suggestions to what could be wrong?
|
The two issues you raise are not related.
With regards to the buffer issue, I believe the developers have misunderstood GETMAIN, virtual storage vs. real storage.
GETMAIN acquires virtual storage only - not real storage - from the virtual storage pool. Real storage to hold the inbound message is only acquired when it is actually needed - when a message arrives, or if the app initializes the before before each MQGET.
I suspect that the developers initialize the (32k) virtual storage buffer (to blanks, for example) before each MQGET. If this is the case, then 32k of real storage is acquired. Initializing the buffer is unnecessary. Why?
Because the actual message length is returned from the MQGET, the app need only refer to that much storage of the buffer. The remaining, unused portion of the buffer can be ignored.
| kraf wrote: |
Unfortunately I am working in an environment where it is not allowed to initialize a large area each time a message is received, which is why we started out on this quest. |
If the app stays alive to consume messages (more than one), it is not necessary to FREEMAIN the buffer from the first message, and GETMAIN a new buffer for each subsequent message. Is this how the app behaves? The same buffer can (and should) be reused for all inbound messages.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| kraf |
| Posted: Thu Sep 13, 2012 1:42 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Hi Bruce2359 and Exerk,
Just to clarify and test whether I have understood you guys correctly:
Instead of using two calls to MQGET, I can just use one call. This can be allowed since the GETMAIN’ed area is virtual storage and is only take up as much space as needed by the incoming message.
So even though I set the buffer size to e.g. 100MB and likewise for the GETMAIN’ed area, the only memory physically allocated is what is needed by the incoming message. Meaning that if I get a message with a size of 10kB, only 10kB of the 4MB GETMAIN’ed area is used?
If this is the case, you have made my day!
I do not initialize anything related to the GETMAIN’ed area and it will be used for all the incoming messages.
Best regards
Kraf |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 13, 2012 5:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Yes, GETMAIN acquires virtual storage only, and validates the virtual storage address. Touching the virtual address initially causes a page-fault (virtual storage address not backed by real frame), which causes a real frame to be allocated.
Do a quick google search for MVS+GETMAIN.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| kraf |
| Posted: Thu Sep 13, 2012 5:12 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2012
Posts: 7
|
Most excellent. Thank you very much for the help.
Best Regards
Kraf |
|
| Back to top |
|
|
|
|
