| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Setting Buffer size after MQGET |
« View previous topic :: View next topic » |
| Author |
Message
|
| raceyrocket |
 Posted: Wed Sep 10, 2008 2:57 am Post subject: Setting Buffer size after MQGET |
 |
|
Newbie
Joined: 10 Sep 2008
Posts: 2
|
I am new to this and this maybe a simple question.
I am reading an MQ with an MQGET with a BROWSE-FIRST option to get the size of message and to be non-destructive. I want to then do another MQGET without the BROWSE-FIRST. I want to be able to set the buffer for the second MQGET to the message size length that came back with the BUFFER-LENGTH-RETURNED from the first get.
The problem is that the buffer is a character variable and the BUFFER-LENGTH-RETURNED is numeric.
I don't know what the largest message will be coming from te MQ so want to be able to set the buffer accordly |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Sep 10, 2008 3:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The doc and programming manual tells that the max msg size is 100 MB.
The default max msg size is 4 MB.
The buffer is independant from its content whether char or not. However the buffer size will vary with single and multiple byte char sets for the same characters. And as you are on the MF and using (presumably) text messages (MQFMT_STRING) did you do a get with Convert option?? Did you set the CCSID on the message before the get if it is needed in a CCSID different from the mainframe's qmgr CCSID??
As well you do not need a browse_first to determine the buffer size. Browse first has other purposes. If you keep the buffer size in memory and allocate a buffer of this size to each get, and up your buffer size whenever the get fails because the buffer is too small, you should be fine and effectively shooting for max msg size on queue....
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Wed Sep 10, 2008 4:03 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
do you mean "datalength" that is returned by the mqget browse?
"buffer" is the storage area that you provide for MQ to place the message in. bufferlength is the length of this buffer.
if datalength is < then bufferlength why modify the buffer? you already alocated the buffer for the browse. so the following mqget is again with buffer and bufferlength and you will get datalength characters returned in your buffer
you can always get smaller messages with bigger buffers, and the number of characters you really mqget is returned in datalength.
if datalength is > then bufferlength, you have to getmain a bigger buffer of size datalength first before you get the message.
buffer smaller then datalength only works if you specify the accept truncated message option.
afaik, yiou already have to use the accept-truncated-msg on your browse
_________________
Regards, Butcher |
|
| Back to top |
|
|
| raceyrocket |
| Posted: Wed Sep 10, 2008 5:42 am Post subject: |
|
|
Newbie
Joined: 10 Sep 2008
Posts: 2
|
Yeah I meant the data length.
The request to me was not to have a large storage area buffer. Are you saying that to have a large buffer is not a big issue when possibly for the majority of message data lengths would be significantly smaller.
I was going to go with the smaller buffer and then if the datalength was bigger do a GETMAIN on the datalength, but can you:-
EXEC CICS GETMAIN SET(ADDRESS OF BUFFER)
LENGTH(datalength)
INITIMG(SPACE)
RESP(RESPONSE-CODE)
END-EXEC.
Then go the MQGET on the BUFFER |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 10, 2008 6:02 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Pre-getting a message in browse mode to determine the actual length of the next message is one of those new-to-MQ programming misconceptions.
Given the virtual RAM world we live in, allocate the maximum message buffer in your program (100meg or whatever the business requirement is); then you won't need to do two MQGETs to get one message.
Another new-to-MQ misconception is the programmer doing an MQINQ to confirm that there are messages in the queue (curdepth) before doing an MQGET to get the message. Again, two MQI calls that can be satisfied with one: MQGET with WAIT.
What are some others?
Welcome to MQ.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Sep 10, 2008 6:33 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Always design applications to allow for other programs to be using the same queue at the same time.
If you do have two MQGETs - how can you be sure the message has not been retrieved by another program in the meantime?
I have heard of programs doing the first MQGET with a zero length buffer then inspecting the MQMD before doing another MQGET.
But bear in the mind the second MQGET should be done using match on the msgid to ensure that you get the same message as before (if it's still there).
Better to inquire on the queue max message length (and have this set to a suitable value not the default 4 MB). Use this as your buffer size in the first place...! |
|
| Back to top |
|
|
| dkeister |
| Posted: Wed Sep 10, 2008 6:49 am Post subject: |
|
|
Disciple
Joined: 25 Mar 2002
Posts: 184
Location: Purchase, New York
|
I find this a very interesting thread. I have designed to set the get buffer to a little bigger than expected message size when browsing (with accept truncated messages set). Occasionally I get 100 meg messages on the queue and since this is across the network, I only occasionally want to get the full message body. Am I being too anal?
_________________
Dean Keister |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 10, 2008 8:32 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| I have designed to set the get buffer to a little bigger than expected message size when browsing (with accept truncated messages set). |
To acomplish what? To conserve resources?
| Quote: |
| Occasionally I get 100 meg messages on the queue and since this is across the network, I only occasionally want to get the full message body. |
The ususal (best practice) is to allocate the buffer for the largest message that the program expects, then do a single MQGET. If the actual message is shorter than the buffer, then the buffer will only contain the actual message - the actual message length is returned in the MQGET call.
[EDIT]
| Quote: |
| ...and since this is across the network, I only occasionally want to get the full message body |
.
All MQGETs are from local queues, so the message has already traversed the network.
I don't see what issue you are addressing with your design.
| Quote: |
| Am I being too anal? |
On the 0-10 anal scale, where do you see yourself? 
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| David.Partridge |
| Posted: Wed Sep 10, 2008 11:22 pm Post subject: |
|
|
Master
Joined: 28 Jun 2001
Posts: 249
|
Please don't pre-allocate HUGE buffers, there's a common misconception that there's no cost associated here, but there most definitely is. If you allocate a large buffer, then space in the page and segment tables needs to be allocated, and also virtual storage set aside in the the page/swap files. Furthermore MQ needs to allocate a number of internal buffers at least as large as yours.
If all your CICS apps preallocated 100MB buffers, then I can imagine you CICS regions going SOS pretty sharply too.
In my opinion, the best approach is to allocate initially a "reasonable" size of buffer, and handle the MQRC_TRUNCATED_MSG_FAILED by reallocating the buffer and retrying.
As a matter of principle, I strongly advise against allowing 100MB messages in your MQ world. This is a classic case where message segmentation and or grouping will help the overall system responsiveness and throughput considerably, though the individual application throughput may not be as good (the greatest good for the greatest number).
_________________
Cheers,
David C. Partridge |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 11, 2008 5:56 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| If you allocate a large buffer, then space in the page and segment tables needs to be allocated, and also virtual storage set aside in the the page/swap files. |
True, unless your m/f is dramatically low on resources (central storage, DASD), MVS can handle this with ease.
Page/segment tables are virtual; and a 100 meg buffer in an application wouuld fit nicely under the 2g bar (31-bit), and would barely be noticed in 16EB (64-bit). Yes, DASD (paging space) is real; but most m/f shops will have more than enough.
| Quote: |
| Furthermore MQ needs to allocate a number of internal buffers at least as large as yours. |
True, but sysprogs set the number/size of the bufferpools; and these can be altered dynamically.
| Quote: |
| If all your CICS apps preallocated 100MB buffers... I strongly advise against allowing 100MB messages ... |
These are things to consider in the design, especially if all applications must process 100 meg messages; but I'd guess this is more the exception than the rule.
Given the abundant resources usually present on a m/f, and the tuning knobs (figurative) available, this would be my platform of choice for implementing 100meg applications while meeting service-level agreements.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Sep 11, 2008 4:36 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| Quote: |
| If you allocate a large buffer, then space in the page and segment tables needs to be allocated, and also virtual storage set aside in the the page/swap files. |
True, unless your m/f is dramatically low on resources (central storage, DASD), MVS can handle this with ease.
Page/segment tables are virtual; and a 100 meg buffer in an application wouuld fit nicely under the 2g bar (31-bit), and would barely be noticed in 16EB (64-bit). Yes, DASD (paging space) is real; but most m/f shops will have more than enough.
| Quote: |
| Furthermore MQ needs to allocate a number of internal buffers at least as large as yours. |
True, but sysprogs set the number/size of the bufferpools; and these can be altered dynamically.
| Quote: |
| If all your CICS apps preallocated 100MB buffers... I strongly advise against allowing 100MB messages ... |
These are things to consider in the design, especially if all applications must process 100 meg messages; but I'd guess this is more the exception than the rule.
Given the abundant resources usually present on a m/f, and the tuning knobs (figurative) available, this would be my platform of choice for implementing 100meg applications while meeting service-level agreements. |
Just because resources and capacity exist, it doesn't mean they should be used ! The MQ message buffer should only be as large as the maximum message length it expects to get, not the absolute maximum length that MQ supports.
It is sad that most designers and developers regard computer systems as having almost limitless resources and have scant regard for using them efficiently. Many times I have seen system problems caused by applications using excessive resources, and when I delve into their design I am shocked by what they think they can get away with.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 11, 2008 6:00 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| not the absolute maximum length that MQ supports |
I wasn't advocating making all messages 100meg. That would truly be a waste of valuable resources. Messages should only as big as they need to be.
But if the business requirement is for 100meg messages, the mainframe - especially MVS - can handle it without any sweat.
Going to the other extreme, there would be a dramatic increase in resource usage if all applications were to be written to do an mqget-browse to establish the exact buffer size, then dynamically allocate a buffer, then do a destructive mqget into the correctly sized buffer. This is clever coding; but not good coding - on any platform.
Clever coding might conserve resources today; but hardware gets upgraded or replaced rendering clever coding pointless.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Sep 14, 2008 8:43 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| I wasn't advocating making all messages 100meg. That would truly be a waste of valuable resources. Messages should only as big as they need to be. |
no problem
| Quote: |
| Going to the other extreme, there would be a dramatic increase in resource usage if all applications were to be written to do an mqget-browse to establish the exact buffer size, then dynamically allocate a buffer, then do a destructive mqget into the correctly sized buffer. This is clever coding; but not good coding - on any platform. |
Agree. The buffer should be initially allocated with sufficient space, and used for every GET. Its a nice touch if the code can reallocate a larger buffer if required.
An alternative is to use asynchronous consume in MQ V7.0, where MQ allocates and manages the message data buffer for you.
_________________
Glenn |
|
| Back to top |
|
|
| dkeister |
| Posted: Mon Sep 15, 2008 6:30 am Post subject: |
|
|
Disciple
Joined: 25 Mar 2002
Posts: 184
Location: Purchase, New York
|
In my post, I may have left out some of the details, but most of my work is not with mainframes. Also, I started programming in 1966 and way back then, 64k held a BIG operating system ... enough of Rip Van Winkle...
Anyway, while concerned about individual machine capacity when pre-allocating buffers, I was concerned with network delays when sending large amounts of data around. An example:
I have a need to identify messages on a queue by looking at data in the first 200 characters. The messages are usually less that 5000 characters but occasionally (once a month) there are 100 M messages. So most of the time 5-6 k buffers do the job with minimal retransmission. I didn't want to reserve system resources for the 'once a month' scenario...
Once I find the message I want, then I browse a second time if the message was truncated.
So, I have a buffer size of 8000 characters. I Browse with accept truncated messages. If the message is the one I want I and no truncation, I'm done. If it is the one I want and truncated, I can rebrowse with the appropriate buffer size.
With this technique, if I'm not looking for the 100 M message, but it (they) are on the queue, I don't have to transmit the 100 M to my client application just to see I don't need it.
PS. Common queue, common function, but once a month there is humongous amounts of data, the rest of the time, less than 5 k...
_________________
Dean Keister |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 15, 2008 6:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
If you are mqgetting a message from a queue, it is a local queue, AND the message has already traversed your network to get to the local queue.
Way back then, when you and I were developing applications on really expensive hardware, best practice, a ROT (rule-of-thumb) was to write the code around hardware constraints. Remember desk-debugging? (For youngsters: on million-dollar hardware compiling a program cost way more than programmer salary. The best practice was to spend hours looking at our code on paper to do debugging.)
We discovered that, over time, the hardware and software environment changed; creative code lost its its value. Now, hardware and software are relatively cheap compared to programmer time. The ROT that made sense back then has long ago become an IROT (Invalid Rule-of-Thumb). Add to this the move from real resources (cpu, central storage, and I/O. Much of this has been virtualized. Certainly, your applications run virtual, in virtual address spaces, and consume virtual resources.
As I see it, the underlying issue in this post has been using valuable system resources vs. abusing valuable system resources.
Requiring two MQGETs to effectively get the next message when a single MQGET will do, deliberately wastes real resources (like cpu). Setting a buffer to the size of the biggest message that might appear in the queue consumes virtual resources (virtual storage), and will only use system resources (real storage) if a really big message arrives.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
