| Author |
Message
|
| lojian |
 Posted: Wed Feb 14, 2007 6:18 pm Post subject: How can I receive a very large message in small pieces |
 |
|
Newbie
Joined: 14 Feb 2007
Posts: 4
|
Hi,
I met a problem with receiving a very big message (20M+) in my program which is running on a constrainted environment. I can not allocate 20M memory to store this message. I wonder is there any way can help me to receive this ONE message serval times, one time for a small piece of the large message and write to the disk file.
I can not change the source code of the PUT program.
Does any one come up some idea, can you tell me, please ?
Thanks.  |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Feb 14, 2007 6:44 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Basically, no.
There's no way to tell an MQGet to return the part of the message buffer from X to Y. You can tell it to retrieve the part of the message buffer from 0 to Y, but not from X to Y.
As far as I remember.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Wed Feb 14, 2007 7:08 pm Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
Could you client to the machine from another one with more resources and get the message?...otherwise you may be SOL  |
|
| Back to top |
|
|
| lojian |
| Posted: Wed Feb 14, 2007 7:57 pm Post subject: |
|
|
Newbie
Joined: 14 Feb 2007
Posts: 4
|
Thanks for help. 
It seems that MQ does not provide this kind of ability.
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 15, 2007 2:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
If the put allows for segmentation you could try to have the qmgr segment the message...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 16, 2007 3:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
If the put allows for segmentation you could try to have the qmgr segment the message...
|
This does require a change on the put side, which was ruled out in an early post...
Never ceases to amaze me how the "partnership" between sender & receiver breaks down when there's a change to be made 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Feb 16, 2007 6:27 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Actually, in this case, I think application segmentation is what is desired. Qmgr segmentation would produce the same result on the GETTING client side - a big buffer with the entire message.
App Segmentation would allow the client to receive each segment as an individualized Get.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| tleichen |
| Posted: Fri Feb 16, 2007 6:45 am Post subject: Re: How can I receive a very large message in small pieces |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
| lojian wrote: |
| ...I can not change the source code of the PUT program.... |
| jefflowrey wrote: |
Actually, in this case, I think application segmentation is what is desired. Qmgr segmentation would produce the same result on the GETTING client side - a big buffer with the entire message.
App Segmentation would allow the client to receive each segment as an individualized Get. |
But how could this be done without changing the source code of the PUT program? 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Feb 16, 2007 6:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Well, it can't.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| jsware |
| Posted: Sat Feb 17, 2007 2:55 pm Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| jefflowrey wrote: |
Actually, in this case, I think application segmentation is what is desired. Qmgr segmentation would produce the same result on the GETTING client side - a big buffer with the entire message.
App Segmentation would allow the client to receive each segment as an individualized Get. |
The qmgr does not have to reassemble a qmgr segmented message at the receiving end. You can use MQMF_SEGMENTATION_ALLOWED at the sending end and get each segment individually. You do not have to use MQGMO_COMPLETE_MSG at the receiving end.
This still requires a put program change which has been ruled out
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Feb 17, 2007 6:54 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
| scottj2512 wrote: |
| jefflowrey wrote: |
Actually, in this case, I think application segmentation is what is desired. Qmgr segmentation would produce the same result on the GETTING client side - a big buffer with the entire message.
App Segmentation would allow the client to receive each segment as an individualized Get. |
The qmgr does not have to reassemble a qmgr segmented message at the receiving end. You can use MQMF_SEGMENTATION_ALLOWED at the sending end and get each segment individually. You do not have to use MQGMO_COMPLETE_MSG at the receiving end.
This still requires a put program change which has been ruled out |
He never specified whether or not the put program had the option MQMF_SEGMENTATION_ALLOWED set or not. Who knows maybe he does'nt have to change the put program, but just the max msg size on the put queue?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jsware |
| Posted: Tue Feb 20, 2007 9:24 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| fjb_saper wrote: |
| He never specified whether or not the put program had the option MQMF_SEGMENTATION_ALLOWED set or not. Who knows maybe he does'nt have to change the put program, but just the max msg size on the put queue? |
Jeff mentioned segmentation saying that app segmentation was required because qmgr segmentation would produce a single large message on the get. My comment relates to the fact that if the putting application could use qmgr segmentation does not force the getting qmgr to use qmgr un-segmentation. The problem is not queue size, its memory availability on the getting application.
Changing the putting application to use qmgr segmentation is as easy as adding the MQMF flag. Lojian can then process each segment separately.
This is all moot tho because the original post said that the putting app could not be changed - kind of painting into a corner
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
|
|
