| Author |
Message
|
| jsware |
 Posted: Fri Dec 22, 2006 5:57 am Post subject: Publishing large messages. |
 |
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
Firstly a Merry Christmas & Happy New Year.
Anybody know the correct mechanism for publishing large messages to MQ pubsub using segmentation? Is there any difference when doing so to Message Broker?
I have a 190MB of data that I want to publish as a single logical message. (Its a master refresh that works in parallel to master data updates which are much smaller).
I can publish smaller messages no problem. Each message has an MQRFH header with the publish command and topic name.
If I want to segment a message, I can tell MQ to segment the message for me. This creates multiple physical messages, the first has the MQRFH header and the rest do not. However, the MQMD format field states the payload starts with an MQRFH header. Presenting this to the MQ broker dumps them all onto the DLQ with the error below.
If there is a section in the manuals that discuss publishing large messages, please direct me to that. I don't mind reading manuals and have looked. I cannot find anywhere that talks about segmentation and publish/subscribe together.
44 4C 48 20 01 00 00 00 DLH ....
C9 08 00 00 53 4F 55 4C É...SOUL
54 45 53 54 2E 44 4F 4F TEST.DOO
4D 53 44 41 59 20 20 20 MSDAY
20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20
20 20 20 20 50 32 31 39 P219
31 36 54 20 20 20 20 20 16T
20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20
20 20 20 20 22 02 00 00 "...
52 03 00 00 4D 51 48 4D R...MQHM
44 45 20 20 0B 00 00 00 DE ....
43 3A 5C 55 74 69 6C 73 C:\Utils
5C 69 68 30 33 5C 72 66 \ih03\rf
68 75 74 69 6C 2E 65 78 hutil.ex
65 20 20 20 32 30 30 36 e 2006
31 32 32 32 31 32 35 35 12221255
33 38 38 35 4D 44 45 20 3885MDE
02 00 00 00 48 00 00 00 ....H...
22 02 00 00 52 03 00 00 "...R...
4D 51 48 52 46 20 20 20 MQHRF
00 00 00 00 41 4D 51 20 ....AMQ
50 32 31 39 31 36 54 20 P21916T
20 20 20 20 77 8D 79 45 w.yE
20 00 16 0C 01 00 00 00 .......
00 00 00 00 03 00 00 00 ........
50 FF 3F 00 52 46 48 20 Pÿ?.RFH
01 00 00 00 44 00 00 00 ....D...
22 02 00 00 B5 01 00 00 "...µ...
20 20 20 20 20 20 20 20
00 00 00 00 4D 51 50 53 ....MQPS
43 6F 6D 6D 61 6E 64 20 Command
50 75 62 6C 69 73 68 20 Publish
4D 51 50 53 54 6F 70 69 MQPSTopi
63 20 54 65 73 74 20 20 c Test
00 00 00 00 00 00 00 00 ........
00 00 00 00 00 00 00 00 ........
Decoding this with MQExplorer shows up with MQRC 2249 (MQRC_MSG_FLAGS_ERROR). I cannot find a discussion in the manuals why this would happen.
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
 |
| JosephGramig |
| Posted: Fri Dec 22, 2006 6:03 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Perhaps you have bad report options specified.
Are you specifying report options?
If so, what options and how?
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3 |
|
| Back to top |
|
|
| jsware |
| Posted: Fri Dec 22, 2006 6:16 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
I just found athat MQ pubsub does not support publishing of message segments which is probably why they get rejected.
I cannot find a statement in the WMB documentation that indicates whether I can publish segmented messages in Message Broker.
Anybody have any ideas of the correct way to do this in Message Broker?
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Dec 22, 2006 2:52 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
What kind of an "event" is it that requires 190mbs to properly represent?
That's a lot of data.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| jsware |
| Posted: Wed Dec 27, 2006 2:13 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| jefflowrey wrote: |
What kind of an "event" is it that requires 190mbs to properly represent?
That's a lot of data. |
The event that is occurring is a new catalogue launch which involves providing a complete refresh of all product details - over 18,000 of them. Individual products are not meaningful in their own right because the cat-roll needs to happen in its entirety. There is a separate "update" event for changes to individual products that happens during the life of a catalogue (e.g. product description changes).
We are using pub-sub to propagate to multiple destinations without the message producer knowing who the consumers are. The consumers are static (though can change between environments - dev/test/uat/prod etc) so the consumers are "statically" subscribed to the topic by an administrator.
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 27, 2006 2:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| scottj2512 wrote: |
Individual products are not meaningful in their own right because the cat-roll needs to happen in its entirety. There is a separate "update" event for changes to individual products that happens during the life of a catalogue (e.g. product description changes).
|
So what's the difference between 1 product changing and 18,000 changing that means all 18,000 have to be a single update rather than 18,000?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jsware |
| Posted: Wed Dec 27, 2006 3:31 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| Vitor wrote: |
| So what's the difference between 1 product changing and 18,000 changing that means all 18,000 have to be a single update rather than 18,000? |
I don't want to get into the semantic discussions around whether it should be one logical message or 18,000 logical messages. There are numerous other threads on this forum that discuss the +/-ves of both ways. Truth is its a file which needs to be sent to multiple destinations and processed as a single file by the consuming applications. The file is a single logical unit of integration that in my opinion cannot be meaningfully subdivided.
Pub/sub is a means by which I can have a 1-n producer-consumer relationship without the producer knowing there are n consumers. Segmentation is a means by which a single logical large message (>4MB consitutes a large message by my standards) can be sent as multiple physical messages.
My question is how (if at all) can I get pub/sub and segmented messaging working together through message broker as the pub/sub engine.
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 27, 2006 3:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| scottj2512 wrote: |
| Truth is its a file which needs to be sent to multiple destinations and processed as a single file by the consuming applications. The file is a single logical unit of integration that in my opinion cannot be meaningfully subdivided. |
It's your application, your requirement and I defend the right of anyone to hold their own opinion. 
No producer ever knows how many consumers a file will be copied to and I question the wisdom of doing a file transfer over a messaging set up such as this, but I again repeat you know your situation best.
AFAIK you can't use segmentation & pub/sub but my knowledge is theoretical (as I suspect is obvious, I belong to the multiple messages school of thought!) so I bow out and await the response of better minds than mine.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
