ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportHandling large message in IIB

Post new topicReply to topic
Handling large message in IIB View previous topic :: View next topic
Author Message
MD
PostPosted: Wed Oct 04, 2017 3:27 am Post subject: Handling large message in IIB Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Hi All,

I have the requirement like below..

I have to develop a flow in IIB.
My input message is XML and it can be upto 100 MB.
I need to transfer this XML message to Flatfile(CSV) and need to be send it to target application as single message.

Is this doable in IIB ?
Please suggest some possible scenarios.

Thanks in advance.
MD
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 04, 2017 4:52 am Post subject: Re: Handling large message in IIB Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24644
Location: Ohio, USA

MD wrote:
I need to transfer this XML message to Flatfile(CSV) and need to be send it to target application as single message.


The target wants 100Mb of data as a single comma delimited record. Seriously? Whoever designed that needs professional help; both in terms of program design and mental health.

MD wrote:
Is this doable in IIB ?


Yes, it's just extremely odd.

MD wrote:
Please suggest some possible scenarios.


There's advice in the InfoCenter for handling large files / messages.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Oct 04, 2017 5:05 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

As an IT professional - recommend a more sensible approach, one that makes implementation easier and doesn't impose limits.

Assuming you do mean a MQ message, using 100 MB in MQ often needs changes to channels, xmit queues etc for the max message length.

What happens when 100 MB is exceeded? You are better off designing a solution that uses multiple messages (e.g. as a message group), which means you can keep each one smaller and avoid any upper size limit.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
MD
PostPosted: Wed Oct 04, 2017 7:53 pm Post subject: Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Hi zpat,

Thanks for the reply.

Even we are thinking to use MQFT and split the message(Grouping) and send.

If we use Grouping, Broker MQ-Input node wait until it receives the last message and then it take all the grouped messages as single message and process right ?
So can Broker handle this ?
Since it is XML, will any parsing errors come?

Thanks
Back to top
View user's profile Send private message
MD
PostPosted: Wed Oct 04, 2017 8:14 pm Post subject: Re: Handling large message in IIB Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Quote:
The target wants 100Mb of data as a single comma delimited record. Seriously?

If the Broker can handle 100 MB message at the incoming side and in transformation, it can be send as multiple records at output(Each record as one CSV file).


Any other advises are also welcome

Thanks
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Oct 04, 2017 11:14 pm Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

Yes, broker can handle grouped messages.

Tick the box on the MQinput node ("All messages available").

This means that when the group is complete on the input queue, then they will get passed to the flow one at a time. If you want to detect the last message in the group you can examine the MQMD flags.

Incomplete groups will remain on the queue, so you might want to use a (long) expiry value to avoid indefinite orphaning.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
MD
PostPosted: Wed Oct 04, 2017 11:27 pm Post subject: Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Quote:
Tick the box on the MQinput node ("All messages available").

I have done it.

Quote:
This means that when the group is complete on the input queue, then they will get passed to the flow one at a time.

Means each message will be processed separately ?
If yes- It will through XML parsing errors right ? since the we are splitting the incoming message based some size(So it will be incomplete/invalid XML).
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Oct 04, 2017 11:47 pm Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

You don't have to parse it immediately. In fact you don't have to parse it at all unless you want to change the contents in some way.

If you are crazy enough to have a single XML message more than 100 MB than you could accumulate it somewhere as a BLOB until complete.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
MD
PostPosted: Thu Oct 05, 2017 1:11 am Post subject: Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

So lets take below example..

The actual message(XML) is 10MB size
I split this message as 1MB each using MQFT and grouping.

When I put this message in queue with MQFT command, each message will be processed separately(10 times) or MQ-Input node will take all of the 10 messages as single message(10MB) and process once ?
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Oct 05, 2017 2:57 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

First of all - why not try it with a small sample flow?

Secondly, I already said

This means that when the group is complete on the input queue, then they will get passed to the flow one at a time. If you want to detect the last message in the group you can examine the MQMD flags.

"one at a time", means the flow is called passing one message at a time (10 messages equals 10 flow invocations).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
MD
PostPosted: Thu Oct 05, 2017 3:45 am Post subject: Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Quote:
First of all - why not try it with a small sample flow?

Yes I did it and I got the same as you said(Messages will be processed separately).

So just for confirmation is there any other way we can process all the messages in single take

Thanks zpat and all for the responses
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Oct 05, 2017 4:13 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

You seem determined to ignore advice.

Just because the messages are presented to the flow one at a time does not mean that you can't concatenate (append) the content into a shared variable or environment variable, or into a file and so on.

When the last message has been added (look at the MQMD flag to detect "last message in group") then you can do your thing.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
MD
PostPosted: Thu Oct 05, 2017 8:02 pm Post subject: Reply with quote

Novice

Joined: 19 Apr 2017
Posts: 11

Okay..

Can help me how to do that concatenation when last message found in the queue.

In my POC I just placed MQ Input node, MQ Output node.
So when I put the 10MB message, 10 messages going to Output node.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Oct 05, 2017 11:41 pm Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

Clearly, you would need to append each (and every) message, as well as the last one.

Some transports can append records (like the file nodes) and you can call the finalise at the end (after the last message).

Others you would not use the output node each time, but only after the last message.

Having given you the general idea, the rest is up to your own initiative and reading of the documentation.

As Jeff says "think, try, repeat".

As with most IIB development, this is not really ideal for an unsupervised beginner. Ask your SME to help.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportHandling large message in IIB
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.