| Author |
Message
|
| MD |
 Posted: Wed Oct 04, 2017 3:27 am Post subject: Handling large message in IIB |
 |
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
Hi All,
I have the requirement like below..
I have to develop a flow in IIB.
My input message is XML and it can be upto 100 MB.
I need to transfer this XML message to Flatfile(CSV) and need to be send it to target application as single message.
Is this doable in IIB ?
Please suggest some possible scenarios.
Thanks in advance.
MD |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Oct 04, 2017 4:52 am Post subject: Re: Handling large message in IIB |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MD wrote: |
| I need to transfer this XML message to Flatfile(CSV) and need to be send it to target application as single message. |
The target wants 100Mb of data as a single comma delimited record. Seriously? Whoever designed that needs professional help; both in terms of program design and mental health.
| MD wrote: |
| Is this doable in IIB ? |
Yes, it's just extremely odd.
| MD wrote: |
| Please suggest some possible scenarios. |
There's advice in the InfoCenter for handling large files / messages.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 04, 2017 5:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
As an IT professional - recommend a more sensible approach, one that makes implementation easier and doesn't impose limits.
Assuming you do mean a MQ message, using 100 MB in MQ often needs changes to channels, xmit queues etc for the max message length.
What happens when 100 MB is exceeded? You are better off designing a solution that uses multiple messages (e.g. as a message group), which means you can keep each one smaller and avoid any upper size limit.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| MD |
| Posted: Wed Oct 04, 2017 7:53 pm Post subject: |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
Hi zpat,
Thanks for the reply.
Even we are thinking to use MQFT and split the message(Grouping) and send.
If we use Grouping, Broker MQ-Input node wait until it receives the last message and then it take all the grouped messages as single message and process right ?
So can Broker handle this ?
Since it is XML, will any parsing errors come?
Thanks  |
|
| Back to top |
|
|
| MD |
| Posted: Wed Oct 04, 2017 8:14 pm Post subject: Re: Handling large message in IIB |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
| Quote: |
| The target wants 100Mb of data as a single comma delimited record. Seriously? |
If the Broker can handle 100 MB message at the incoming side and in transformation, it can be send as multiple records at output(Each record as one CSV file).
Any other advises are also welcome
Thanks |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 04, 2017 11:14 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, broker can handle grouped messages.
Tick the box on the MQinput node ("All messages available").
This means that when the group is complete on the input queue, then they will get passed to the flow one at a time. If you want to detect the last message in the group you can examine the MQMD flags.
Incomplete groups will remain on the queue, so you might want to use a (long) expiry value to avoid indefinite orphaning.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| MD |
| Posted: Wed Oct 04, 2017 11:27 pm Post subject: |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
| Quote: |
| Tick the box on the MQinput node ("All messages available"). |
I have done it.
| Quote: |
| This means that when the group is complete on the input queue, then they will get passed to the flow one at a time. |
Means each message will be processed separately ?
If yes- It will through XML parsing errors right ? since the we are splitting the incoming message based some size(So it will be incomplete/invalid XML). |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 04, 2017 11:47 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You don't have to parse it immediately. In fact you don't have to parse it at all unless you want to change the contents in some way.
If you are crazy enough to have a single XML message more than 100 MB than you could accumulate it somewhere as a BLOB until complete.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| MD |
| Posted: Thu Oct 05, 2017 1:11 am Post subject: |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
So lets take below example..
The actual message(XML) is 10MB size
I split this message as 1MB each using MQFT and grouping.
When I put this message in queue with MQFT command, each message will be processed separately(10 times) or MQ-Input node will take all of the 10 messages as single message(10MB) and process once ? |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 05, 2017 2:57 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
First of all - why not try it with a small sample flow?
Secondly, I already said
This means that when the group is complete on the input queue, then they will get passed to the flow one at a time. If you want to detect the last message in the group you can examine the MQMD flags.
"one at a time", means the flow is called passing one message at a time (10 messages equals 10 flow invocations).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| MD |
| Posted: Thu Oct 05, 2017 3:45 am Post subject: |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
| Quote: |
| First of all - why not try it with a small sample flow? |
Yes I did it and I got the same as you said(Messages will be processed separately).
So just for confirmation is there any other way we can process all the messages in single take
Thanks zpat and all for the responses |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 05, 2017 4:13 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You seem determined to ignore advice.
Just because the messages are presented to the flow one at a time does not mean that you can't concatenate (append) the content into a shared variable or environment variable, or into a file and so on.
When the last message has been added (look at the MQMD flag to detect "last message in group") then you can do your thing.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| MD |
| Posted: Thu Oct 05, 2017 8:02 pm Post subject: |
|
|
Novice
Joined: 19 Apr 2017
Posts: 15
|
Okay..
Can help me how to do that concatenation when last message found in the queue.
In my POC I just placed MQ Input node, MQ Output node.
So when I put the 10MB message, 10 messages going to Output node. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 05, 2017 11:41 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Clearly, you would need to append each (and every) message, as well as the last one.
Some transports can append records (like the file nodes) and you can call the finalise at the end (after the last message).
Others you would not use the output node each time, but only after the last message.
Having given you the general idea, the rest is up to your own initiative and reading of the documentation.
As Jeff says "think, try, repeat".
As with most IIB development, this is not really ideal for an unsupervised beginner. Ask your SME to help.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
