| Author |
Message
|
| cyvwmb |
 Posted: Mon Mar 18, 2013 3:48 am Post subject: Reg : Issue on Collector Node(need help urgently).. |
 |
|
Newbie
Joined: 18 Mar 2013
Posts: 9
|
Hi,
This is my message flow order.
MQINPUT -->CollectorNode-->ComputeNode-->Fileout Node.
Here I am transforming XML to CWF format.....In collector node i given 15 mins for timeout.
i have number of input XML messages(73 input messages) it contains employee details records(Each input file contains 3,05 MB (3.206.157 bytes)) ..
My users needs every employee details records in one file it must contains 30k records. and each xml contains 800 employee records.
For Ex :
Input XML message.
first input xml
<Emp>
<row>
<fname>Bhanu</fname>
<lname>Prakkash</lname>
<comp>xxxx</comp>
</row>
<row>
<fname>sandeep</fname>
<lname>mopur</lname>
<comp>xxxx</comp>
</row>
</Emp>
second input xml
<Emp>
<row>
<fname>aaaaaa</fname>
<lname>bbbbbbb</lname>
<comp>xxxx</comp>
</row>
<row>
<fname>cccccccc</fname>
<lname>ddddddd</lname>
<comp>xxxx</comp>
</row>
</Emp>
File output must comes like this...
H123456 ----Header must be created
Bhanu prakkash xxxxxx
sandeep mopur xxxxxx
aaaaaa bbbbbbb xxxxxx
ccccccc ddddddd xxxxxxx
so here my Doubt is we need to create one Header and rest of whole is details of Employee...
for my logic one XML file is creating but in I need to append 2nd XML file as well....
why i used Collector Node in this flow to collect all input XML message at one time... but 2nd xml message is not appending file out ..
Any one can help me please
javascript:emoticon(' ')javascript:emoticon('')
ThanQ.. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Mon Mar 18, 2013 4:13 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
You are far better off 'collecting' the data in a DB Table than using the collector node especially when pretty well every collection will be output as a result of a collection timeout.
Your requirement is pretty horrible in the first place and using a collector node is going to make it even worse.
Using a dB table means you can make the right decisions about how you package up the data at the right time.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Mar 18, 2013 4:27 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
I agree with smdavies99 in that a Collector node is not the tool to use with large messages.
The best performing alternative would be to read the first message with MQInput and the rest of the messages with MQGet, apply large message processing principles to propagate reasonably sized batches of output records and run the FileOutput node in append mode. But to implement that properly you need to be a *very* experienced Message Broker developer. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 18, 2013 5:02 am Post subject: Re: Reg : Issue on Collector Node(need help urgently).. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cyvwmb wrote: |
| why i used Collector Node in this flow to collect all input XML message at one time... but 2nd xml message is not appending file out .. |
What are you doing to indicate all the XMLs are part of the same collection?
I agree with the other posters; this is contact admin.
Put the data in a dbase and have a separate process convert db to file when all the XML has been processed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 18, 2013 5:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| run the FileOutput node in append mode. |
Assumes the OP is running a version of WMB where the FileOutput node has an append mode.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Mar 18, 2013 5:19 am Post subject: Re: Reg : Issue on Collector Node(need help urgently).. |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Vitor wrote: |
I agree with the other posters; this is contact admin.
|
Yes, it's contact bloody admin  |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 18, 2013 5:40 am Post subject: Re: Reg : Issue on Collector Node(need help urgently).. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| Vitor wrote: |
I agree with the other posters; this is contact admin.
|
Yes, it's contact bloody admin |
Well this new policy is going to seriously tone down the level of my criticism.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| cyvwmb |
| Posted: Mon Mar 18, 2013 11:36 pm Post subject: |
|
|
Newbie
Joined: 18 Mar 2013
Posts: 9
|
Hi all
Thank you to all for replying me..
In FileOut Node i using append mode.
File output must comes like this...
H123456 ----Header must be created(mandatory)
Bhanu prakkash xxxxxx
sandeep mopur xxxxxx
aaaaaa bbbbbbb xxxxxx
ccccccc ddddddd xxxxxxx
Header must be one row for whole cwf output file messages...
If i will take single XML file every time header row will create duplicates...
so Database is not possible... javascript:emoticon(' ') |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Mar 19, 2013 4:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cyvwmb wrote: |
| If i will take single XML file every time header row will create duplicates... |
Not if you only generate the output once all the XML files have been processed, as I and others have suggested. If you're not going to read what we're posting why are you bothering?
| cyvwmb wrote: |
| so Database is not possible... |
Of course it is. But if you believe not, or don't believe this is the best advice then go in peace and good fortune to you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
