| Author |
Message
|
| matuwe |
 Posted: Fri Feb 19, 2010 5:19 am Post subject: Dynamic runtime FileInput |
 |
|
Master
Joined: 05 Dec 2007
Posts: 296
|
Hi,
I have 50 directories that i need to read and process the file. Is there anyway that I can create the fileInput directory, dynamically by values received from database, or is it better to read the database using compute and read the file directories in a java compute node.
This will also apply to MQ Input. In total I have 200 Input and I do not want to create/ end up with 200 flows. |
|
| Back to top |
|
 |
| matuwe |
| Posted: Fri Feb 19, 2010 5:34 am Post subject: |
|
|
Master
Joined: 05 Dec 2007
Posts: 296
|
I am hoping to read all my input sources from database e.g
If it is file source, read file from directory , wrap the file into xml “NO VALIDATION”
<xml>
<filename>
<filepattern>
<SourceDir>
<Data>Original file data</Data>
</xml>
And if it is Q
<xml>
<QNAME>
<QMGR>
<Data>Original data from Q</Data>
</xml>
Then send this to a Q. I need to make this dynamic at runtime
Thanks
-D |
|
| Back to top |
|
|
| Herbert |
| Posted: Fri Feb 19, 2010 5:51 am Post subject: |
|
|
Centurion
Joined: 05 Dec 2006
Posts: 146
Location: Leersum, The Netherlands
|
| matuwe wrote: |
<xml>
<QNAME>
<QMGR>
<Data>Original data from Q</Data>
</xml>
Then send this to a Q. I need to make this dynamic at runtime |
AFAIK this is not possible with MQInput, I did have the same problem in the past ( with WBIMB 5.0 I think ), N departments from N countries where sending N message-types. Every N*N*N combination has it's own queue. The processing in the broker was the same. My wish that the MQ boys did solve this at MQ level and did give mu just one queue to read from did not make it. I ended up with a flow with a lot of MQInput nodes, visual not a nice solution, but it did work. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 19, 2010 7:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can use an MQGet node to read from any queue you want and make a decision about which queue at runtime.
There is no FileGet node. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Feb 19, 2010 8:14 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You can have multiple input nodes in a single flow, however this will create a run time instance for each one.
As always - try to avoid files (in favour of messages).
You can use alias queues to re-direct different queue names to a common queue. |
|
| Back to top |
|
|
| matuwe |
| Posted: Mon Feb 22, 2010 11:20 pm Post subject: |
|
|
Master
Joined: 05 Dec 2007
Posts: 296
|
I think I have made it wok, even though I might have compromised perfomance a litlle.
I have a flow that has Timer >> Compute Node >> Java Node > MQOut Node
The compute node reads all the directories from the database into cache Shared row. The Java Node, read the Array of directories and checks if there are any files, if there are creates an XML with file details and data into a common Q.
I just made another flow same as this one, using the MQ Get to danamically read different Q's and also pass the data into a common Q. It works but I am now stuck with trying to find the original message MQMD MQ header, after the MQ get  .
This flow is timer >> compute node >> MQ Get >> compute >> MQOUT.
 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2010 4:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matuwe wrote: |
| I think I have made it wok, even though I might have compromised perfomance a litlle. |
You think? 
You've also made a rod for your own back.
This is the sort of design requirement that needs to be pushed back. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| matuwe |
| Posted: Tue Feb 23, 2010 5:46 am Post subject: |
|
|
Master
Joined: 05 Dec 2007
Posts: 296
|
I am confused....
Please help.. How do I do this a sdynamic so that when i deploy to different countries, with different number of file directories and different no of MQ input.. I imagened this to be a central database table that is maintaned to add all file system to be scanned, and automatically my broker will poll what ever is on the database... Some messages do not even need to be transformed, just in----out and audit trail.
I am really sorry as I am new to Design |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Feb 23, 2010 6:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Why not code a flow with multiple fileinput nodes?
If you are using your own code to do something provided by the WMB product then either (a) IBM have their product design wrong or (b) you have your solution design wrong.
Which is more likely?
You can use bar overides to change the MQ queue names and file directories as needed.
I wouldn't like to see 200 flows, or one flow with 200 input nodes but something in between might be manageable such as 20 flows with 10 input nodes each.
Avoid your own Java code i/o. It will be a support nightmare longer term.
IBM have really opened a can of worms with this Java support when combined with inexperience of the WMB product.
Actually a fileget node would be useful in certain situations. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2010 7:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matuwe wrote: |
| How do I do this a sdynamic so that when i deploy to different countries, with different number of file directories and different no of MQ input |
Well you think seriously about how you handle the data & accept that if you have something very flexible it needs a lot of work to keep it straight. 
| matuwe wrote: |
| I imagened this to be a central database table that is maintaned to add all file system to be scanned, and automatically my broker will poll what ever is on the database |
Well yes but you've just moved the problem. Now you have to maintain either a database that's accessable (reliably) from every county or a separate database in each locality. In either event it has to be kept up to date with all the different directories and queues that you're using, a process hard to automate and bound to go wrong sooner or later.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fatima |
| Posted: Wed Feb 24, 2010 12:19 am Post subject: Dynamic runtime FileInput |
|
|
Newbie
Joined: 30 Sep 2009
Posts: 7
|
| What I hear the guys saying is... Keep the design as simple as possible. Ensure that the code is maintainable going forward. Very Important, always remember to Keep It Simple. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Feb 24, 2010 12:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
It takes more skill to find a simpler solution and if often means uncovering the genuine requirements that have become confused with a possible implementation.
We used to call it lateral thinking, there is always a better way and it doesn't have to take longer either. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Sun Feb 28, 2010 11:27 pm Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
I prefer "no code" version.... so "many" input nodes will work for files....and for queues the MQGet will do the work. I'd rather update a flow input node than introducing another failure point in a DB.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
|
|
