| Author |
Message
|
| jrsdm |
 Posted: Sun Oct 28, 2012 3:21 am Post subject: Trigger Fileinput node in message broker |
 |
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
Hi,
There is batch process in which will trigger the message flow which contain the file input node (Which pull the file from the remote directory).
The batch process is only trigger when it see the file is ready to Pickup, so what it will do it will put triggering message in the queue.
So my problem is
1. how I can trigger the file input node at that time so that it will start picking it up the file and rest of the time not poll.
2.After everything is done it stop polling.
It only poll the directory when message flow is trigger.
Looking forward for your suggestion. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Sun Oct 28, 2012 4:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Edit.
To trigger a file input node to process a file, on unix, you would use a command called 'mv'.
To cause a FileInput node to *not* poll, you will *not* use a FileInput node. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Oct 28, 2012 4:47 am Post subject: Re: Trigger Fileinput node in message broker |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jrsdm wrote: |
So my problem is
1. how I can trigger the file input node at that time so that it will start picking it up the file and rest of the time not poll.
2.After everything is done it stop polling.
|
My question is why do you want to do this? What's wrong with the flow polling away waiting for a file? As my most worthy associate says, if all you're trying to do is stop the flow picking the file up before it's complete, write it to a different directory and use your "trigger" process to move the file to the polled directory.
| jrsdm wrote: |
| Looking forward for your suggestion. |
If you're determined to stop the polling, have the flow is a stopped state. Use the trigger process to issue an administrative command to start the flow, then have the flow issue another administrative command to stop itself once file processing is complete.
This of course opens a number of issues, like a script with administrative control over the broker & the security issues that exposes, the various points at which this can all break due to infrastructure problems (e.g. the batch process to trigger the flow not being able to run), the resource costs of starting and stopping the flow, etc, etc.
This is why most people just move the file when it's ready.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jrsdm |
| Posted: Sun Oct 28, 2012 10:47 am Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
Hi , Thanks for the information,
As this is the batch process a,it will not run everyday , It will run once a week.so Don't want to poll always to that directory.
The batch process will trigger the message flow and then it will start to process the file.
Thanks |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Oct 28, 2012 11:05 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jrsdm wrote: |
| Don't want to poll always to that directory. |
Again, why not?
| jrsdm wrote: |
| The batch process will trigger the message flow and then it will start to process the file. |
The process can't "trigger" the flow; only the input node can do that. If the process is starting the flow, you'll need another process to stop it or it will be polling like you don't want.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jrsdm |
| Posted: Sun Oct 28, 2012 1:48 pm Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
Guys as per your suggestion ,I will come up with this solutioning.
1. Scheduler trigger the flow with the filename/server_name/.
2. The message will passes will trigger the fileinput node
Here
Schedulermessage-> mqinput node ->Fileinput node-> Mqoutput
Mqinput ->Compute->fileoutput node.
As My file is in the remote directory which is of 6 GB
What will be the best approach
1)Do I need to right a script which get the files from the remote server
Or
the fileinput node do the job of sFTP
Please advice. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Sun Oct 28, 2012 10:56 pm Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi jrsdm,
6 G file is quite a big message for broker to handle in one piece. Can you split it somehow to smaller pieces? Perhaps even on the remote end before broker's actions?
--
Marko |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 29, 2012 1:16 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you process the file record by record - it will not all be in memory at the same time.
You can change the scan interval of the file input node. Certainly if the file arrives once a week, make the scan interval something like hourly (or at least the max allowed if less than that). |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 29, 2012 5:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jrsdm wrote: |
| 1. Scheduler trigger the flow with the filename/server_name/. |
How do you plan to trigger a flow starting with a FileInput node?
| jrsdm wrote: |
2. The message will passes will trigger the fileinput nodeWhat will be the best approach
1)Do I need to right a script which get the files from the remote server
Or
the fileinput node do the job of sFTP |
I don't see why you wouldn't use sFTP through the FileInput node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jrsdm |
| Posted: Mon Oct 29, 2012 4:22 pm Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
Hi ,
1. Why I dont want to use the functionality of fileinput node to sftp because it is remote directory out of our network and because of network issue I dont want to the same and done the task of sftp from script.
I need suggestion.
Let take a example , I have one 3 GB xml file in the directory where the file input node is watching.
I want to process the records in the xml file one by one.
My xml file look like this.
<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- Edited by XMLSpy® -->
<CATALOG1>
<CD>
<TITLE>Empire Burlesque</TITLE>
<ARTIST>Bob Dylan</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>Columbia</COMPANY>
<PRICE>10.90</PRICE>
<YEAR>1985</YEAR>
</CD>
<CD>
<TITLE>Unchain my heart</TITLE>
<ARTIST>Joe Cocker</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>EMI</COMPANY>
<PRICE>8.20</PRICE>
<YEAR>1987</YEAR>
</CD>
</CATALOG1>
<CATALOG2>
<CD>
<TITLE>Empire Burlesque</TITLE>
<ARTIST>Bob Dylan</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>Columbia</COMPANY>
<PRICE>10.90</PRICE>
<YEAR>1985</YEAR>
</CD>
<CD>
<TITLE>Hide your heart</TITLE>
<ARTIST>Bonnie Tyler</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS Records</COMPANY>
<PRICE>9.90</PRICE>
<YEAR>1988</YEAR>
</CD>
<CD>
<TITLE>Unchain my heart</TITLE>
<ARTIST>Joe Cocker</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>EMI</COMPANY>
<PRICE>8.20</PRICE>
<YEAR>1987</YEAR>
</CD>
</CATALOG2>
<CATALOG3>
----
</CATALOG3>
.
.
.
.
.
.
N
In this case I need to read the catalog1 as 1 record and catalog2 as second so on.
So what should be the record detection property of the file input node any advice.
3. If I need to split the xml then how we can do that so that it does not cause XML parsing error. |
|
| Back to top |
|
|
| jrsdm |
| Posted: Mon Oct 29, 2012 4:24 pm Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
Hi Marco,
How I can achieve that (split it somehow to smaller pieces)?
Thanks |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 29, 2012 6:05 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jrsdm wrote: |
| 1. Why I dont want to use the functionality of fileinput node to sftp because it is remote directory out of our network and because of network issue I dont want to the same and done the task of sftp from script. |
So how exactly is using sFTP from a script "better" than using sFTP from a FileInput node? Given that the FileInput node will automatically retry in the event of a network failure where you'll have to code your script to do that. Unless you plan to use the script to start your flow once the transfer is complete to prevent the polling you're so dead against.
Which leaves you with a script that has administrative control over your broker and the FileInput node still polling after the file has been processed because nothing is stopping it.
| jrsdm wrote: |
I need suggestion.
Let take a example , I have one 3 GB xml file in the directory where the file input node is watching.
I want to process the records in the xml file one by one.
|
What you need is to look up the advice on handling large messages in the broker InfoCenter. There's even a supplied sample which does exactly what you're describing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jrsdm |
| Posted: Tue Oct 30, 2012 3:35 am Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
The scheduler will schedule the job when it schedule to run, so my plan is to move the sftp the big file to local box and move to the directory where file input node is watching.
There is one more option that has suggested that split the file into smaller, so How we can do that any idea? |
|
| Back to top |
|
|
| jrsdm |
| Posted: Tue Oct 30, 2012 3:37 am Post subject: |
|
|
Apprentice
Joined: 24 Oct 2012
Posts: 27
|
| Thanks I will look into the large message simple. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 30, 2012 4:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jrsdm wrote: |
| The scheduler will schedule the job when it schedule to run, so my plan is to move the sftp the big file to local box and move to the directory where file input node is watching. |
That "watching" is the polling you didn't want to happen originally. I'm glad you've seen sense.
| jrsdm wrote: |
| There is one more option that has suggested that split the file into smaller, so How we can do that any idea? |
The easiest way is to use the functionality of the large message sample or similar.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
