| Author |
Message
|
| swethabala |
 Posted: Fri Feb 12, 2010 2:31 am Post subject: File input node properties |
 |
|
Apprentice
Joined: 25 Aug 2009
Posts: 44
|
Hi,
In one of our message flows File input node is acting very strange. The node is picking the same file multiple times.
There is ETM Emitter node to catch the event and the event is looping multiple times.
Could you please let me know how File input node work incase of huge files. if something doesnt get pick in 2 minutes time or so it will pick the same file again?
Please let me know this is an urgent issue in our project.
Thanks in advance
Swetha |
|
| Back to top |
|
 |
| Gaya3 |
| Posted: Fri Feb 12, 2010 4:01 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
1. you have give proper file pattern in the file input node
2. you can increase or decrese the polling delay of file input node.
I never come across the file input doing something like this... 
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 12, 2010 4:20 am Post subject: Re: File input node properties |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| swethabala wrote: |
| Could you please let me know how File input node work incase of huge files. if something doesnt get pick in 2 minutes time or so it will pick the same file again? |
It will pick the file up as soon the poll succeeds for a file with a name that matches appears in the pattern in the node. So if you're sending a very large file and the OS creates a file with that name without properly locking it (I'm thinking of Windows specifically here where I've seen this) then the FileInput node will see the partially completed file, successfully open it, process the contents, attempt to move or delete it (as configured), fail because the file is open & still being written to then go off on it's polling once more. At which point it finds the file again and goes round for another go until the file is complete and the node can move or delete it.
Your best bet is to create file REALINPUT as NOTREALINPUT, and rename it once you're certain the full file has been written (e.g. the FTP has completed) and ensure the FileInput pattern only matches REAL*.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| swethabala |
| Posted: Fri Feb 12, 2010 5:34 am Post subject: |
|
|
Apprentice
Joined: 25 Aug 2009
Posts: 44
|
Vitor,
Thanks for your inputs... we are deleting the file manually for now when it got read in WMB. but this is not the perminent solution right..
As per your inputs if we change name also it will be temporary for only that test. In production we can not do that. Please let me know if we can configure anything in File Input node "not to pick the same data". or something like longer time interval to pick the file. Is changing the polling interval is enough for extending the time interval?
Once again thanks for your inputs. they were useful |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Fri Feb 12, 2010 6:07 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
will you please post the properties of file input node over here....
input directory
File name or pattern
Action on Successful processing
Polling
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 12, 2010 6:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| swethabala wrote: |
| Please let me know if we can configure anything in File Input node "not to pick the same data". |
No, because the node can't tell it's the same data. It avoids reading the same file twice by moving or deleting it once it's done. This fails in the circumstance I describe above.
| swethabala wrote: |
| Is changing the polling interval is enough for extending the time interval? |
Yes, but you need to pick a time interval that's always going to be long enough no matter how large the file but short enough that the file isn't sitting around waiting to be processed (or overwritten by the next file).
Good luck with that. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Feb 12, 2010 6:55 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Don't use files for business application integration that must have transactional integrity.
Use MQ messages, that's why IBM developed MQ.
Generate messages right from the start - don't use files for any part of the journey - only MQ and databases (preferably with 2 phase commit).
Break up large amounts of data using grouped or segemented data (if it is truly one item which I doubt).
Adding file transport to WMB, is unfortunately leading to a new generation of transactionally unsafe applications.
How many people truly understand the classic ACID principles these days I wonder? |
|
| Back to top |
|
|
| swethabala |
| Posted: Sun Feb 14, 2010 10:29 pm Post subject: |
|
|
Apprentice
Joined: 25 Aug 2009
Posts: 44
|
Hi Jedi,
You are right, we should use MQ Input for WMB integration. However our client is using File transfer only for this interface. we are not left with any other option except making this interface run.
I have made a change in the code and parser properties and testing it now. I am getting an exception at parser as bolow:
"000|5426|E|000|00|{[2488]Error detected, rethrowing}.{[5285]ImbRecoverableException caught from worker->parseNext.(MSP_SAPTN_Forecast;TDS;/BS_CNH_CRF_Forecast/RecordData/EOR)}.{[5421]TDS General Error(/BS_CNH_CRF_Forecast/RecordData(6343 of unbounded)/PlantCode;317100)}.{[5426]Not enough data in bitstream}.#"
Could you please let me know why this kind of error is appearing? |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Sun Feb 14, 2010 10:33 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi swethabala,
| swethabala wrote: |
"000|5426|E|000|00|{[2488]Error detected, rethrowing}.{[5285]ImbRecoverableException caught from worker->parseNext.(MSP_SAPTN_Forecast;TDS;/BS_CNH_CRF_Forecast/RecordData/EOR)}.{[5421]TDS General Error(/BS_CNH_CRF_Forecast/RecordData(6343 of unbounded)/PlantCode;317100)}.{[5426]Not enough data in bitstream}.#"
Could you please let me know why this kind of error is appearing? |
Lesser number of characters in the record than what the message defn requires.
Regards. |
|
| Back to top |
|
|
| swethabala |
| Posted: Mon Feb 15, 2010 12:13 am Post subject: |
|
|
Apprentice
Joined: 25 Aug 2009
Posts: 44
|
Thank you, the exception is rectified now. However there is another exception "{[2652]Dequeued failed message. Propagating a message to the failure terminal}."
Could you please let me know in which case this exception will come? |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Mon Feb 15, 2010 1:15 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
|
| Back to top |
|
|
| zpat |
| Posted: Mon Feb 15, 2010 1:26 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| swethabala wrote: |
Hi Jedi,
You are right, we should use MQ Input for WMB integration. However our client is using File transfer only for this interface. we are not left with any other option except making this interface run.
|
If your client is using FTP internally - they do have a choice to use MQ and you should strongly recommend it, not just live with a bad design.
It is your job as an IT professional to get the optimum design.
Even if the FTP is external, you can ask - sometimes both parties use MQ and just don't communicate the fact. |
|
| Back to top |
|
|
| swethabala |
| Posted: Mon Feb 15, 2010 1:45 am Post subject: |
|
|
Apprentice
Joined: 25 Aug 2009
Posts: 44
|
Hi Friends,
Thanks for your suggestions.
I have found the issue. The actual issue was in the processing time in functional monitoring fields. We have an emitter node for tracking functional monitoring events.
This node was taking lot of time to extract the data and meanwhile File input is catching the same input file again. I have rectified this problem by disabling some unimported monitoring fields. Now the input is getting processed successfully and its not getting in to loop.
As per your suggestion I will advice my onshore team about changing the File input to MQ input to avoid this kind of problems in future. |
|
| Back to top |
|
|
|
|
