Author |
Message
|
nmaddisetti |
Posted: Thu Jan 21, 2010 6:49 am Post subject: Issue with File name or pattern in FileInput node(Urgent) |
|
|
Centurion
Joined: 06 Oct 2004 Posts: 145
|
Hi All,
We have a flow3 with file input node and it is looking for the files *.xml in a folder
Another system is writing the files onto this folder with file name abc.txt-01212010094700.tmp
We have another message flow2 to rename above file from .tmp to .xml
We expect the flow3 to pickup the files once flow2 renames from .tmp to .xml
But some times .tmp files are getting picked up by flow3 this is causing flow3 to fail because of some dependancy of data in database inserted by flow1 which will run prior to flow3 and flow2.
To summarise:
flow1 will insert data into database then triggers flow2 with file name and path for renaming.
flow2 will rename the file from .tmp to .xml
flow3 should pick the files which were renamed to .xml and this flow3 has a dependancy on the data inserted by flow1.
flow3 is picking up the .tmp files and by that time flow1 is not finshed so flow3 is failing.
Note:
1) flow3 file input node is configured to read files from another server using FTP.
2) flow2 is doing rname by connecting to FTP server using java in JCN.
I did not understand why FileInput node picking up files with out matching pattern.
Can you please some one throw some info on this.
Thanks in Advance,
Venkat. |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Jan 21, 2010 7:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Take a user trace of flow 3. |
|
Back to top |
|
 |
zpat |
Posted: Thu Jan 21, 2010 7:02 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Why not use transactional messages (i.e. MQ). Even if the data has to arrive as a file, why not use messages after that?
File processing is meant (in my view) for relatively simple situations. Your design looks inelegant to me.
The fileinput and fileoutput nodes are better for use at the outside edge of the broker, rather than a means of moving data around between message flows.
Files are inherently transactionally unsafe and error prone - their disadvantages are why databases and transactional messaging were invented. Use files only when necessary in the broker (preferably don't use them at all).
This is my personal view if you want high performance, scalable, reliable designs anyway. |
|
Back to top |
|
 |
nmaddisetti |
Posted: Thu Jan 21, 2010 7:22 am Post subject: |
|
|
Centurion
Joined: 06 Oct 2004 Posts: 145
|
Quote: |
Take a user trace of flow 3. |
It is happening in production and we dont see in Test environments and It is not happening all the time so client wont agree to enable trace and leave like that until it happens again.but I will request for that.
Hi zpat,
I agree with you but in our project the xml files size is more than 100 MB and the application writing these files doesnt have capability to write files to queues with segmentation. So I dont have any option rather than using fileinput node. As of now we saw file size of 160 MB and we may see 250 MB also.
And comming to my question why .tmp files are getting picked when the pattern is *.xml.
any thoughts please.
Thanks in Advance,
Venkat. |
|
Back to top |
|
 |
zpat |
Posted: Thu Jan 21, 2010 8:22 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Large files are a sign that a batch mentality has taken hold.
Unless you have single XML objects > 100 MB, you can send more smaller messages.
WMB works better with smaller, more atomic transactions. It's not a bulk file movement product!
Rather than focus on fixing this problem, I would suggest re-designing this thing end to end to be more transactional, atomic and event driven.
Maybe the first flow should break up the large file into many individual messages for further processing. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Jan 21, 2010 8:28 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
nmaddisetti wrote: |
And comming to my question why .tmp files are getting picked when the pattern is *.xml. |
Without a user trace as previously suggested, it's hard to be definitive but it sounds like something's going wrong in the database trigger / ftp scenario so flow 3's going off too soon.
Moving to the larger point, if you're handling 160Mb XML documents, expecting it to go to 250Mb and (if I'm any judge) just continuing to grow then I hope you've got good code in WMB and it's tuned perfectly. That's a lot of data to have in a single message tree. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
nmaddisetti |
Posted: Thu Jan 21, 2010 9:26 am Post subject: |
|
|
Centurion
Joined: 06 Oct 2004 Posts: 145
|
Hi Zpat,
It is a single large xml object. We cant divide it to small chunks.
Hi Vitor,
Can you please give more details on:database trigger / ftp scenario so flow 3's going off too soon.
I did not understand.
Is It something to do with FTP scan dealy?
Thanks in Advance,
Venkat. |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Jan 21, 2010 9:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
FTP does not provide reliable file locking, or in fact any file locking at all.
Your best bet for ensuring that this whole interlocked process is reliable is to move the flow to a local directory as step 1. Then trigger flow 2 from that directory, and have flow 2 write the file to a second local directory, where it will be read by flow 3. As long as flow 2 and flow 3 are in the same EG, they will not contend and flow 3 won't read the file until flow2 is done with it. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Jan 21, 2010 9:39 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
nmaddisetti wrote: |
It is a single large xml object. We cant divide it to small chunks. |
Then my comments about how you've got WMB set up become twice as important. If you have to manipulate (in future times) as a 250+Mb solid block of data then things are going to get interesting.
nmaddisetti wrote: |
Can you please give more details on:database trigger / ftp scenario so flow 3's going off too soon.
I did not understand.
Is It something to do with FTP scan dealy? |
Well that's a place to start looking. Clearly this long and error-prone method of getting a file has developed an error and things are being triggered in the wrong sequence. Without a user trace I'm guessing, but when you say your flow3 is reading the tmp file, do you mean it's reading the tmp file or the results indicate it's reading the tmp file? Because if the tmp file is being renamed to xml too soon, this would yield that effect.
Another good reason for not using files, and not using huge ones that take ages to be available on the file system. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
nmaddisetti |
Posted: Thu Jan 21, 2010 10:55 am Post subject: |
|
|
Centurion
Joined: 06 Oct 2004 Posts: 145
|
Hi Jeff,
To avoid file locking issue only we are writing file as .tmp and by the time rename flow is triggered to rename the file .tmp file is 100% good file. And flows with additional instances were running in single execution group only.
Even if I change the design to bring the file to local directory as long as I use WMB the current issue or file locking cant be eliminated. May be our design should include MQ File Transfer in the picture ( Will it solve? )
Hi Vitor,
As of now only one time 160 MB and most of the time file size will be 20,30,40 MB and there is chance of large files once next phase of code come into picture.
It is reading the .tmp file and failing
the rename of the file is happening in about 4 sec this was observed based on the logs.
The failure of the flow3 is logged in DB at time T ( systimestamp inserted into table from Oracle box)
And rename flow which is flow2 showing the log as successfully renamed after T+4 sec( broker CURRENT_TIMESTAMP in trace file).
Actually action on failure is Delete so as per the logs observation file was not delted for about 4 sec after the failure.
Broker and Orable box are in sync with same timezone.
Currently FTP scan delay is 5 sec in the running flow. Should I put it back to 60 which is default.
Any guess based on the above information before I get trace to post. |
|
Back to top |
|
 |
zpat |
Posted: Thu Jan 21, 2010 11:03 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Rather than swallowing spiders to catch flies (etc) why not using MQ message grouping, or a database table to hold this amount of data?
Databases are transactionally safe, that's what you want - don't try to do it with files and FTP, it's just wrong. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Jan 21, 2010 11:11 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
zpat wrote: |
Databases are transactionally safe, that's what you want - don't try to do it with files and FTP, it's just wrong. |
Or push back on the application people and get the ending application to put the data in a message on a queue. Ideally in smaller chunks.
I'm forced to wonder somewhat rhetorically if this thing has an actual business justification for producing one single large file, or if it's just always done that? Does all that data really need to be together because there's huge affinity or is it just inertia? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
zpat |
Posted: Fri Jan 22, 2010 3:45 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Developers need to be willing to challenge designs. It can seem harder or even impolite to question the "requirement".
However the real requirement is always to implement designs that you can really trust to be fit for purpose. The sooner this is done the better. |
|
Back to top |
|
 |
smdavies99 |
Posted: Fri Jan 22, 2010 3:59 am Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
zpat wrote: |
Developers need to be willing to challenge designs. It can seem harder or even impolite to question the "requirement".
|
Or even culturally improper to question 'ones betters'. _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
Vitor |
Posted: Fri Jan 22, 2010 7:27 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
smdavies99 wrote: |
Or even culturally improper to question 'ones betters'. |
Thankfully this last statement so seldom applies to management...  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
|