ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Issue with File name or pattern in FileInput node(Urgent)

Post new topic  Reply to topic
 Issue with File name or pattern in FileInput node(Urgent) « View previous topic :: View next topic » 
Author Message
nmaddisetti
PostPosted: Thu Jan 21, 2010 6:49 am    Post subject: Issue with File name or pattern in FileInput node(Urgent) Reply with quote

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi All,

We have a flow3 with file input node and it is looking for the files *.xml in a folder

Another system is writing the files onto this folder with file name abc.txt-01212010094700.tmp

We have another message flow2 to rename above file from .tmp to .xml

We expect the flow3 to pickup the files once flow2 renames from .tmp to .xml

But some times .tmp files are getting picked up by flow3 this is causing flow3 to fail because of some dependancy of data in database inserted by flow1 which will run prior to flow3 and flow2.

To summarise:

flow1 will insert data into database then triggers flow2 with file name and path for renaming.
flow2 will rename the file from .tmp to .xml
flow3 should pick the files which were renamed to .xml and this flow3 has a dependancy on the data inserted by flow1.

flow3 is picking up the .tmp files and by that time flow1 is not finshed so flow3 is failing.

Note:
1) flow3 file input node is configured to read files from another server using FTP.
2) flow2 is doing rname by connecting to FTP server using java in JCN.

I did not understand why FileInput node picking up files with out matching pattern.
Can you please some one throw some info on this.

Thanks in Advance,
Venkat.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Jan 21, 2010 7:01 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Take a user trace of flow 3.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Jan 21, 2010 7:02 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Why not use transactional messages (i.e. MQ). Even if the data has to arrive as a file, why not use messages after that?

File processing is meant (in my view) for relatively simple situations. Your design looks inelegant to me.

The fileinput and fileoutput nodes are better for use at the outside edge of the broker, rather than a means of moving data around between message flows.

Files are inherently transactionally unsafe and error prone - their disadvantages are why databases and transactional messaging were invented. Use files only when necessary in the broker (preferably don't use them at all).

This is my personal view if you want high performance, scalable, reliable designs anyway.
Back to top
View user's profile Send private message
nmaddisetti
PostPosted: Thu Jan 21, 2010 7:22 am    Post subject: Reply with quote

Centurion

Joined: 06 Oct 2004
Posts: 145

Quote:
Take a user trace of flow 3.


It is happening in production and we dont see in Test environments and It is not happening all the time so client wont agree to enable trace and leave like that until it happens again.but I will request for that.



Hi zpat,

I agree with you but in our project the xml files size is more than 100 MB and the application writing these files doesnt have capability to write files to queues with segmentation. So I dont have any option rather than using fileinput node. As of now we saw file size of 160 MB and we may see 250 MB also.

And comming to my question why .tmp files are getting picked when the pattern is *.xml.

any thoughts please.

Thanks in Advance,
Venkat.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Jan 21, 2010 8:22 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Large files are a sign that a batch mentality has taken hold.

Unless you have single XML objects > 100 MB, you can send more smaller messages.

WMB works better with smaller, more atomic transactions. It's not a bulk file movement product!

Rather than focus on fixing this problem, I would suggest re-designing this thing end to end to be more transactional, atomic and event driven.

Maybe the first flow should break up the large file into many individual messages for further processing.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 21, 2010 8:28 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

nmaddisetti wrote:
And comming to my question why .tmp files are getting picked when the pattern is *.xml.


Without a user trace as previously suggested, it's hard to be definitive but it sounds like something's going wrong in the database trigger / ftp scenario so flow 3's going off too soon.

Moving to the larger point, if you're handling 160Mb XML documents, expecting it to go to 250Mb and (if I'm any judge) just continuing to grow then I hope you've got good code in WMB and it's tuned perfectly. That's a lot of data to have in a single message tree.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
nmaddisetti
PostPosted: Thu Jan 21, 2010 9:26 am    Post subject: Reply with quote

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi Zpat,

It is a single large xml object. We cant divide it to small chunks.

Hi Vitor,

Can you please give more details on:database trigger / ftp scenario so flow 3's going off too soon.
I did not understand.
Is It something to do with FTP scan dealy?

Thanks in Advance,
Venkat.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Jan 21, 2010 9:38 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

FTP does not provide reliable file locking, or in fact any file locking at all.

Your best bet for ensuring that this whole interlocked process is reliable is to move the flow to a local directory as step 1. Then trigger flow 2 from that directory, and have flow 2 write the file to a second local directory, where it will be read by flow 3. As long as flow 2 and flow 3 are in the same EG, they will not contend and flow 3 won't read the file until flow2 is done with it.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 21, 2010 9:39 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

nmaddisetti wrote:
It is a single large xml object. We cant divide it to small chunks.


Then my comments about how you've got WMB set up become twice as important. If you have to manipulate (in future times) as a 250+Mb solid block of data then things are going to get interesting.

nmaddisetti wrote:
Can you please give more details on:database trigger / ftp scenario so flow 3's going off too soon.
I did not understand.
Is It something to do with FTP scan dealy?


Well that's a place to start looking. Clearly this long and error-prone method of getting a file has developed an error and things are being triggered in the wrong sequence. Without a user trace I'm guessing, but when you say your flow3 is reading the tmp file, do you mean it's reading the tmp file or the results indicate it's reading the tmp file? Because if the tmp file is being renamed to xml too soon, this would yield that effect.

Another good reason for not using files, and not using huge ones that take ages to be available on the file system.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
nmaddisetti
PostPosted: Thu Jan 21, 2010 10:55 am    Post subject: Reply with quote

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi Jeff,

To avoid file locking issue only we are writing file as .tmp and by the time rename flow is triggered to rename the file .tmp file is 100% good file. And flows with additional instances were running in single execution group only.

Even if I change the design to bring the file to local directory as long as I use WMB the current issue or file locking cant be eliminated. May be our design should include MQ File Transfer in the picture ( Will it solve? )

Hi Vitor,

As of now only one time 160 MB and most of the time file size will be 20,30,40 MB and there is chance of large files once next phase of code come into picture.

It is reading the .tmp file and failing
the rename of the file is happening in about 4 sec this was observed based on the logs.

The failure of the flow3 is logged in DB at time T ( systimestamp inserted into table from Oracle box)
And rename flow which is flow2 showing the log as successfully renamed after T+4 sec( broker CURRENT_TIMESTAMP in trace file).
Actually action on failure is Delete so as per the logs observation file was not delted for about 4 sec after the failure.
Broker and Orable box are in sync with same timezone.

Currently FTP scan delay is 5 sec in the running flow. Should I put it back to 60 which is default.

Any guess based on the above information before I get trace to post.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Jan 21, 2010 11:03 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Rather than swallowing spiders to catch flies (etc) why not using MQ message grouping, or a database table to hold this amount of data?

Databases are transactionally safe, that's what you want - don't try to do it with files and FTP, it's just wrong.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 21, 2010 11:11 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

zpat wrote:
Databases are transactionally safe, that's what you want - don't try to do it with files and FTP, it's just wrong.


Or push back on the application people and get the ending application to put the data in a message on a queue. Ideally in smaller chunks.

I'm forced to wonder somewhat rhetorically if this thing has an actual business justification for producing one single large file, or if it's just always done that? Does all that data really need to be together because there's huge affinity or is it just inertia?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Fri Jan 22, 2010 3:45 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Developers need to be willing to challenge designs. It can seem harder or even impolite to question the "requirement".

However the real requirement is always to implement designs that you can really trust to be fit for purpose. The sooner this is done the better.
Back to top
View user's profile Send private message
smdavies99
PostPosted: Fri Jan 22, 2010 3:59 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

zpat wrote:
Developers need to be willing to challenge designs. It can seem harder or even impolite to question the "requirement".


Or even culturally improper to question 'ones betters'.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Jan 22, 2010 7:27 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

smdavies99 wrote:
Or even culturally improper to question 'ones betters'.


Thankfully this last statement so seldom applies to management...
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Issue with File name or pattern in FileInput node(Urgent)
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.