| Author |
Message
|
| pritch75 |
 Posted: Wed Jul 02, 2008 8:16 am Post subject: FileInput node and Clustering |
 |
|
Newbie
Joined: 02 Jul 2008
Posts: 3
|
Does the FileInput node work in a clustered environment?
We're using FileInput as input to a flow that is deployed to a clustered Broker. The node polls on an FTP server for a particular file mask. Since the files can be rather large and they are not transferred instantaneously, it appears that the FileInput nodes from both servers attempt to pull down the file and process it. This is creating duplicate processing for us. Is there something that we need to do for the FileInput node in a clustered environment? |
|
| Back to top |
|
 |
| mqpaul |
| Posted: Mon Jul 07, 2008 2:30 am Post subject: The non-FTP file input node can work OK clustered |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
The file input node uses OS file locks to prevent multiple execution groups accessing the same file simultaneously. This works OK for files accessed through the OS file system API. But it may not work for remote files accessed through FTP, since it's down to the FTP server to lock files. As far as I know, the file node can't lock the FTP files.
Question: Do your different clustered file input nodes share the same input directory (that's the local one, not the FTP server's directory)? If they do, the file input node should lock the target file for the FTP transfer. You might get a BIP3385 exception from other EGs trying to access the same output file.
_________________
Paul |
|
| Back to top |
|
|
| JULLRICH |
| Posted: Mon Jan 11, 2010 8:11 am Post subject: |
|
|
Apprentice
Joined: 11 Aug 2005
Posts: 42
Location: München
|
Hi mates,
I do have difficulties with that.
I do have a clustered environment (WMQ Cluster) with 2 Brokers. Each on a different machine.
In parallel I do have a third party system triggering my middleware while putting files to a directory.
This directory is mounted (NFS) to both brokers.
Sometimes it happens, that one file will be processed by both brokers. So the file will be duplicated by my middleware.
I'm very very sure, that the problem is in the broker, because I could retest this behaviour by hand.
But, this happens only sometimes - mostly it is working fine.
Does anybody has an idea on how to prevent duplicate processing.
Problem here: we do use to process files with the same name.
Regards
Jens |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jan 11, 2010 8:28 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You have re-discovered the problems with using files, they are non-transactional and loss or duplication is an inherent risk, with only limited ways to overcome these issues possible.
Better to send MQ messages into the broker whenever possible, or use a serialised process to handle the initial contact with the file (perhaps convert to messages before further processing).
Perhaps a locking method could be devised that would work over brokers but this would be a job for IBM to develop.
Any use of files can be a transactional integrity and/or scaleability weak point. I would recommend the end-point application directly generates messages instead, if the transactions are that important and need to be processed by different brokers at the same time. |
|
| Back to top |
|
|
| JULLRICH |
| Posted: Mon Jan 11, 2010 8:47 am Post subject: |
|
|
Apprentice
Joined: 11 Aug 2005
Posts: 42
Location: München
|
Well in my opinion this is the best you said.
But, because of the FileLock that is done by the broker according to its documentation, I will try to raise a PMR at IBM - lets see the result!
Regards
Jens |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jan 11, 2010 10:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
An earlier reply gave you a clear statement.
The fileInputNode uses an Operating System Lock. This stops two flows on the same machine from accessing the file.
Your situation is one where you require a Multi System Lock.
I'm guessing here but I think you are going to be SOL with getting IBM to fix this as AFAIK, the only platform that can currently do this AND runs Broker is Z/OS.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
