| Author |
Message
|
| pottas |
 Posted: Thu Apr 16, 2015 4:09 am Post subject: File Processing error |
 |
|
Disciple
Joined: 27 Oct 2005
Posts: 185
Location: South Africa
|
Guys,
We have the following very simple implementation using File Nodes:
FileInput --> Compute --> MQ
A File gets sent to an AIX location (between 8 - 10 Mb size), in the Compute node, we don't do anything with the file or its contents. We simply update an Audit database to record that we received the file.
It then gets sent to an MQ Remote queue, which has its associated MQ Local queue at another destination.
The problem is, only sometimes only a part of the file reaches its destination. So, it will process this daily file for a number of days without any issues, just to process only half of it at a certain instance.
We have been using the file nodes for a number of implementations without any issues.
So, my thinking is, it might be a combination of two factors: the file size, and the time it lands. Could it be that, if two files land at the same time, and the File processor starts processing, it might dump only a part of the message onto MQ?
Is there anyone out there that have experienced a similar issue? |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Apr 16, 2015 4:32 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Can you guarantee that the file name is unique? (file name + time stamp). So that if 2 files arrive at about the same time there is no confusion? What I suspect is that the file is not yet complete when the broker is done with the current content. To avoid that, first move the file into the polled directory with a different name. Than do a rename when complete.
Also make sure the file is not sent with FTP but with a managed file transfer. FTP can break off the connection and you are left with parts of a file... That would also fit your scenario...
Do you have controls in the file? Like header / footer /total number of transactions, records etc..., total amounts ... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| pottas |
| Posted: Thu Apr 16, 2015 4:53 am Post subject: |
|
|
Disciple
Joined: 27 Oct 2005
Posts: 185
Location: South Africa
|
Thanks fjb_saper,
The file name is unique, yes, that I can confirm.
Also, yes, there are header/footer details in the files, but we don't even interrogate the contents of the file in the Message Broker, we handle it as a BLOB.
Also, we don't send the file with FTP, we read the file out of the file location, and dump it onto MQ.
What I can see, is we have no additional instances set on the flow that holds the File Input Node. I have requested Support to increase that in the hope of addressing this issue, but I do have my doubts though.
I also noticed that there is a different file that landed just about the same time (seconds difference) as this file, so could it be that this might affect the processing? I have a single FileInput Node that handles all files sent.
Another option maybe to have a File Node for each type of file? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 16, 2015 4:57 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| pottas wrote: |
Thanks fjb_saper,
The file name is unique, yes, that I can confirm.
Also, yes, there are header/footer details in the files, but we don't even interrogate the contents of the file in the Message Broker, we handle it as a BLOB.
Also, we don't send the file with FTP, we read the file out of the file location, and dump it onto MQ.
What I can see, is we have no additional instances set on the flow that holds the File Input Node. I have requested Support to increase that in the hope of addressing this issue, but I do have my doubts though.
I also noticed that there is a different file that landed just about the same time (seconds difference) as this file, so could it be that this might affect the processing? I have a single FileInput Node that handles all files sent.
Another option maybe to have a File Node for each type of file? |
Sorry, it wasn't about sending the file with ftp, more receiving it via an ftp / sftp protocol... Expect that if you are sending an incomplete file, may be the reason was that you received an incomplete file to begin with... , or that the broker got to the end of the file before the file was fully written (hence the rename when complete as a better option)
_________________
MQ & Broker admin
Last edited by fjb_saper on Thu Apr 16, 2015 5:00 am; edited 1 time in total |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Apr 16, 2015 5:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Unix, being a piece of rubbish, does not lock files by default. The WMB file nodes do use locking - but even this doesn't stop other software reading or writing files without locking (such as ftp and sftp does).
So the only way to avoid a (partial) file being processed prematurely (while still being written) is to use a rename technique.
ftp or sftp the file with name A, and then rename it to name B (rename function is available as part of ftp and sftp subcommands).
Make sure your file node pattern mask is set to process files of name B only (or a suitable pattern).
This way you won't get partial files into the file node.
Even better - use MQ instead of files, there are so many good reasons for this advice, it's not just about picking one acronym over another.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Thu Apr 16, 2015 5:06 am; edited 1 time in total |
|
| Back to top |
|
|
| pottas |
| Posted: Thu Apr 16, 2015 5:00 am Post subject: |
|
|
Disciple
Joined: 27 Oct 2005
Posts: 185
Location: South Africa
|
| Oh, OK, well then I can confirm that we receive the file in perfect order. The file is complete where it lands in our FTP location, and it is still in perfect order once it was processed in our Archive directory. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 16, 2015 5:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| pottas wrote: |
| Oh, OK, well then I can confirm that we receive the file in perfect order. The file is complete where it lands in our FTP location, and it is still in perfect order once it was processed in our Archive directory. |
But is it complete when you look at the archive location? You say you are sending the file as an MQ message... What does the receiving party do if the buffer will not accommodate the full file? Truncate?
What are the GMO options of the receiving side?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 16, 2015 5:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| pottas wrote: |
| Oh, OK, well then I can confirm that we receive the file in perfect order. The file is complete where it lands in our FTP location, and it is still in perfect order once it was processed in our Archive directory. |
This is not as conclusive as you think. If broker starts reading the file while ftp is still writing it, it will copy the file in the state it's in to the transit in directory & process it. Once it's finished processing that incomplete file it will move the now-completed file to the archive directory.
Behaviour noted on WMBv7.0.0.6 running on Solaris.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Apr 16, 2015 5:07 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
See my post above - applies to all versions of the rubbish shareware OS known as Unix or Linux.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 16, 2015 5:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| rubbish shareware OS known as Unix or Linux. |
We pay a lot of money for our Solaris.
It's a shame distributed chipsets don't run z/OS but hey.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Apr 16, 2015 5:28 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The problem can be reduced by increasing the directory scan interval on the fileinput node. This reduces the chance of catching a half-baked file.
However the rename method is the best way to avoid this issue and this should have been written in huge letters all over the documentation.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 16, 2015 5:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| However the rename method is the best way to avoid this issue and this should have been written in huge letters all over the documentation. |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| pottas |
| Posted: Thu Apr 16, 2015 6:26 am Post subject: |
|
|
Disciple
Joined: 27 Oct 2005
Posts: 185
Location: South Africa
|
Thanks guys, you helped a heck of a lot, we should flag this thread somehow to assist other guys with the same issue.
As for Unix, in my native language, I have a lot of harsher words for it, 'cause it surely looks like an issue with it.
Si, I will do the following:
Immediate action: Increase the scan interval to something like every 30 minutes;
Long Term Action: Use the rename function as all of you guys proposed.
Again, I thanks you all. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 16, 2015 6:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| pottas wrote: |
Thanks guys, you helped a heck of a lot, we should flag this thread somehow to assist other guys with the same issue.
As for Unix, in my native language, I have a lot of harsher words for it, 'cause it surely looks like an issue with it.
Si, I will do the following:
Immediate action: Increase the scan interval to something like every 30 minutes;
Long Term Action: Use the rename function as all of you guys proposed.
Again, I thanks you all. |
Not so much an issue with Unix/Linux as an issue with FTP across all distributed platforms...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Apr 16, 2015 7:57 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Any OS that makes file integrity optional is a pile of junk.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
