| Author |
Message
|
| harristm |
 Posted: Tue Dec 18, 2012 8:16 am Post subject: text file validation |
 |
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
I have a CSV file that I'm processing with file input, esql compute, and file output nodes. I need to validate the file before clearing out the table that I'm going to load the file into. If I issue the table delete before validating and find the file is bad, I end up with an empty table which is worse than a stale table. The file can grow to an open ended size so I use a delimited record detection which reads the file one record at a time.
The file has a header, body records and a trailer.
I need to compare the body record count and checksum from the trailer as part of the validation.
If I find a mis-match between record count or checksum in the trailer, I throw an exception and the flow stops.
So far, so good.
The problem crops up when the file that errored out was corrected and re-sent. It turns out that the exception that the flow threw did not clear the pending file being written by the file output node from the previous pass. It was left in mqsitransit from before.
If a compute node throws an exception, why doesn't the work that is in process, (records being written by the File Output Node) related to that flow get cleaned up and flushed?
What is the workaround for this.
I am using WMB 6.1.0.5 I have asked that we upgrade to version 8 but this function needs to be done before the upgrade. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Dec 18, 2012 8:27 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Dec 18, 2012 8:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
it is as if you didn't actually read the message posted. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Dec 18, 2012 9:17 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
it is as if you didn't actually read the message posted. |
I did read and restated the obvious since some people are not accepting of the end-of-life. I can understand the lack of acceptance for O-Care 'health' panels since the upgrade path is terminal. However, the IBM upgrade path is renewable and therefore do not understand why people resist it so.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Dec 18, 2012 9:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| mqjeff wrote: |
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
it is as if you didn't actually read the message posted. |
I did read and restated the obvious since some people are not accepting of the end-of-life. I can understand the lack of acceptance for O-Care 'health' panels since the upgrade path is terminal. However, the IBM upgrade path is renewable and therefore do not understand why people resist it so. |
And yet, you ignored entirely the part where
| Quote: |
| I am using WMB 6.1.0.5 I have asked that we upgrade to version 8 but this function needs to be done before the upgrade. |
And you gave an answer that solved the wrong problem.
The problem is that the file remains in the mqsitransit when an error has been thrown, not that the table is stale. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Dec 18, 2012 9:26 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
| However, the IBM upgrade path is renewable and therefore do not understand why people resist it so. |
Upgrading systems is not as easy as you might like to make out. It can take weeks of testing and even then your fingers are crossed tightly when you make the switchover.
All sorts of things appear out of the fog to trip you up. For many developers this is not something they like doing. After it isn't cool new development 
but often requires digging deep in old code and going quietly mad trying to understand why it does not work.   
I have a system on the west coast of the US that needs upgrading. The customer has given us a 4hr window in Nov 2013 to do it. If we don't get it cut over in that time, the next window is 6 months later.
It is dammed hard work for very little reward and perfectly understandable that people resist it, often with a vengance.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Dec 18, 2012 9:32 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| lancelotlinc wrote: |
| mqjeff wrote: |
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
it is as if you didn't actually read the message posted. |
I did read and restated the obvious since some people are not accepting of the end-of-life. I can understand the lack of acceptance for O-Care 'health' panels since the upgrade path is terminal. However, the IBM upgrade path is renewable and therefore do not understand why people resist it so. |
And yet, you ignored entirely the part where
| Quote: |
| I am using WMB 6.1.0.5 I have asked that we upgrade to version 8 but this function needs to be done before the upgrade. |
And you gave an answer that solved the wrong problem.
The problem is that the file remains in the mqsitransit when an error has been thrown, not that the table is stale. |
S/He's on the wrong version. This is his root cause for the file handling issue..
http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/topic/com.ibm.etools.mft.samples.BatchProcessing.doc/doc/introduction.htm
" end up with an empty table which is worse than a stale table. "
Which is what I addressed since the FileOutput mqsitransit issue will be solved when S/He upgrades.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| harristm |
| Posted: Tue Dec 18, 2012 9:34 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
Well, I can see where you would suggest that, but the application does not have create table authority and second I would rather keep this at a file level and have it validated before getting the database involved.
I am hoping that trailer records on big CSV files cannot be that rare.
but thank you for your suggestion. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Dec 18, 2012 9:37 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| harristm wrote: |
| lancelotlinc wrote: |
Create a temporary table. Copy the stale info to the temp table. If exception, copy the temp table back to original table.
Update to WMB 8. WMB 6 is end of life. |
Well, I can see where you would suggest that, but the application does not have create table authority and second I would rather keep this at a file level and have it validated before getting the database involved.
I am hoping that trailer records on big CSV files cannot be that rare.
but thank you for your suggestion. |
Then download the data from the table into Global Cache and refresh it from there in your error handler:
http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/bn23730_.htm
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| harristm |
| Posted: Tue Dec 18, 2012 9:47 am Post subject: Do future file output nodes have an error input |
|
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
I think this could be solved if I had an additional input on a File Output node that would instead of 'file complete' say 'file errored or failed' which would flush whatever the File Output node was working on.
Did this idea get implemented in WMB V7 or V8?
Thanks |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Dec 18, 2012 9:52 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Why can't you wire your catch terminal to the Finish File terminal? If an exception happens, hit the Finish File terminal in your error handler.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| harristm |
| Posted: Tue Dec 18, 2012 10:01 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
| lancelotlinc wrote: |
| Why can't you wire your catch terminal to the Finish File terminal? If an exception happens, hit the Finish File terminal in your error handler. |
Well, that does chase the file out of mqsitransit, but I do not want the file to get written out because, that would imply that the file was good and should continue processing. |
|
| Back to top |
|
|
| harristm |
| Posted: Tue Dec 18, 2012 10:15 am Post subject: Global Cache |
|
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
I checked the WMB 6.1 help and nothing came up for a Global Cache. Your reference lists a V8 help.
I know I am woefully behind versions but let me ask this, what did you drive to work this morning? If you are lucky enough to be driving a brand new car, how old was the one you traded in? upgrades are not a small investment and often hard to explain 'WMB 6 is end of life' as a reason to upgrade.
If I treated this CSV as a single file, not parsed, what is the file size that I could commit to being able to handle without causing resource or contention problems? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 18, 2012 10:27 am Post subject: Re: Global Cache |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| harristm wrote: |
| I know I am woefully behind versions but let me ask this, what did you drive to work this morning? If you are lucky enough to be driving a brand new car, how old was the one you traded in? |
I would imagine it was new and the previous one was about 3 months old, traded in because the ash trays were full. The poster is the most successful and influential IT person in the industry. According to his posts.
| harristm wrote: |
| upgrades are not a small investment |
They should be. License is not an issue and the cost of implementation (regression testing etc) should be easily absorbed as part of the life cycle.
| harristm wrote: |
| often hard to explain 'WMB 6 is end of life' as a reason to upgrade. |
That's the easy one to explain. You just tell whoever owns the business risk that once the software goes end of life, it's out of support and the next time something goes wrong IBM will not assist, which means the next time you want to change the hardware or patch the OS you either roll the dice & pray whatever you're running on that will still run or accept that you're stuck with current hardware & OS. You'll find that if whatever you're running on WMB is used in any way to make your business money the upgrade is easy.
Much easier to explain than "we need to use the new Global Cache feature in v8".
| harristm wrote: |
| If I treated this CSV as a single file, not parsed, what is the file size that I could commit to being able to handle without causing resource or contention problems? |
Depends on how much resource you have available on your platform and what else is contending with it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| harristm |
| Posted: Tue Dec 18, 2012 10:42 am Post subject: New Feature - p-l-e-a-s-e |
|
|
Apprentice
Joined: 02 Aug 2012
Posts: 29
|
I would like to see a new input to the File Output node that would flush the file that it was writing to or a coordination of the File Output node with the exception from another node in the flow.
An alternative and less obvious implementation would be a flag that could be set in the file complete message that would cause the file to be flushed instead of written to the output directory. |
|
| Back to top |
|
|
|
|
