| Author |
Message
|
| Omuch |
 Posted: Sun Mar 25, 2012 12:24 am Post subject: Overwrite file after a failure |
 |
|
Acolyte
Joined: 23 Mar 2012
Posts: 54
|
Hi
(Version 6.1)
I have a flow that writes recoreds to a file as unmodified data, and second flow that reads that file when it's finished. The name of the file must be constant name.
FLOW1: FileOutputNode-> FILE
FLOW2: FILE -> FileInputNode ...
When I get an applicative failure in FLOW1' i dont want to finish the file,
but I want to start the whole process again, It means that FLOW1 will write again to the same file in the mqsitransit directory and not write a new file.
One solution that i have is a JAVA method that deletes the file in mqsitransit directory when i have a failure.
Does anybody have a better/safer idea?
Thanks |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sun Mar 25, 2012 3:22 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
What about not using files?
Have you thought about other methods?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Mar 25, 2012 4:09 am Post subject: Re: Overwrite file after a failure |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Omuch wrote: |
When I get an applicative failure in FLOW1' i dont want to finish the file,
but I want to start the whole process again, |
Why? Do you think peristence is it's own reward and any application coding error will have magically fixed itself in the meantime? You might get lucky if the error is a transient database or network problem but there are a large number of problems that don't respond to being beaten again and again so this design is just going to loop endlessly
| Omuch wrote: |
| It means that FLOW1 will write again to the same file in the mqsitransit directory and not write a new file. |
I'd be slightly surprised to discover this as any file in the mqsitransit directory should be renamed by the broker to a uniqiue name with a GUID. I'm also interested to know why you even care what's in the mqsitransit directory as that's the broker's business not yours. I suspect it's something to do with this comment of yours:
| Omuch wrote: |
| The name of the file must be constant name. |
You're so hung up on the name not changing you bold it and of course it's simple enough to ensure source & output have the same name at each end of the flow. I wonder if you're so keen on this you've done something / are attempting something to ensure the name stays the same during processing which is undesirable, dangerous and I would have said impossible.
| Omuch wrote: |
| One solution that i have is a JAVA method that deletes the file in mqsitransit directory when i have a failure. |
Why? The broker should be removing the file from the mqsitransit directory on failure and placing it into the mqsibackout directory (with it's original name before you huff at me). So it seems to me that the solution to your issue is to use the WMQ file capabiliies the way they're designed unless you can explain more clearly why your requirements prevent this.
| Omuch wrote: |
| Does anybody have a better/safer idea? |
I doubt there is a worse idea than having custom Java moving and renaming files while WMB is trying to use them so logically all ideas will be better and safer.
My better / safer idea: rethink this design! Especially rethink this "file name cannot change" thing; if this is an external (business) requirement what does it actually mean? I would take it to mean "cannot change on the file system for audit reasons" rather than "cannot change at any time every even during processing". Because if it's the latter you could never move the file in Windows, gzip it, sftp it or any other process that uses an intermediate file name as part of it's processing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Mar 25, 2012 7:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You have not demonstrated any good positive reason for using a file.
You have not demonstrated any good positive reason for using a file "of the same name", other than the basic requirement to have a FileInput node read the file.
You should be in planning to upgrade to MB v7, or MB v8. You should be strongly considering whether you should be doing any new development in v6.1 at this point in time.
Your basic design seems to be based on requirements that you haven't explained very well, so I won't say anything other than - it's better to use something OTHER than files when communicating between two message flows. |
|
| Back to top |
|
|
| Omuch |
| Posted: Sun Mar 25, 2012 11:38 am Post subject: Re: Overwrite file after a failure |
|
|
Acolyte
Joined: 23 Mar 2012
Posts: 54
|
| Vitor wrote: |
Why? Do you think peristence is it's own reward and any application coding error will have magically fixed itself in the meantime? You might get lucky if the error is a transient database or network problem but there are a large number of problems that don't respond to being beaten again and again so this design is just going to loop endlessly |
I will explain.
The source of the file data is from DB.
I get the data as large XML file (50 MB) with repeated entities, I read it as Parsed Record Sequence of the entity, transform it and write it to the file.
The chances that I will get an error in the data are less.
In any case the whole process is running every 24 hours. It contact admin that if there is an error in the DB it will be fixed until the next running.
| Vitor wrote: |
I I'm also interested to know why you even care what's in the mqsitransit directory as that's the broker's business not yours.
...
Why? The broker should be removing the file from the mqsitransit directory on failure and placing it into the mqsibackout directory
|
I do care, if I get an error in the data the broker doesnt remove the file, becasue its applicative error not a failure on writing the file.
So I need to remove the unfinished file from the mqsitransit before the next process of the flow.
The file name issue is less critical, of course I can use intermediate file name , In any case I need to remove the file from the directory
As I mentioned the process i srunning every, and its running about 24:00 so the option to concat the date to the name is risky , because of the changing of the date. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Mar 25, 2012 1:31 pm Post subject: Re: Overwrite file after a failure |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Omuch wrote: |
| I get the data as large XML file (50 MB) with repeated entities, I read it as Parsed Record Sequence of the entity, transform it and write it to the file. |
That's very, very, very odd. If it's XML why not use an XML parser? Not only is this the obvious choice for an XML document, it's more efficient.
| Omuch wrote: |
| The chances that I will get an error in the data are less. |
But not zero.
| Omuch wrote: |
| In any case the whole process is running every 24 hours. It contact admin that if there is an error in the DB it will be fixed until the next running. |
So if this fails the business don't need the data for another 24 hours. That's as odd as not using an XML parser for an XML document. Or to reference your other thread, not using a scheduler to schedule a task.
| Omuch wrote: |
| I do care, if I get an error in the data the broker doesnt remove the file, becasue its applicative error not a failure on writing the file. |
An application error is still an error to the flow if that's what you configure it to be. Indeed, WMB's default position is that all errors are errors.
| Omuch wrote: |
| So I need to remove the unfinished file from the mqsitransit before the next process of the flow. |
If WMB thinks the flow works it will move the file from the mqsitransit directory to the mqsiarchive directory (or whatever action you've specified for completion). If WMB thinks there's been an error it will move the file from the mqsitransit directory to the mqsibackout directory. If the file is being left in the mqsitransit directory it's because of something you have set up, either by accident or design. From your description I'd suspect that in the event of a problem you are neither finishing the file nor correctly throwing an error but are leaving the transaction open. This is a code bug in your flow.
| Omuch wrote: |
| The file name issue is less critical, of course I can use intermediate file name |
I'll repeat myself - you should never need to or want to know or use the intermediate file name.
| Omuch wrote: |
| In any case I need to remove the file from the directory |
Not if you have everything set up & coded correctly.
| Omuch wrote: |
| As I mentioned the process i srunning every, and its running about 24:00 so the option to concat the date to the name is risky , because of the changing of the date. |
So don't do it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Mar 25, 2012 1:55 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
And, again, there's absolutely no reason that you should use a file to pass data from one message flow to another in the same logical business process.
None.
So you should never have two flows that perform related work that are constructed as
FileInput(external location)->....FileOutput(directory on broker machine)
FileInput(directory on broker machine)->...
This just makes your whole flow subject to issues that are extremely easy to avoid by using MQOuptut/MQInput or almost any of the other transports available with Broker. |
|
| Back to top |
|
|
| Omuch |
| Posted: Mon Mar 26, 2012 5:06 am Post subject: Re: Overwrite file after a failure |
|
|
Acolyte
Joined: 23 Mar 2012
Posts: 54
|
| Vitor wrote: |
That's very, very, very odd. If it's XML why not use an XML parser? Not only is this the obvious choice for an XML document, it's more efficient.
|
I don't know how much you experienced with large XML, the broker cant handle parsing large xml (bigger about 10-15 MB) as I mentioned my file is about 50MB.
Therefore I use fileinput to read the data in parts- entities of the XML.
| Vitor wrote: |
An application error is still an error to the flow if that's what you configure it to be. Indeed, WMB's default position is that all errors are errors.
|
Of course its an error, but how you can configure it that the broker will fail the whole file.
For the broker its doesnt matter what is in the file, as long as he can write it.
If the broker cant write it he will not write it, but all the records that were written before and after the error will still be in the file.
In my
| Vitor wrote: |
If WMB thinks there's been an error it will move the file from the mqsitransit directory to the mqsibackout directory. If the file is being left in the mqsitransit directory it's because of something you have set up, either by accident or design. From your description I'd suspect that in the event of a problem you are neither finishing the file nor correctly throwing an error but are leaving the transaction open. This is a code bug in your flow. |
Im not leaving the transaction open, I have exception handling in my flow.
I didnt understand how the broker can know to fail the whole file because of an error in one/some records of the file
Can you give an example of handling an error so that the broker will fail the file and move it. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 26, 2012 5:31 am Post subject: Re: Overwrite file after a failure |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Omuch wrote: |
I don't know how much you experienced with large XML, the broker cant handle parsing large xml (bigger about 10-15 MB) as I mentioned my file is about 50MB.
Therefore I use fileinput to read the data in parts- entities of the XML. |
I've got a fair exposure to large XML, and have developed solutions to process 50-75MB XML files through WMB v6.1 & WMBv7. That's not to say you don't need to be mindful of the size you're processing, but it works.
(IIRC my record was an XML document 125Mb in size. WMBv6.1, running on Solaris, document was a Java object burped out into a file as XML somehow - "serialized"???)
| Omuch wrote: |
I didnt understand how the broker can know to fail the whole file because of an error in one/some records of the file
Can you give an example of handling an error so that the broker will fail the file and move it. |
Because it is (or should be) managing the file as a single transaction. However you're chopping it up is breaking that.
As to an example, any of the supplied file samples do this. As I said, this is the default position.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Omuch |
| Posted: Mon Mar 26, 2012 6:01 am Post subject: Re: Overwrite file after a failure |
|
|
Acolyte
Joined: 23 Mar 2012
Posts: 54
|
| Vitor wrote: |
| I've got a fair exposure to large XML, and have developed solutions to process 50-75MB XML files through WMB v6.1 & WMBv7. That's not to say you don't need to be mindful of the size you're processing, but it works. |
Did you used the Default Parsers? From what I know they cant parse a whole large XML? or you parsed only part of the xml in some kind of loop?
| Vitor wrote: |
Because it is (or should be) managing the file as a single transaction. However you're chopping it up is breaking that. |
I think here I was not clear enough.
When I tried to read the xml file by parsed record sequence of the entity, and then in the same transaction do the transformation on the data, because the size of the file, the broker couldnt handle it .
I tried it on small files and it worked.
So I Changed the design and now when I read the entity from the file I directly put it on queue.
Eventually the flow handle the whole file by thread for each entity , So its not a single transaction and therefor if there is an error the file is still in the mqsitransit . |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 26, 2012 6:19 am Post subject: Re: Overwrite file after a failure |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Omuch wrote: |
| Did you used the Default Parsers? From what I know they cant parse a whole large XML? or you parsed only part of the xml in some kind of loop? |
Parsed in directly with the XMLNSC parser, processed (obviously) by moving through the structure.
How do you know what you know?
| Omuch wrote: |
| When I tried to read the xml file by parsed record sequence of the entity, and then in the same transaction do the transformation on the data, because the size of the file, the broker couldnt handle it . |
Okay, I don't understand why you ever tried reading an XML file with parsed record sequence. You've also not indicated how you decided the file size was the problem and eliminated your code as the problem.
| Omuch wrote: |
| Eventually the flow handle the whole file by thread for each entity , So its not a single transaction and therefor if there is an error the file is still in the mqsitransit . |
Ok, your design is hosed. If you're taking a single input file it shouldn't end up in separate threads unless you're doing something quite specific and possibly Java related in your transformation. Certainly WMB won't understand how it got from 1 file to multiple threads and this will break the transaction model, causing the file to be left in the mqsittransit directory.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Omuch |
| Posted: Mon Mar 26, 2012 6:38 am Post subject: Re: Overwrite file after a failure |
|
|
Acolyte
Joined: 23 Mar 2012
Posts: 54
|
| Vitor wrote: |
Parsed in directly with the XMLNSC parser, processed (obviously) by moving through the structure.
How do you know what you know?
|
From my experience and from IBM Consultants, when ypu parsed the large XML the broker just parsed it without any problem?
| Vitor wrote: |
Okay, I don't understand why you ever tried reading an XML file with parsed record sequence.
|
The XML file include repeated entities , becasue I failed to parse the whole file the option that I Had is to read it in parts, and to process each part separated.
| Vitor wrote: |
You've also not indicated how you decided the file size was the problem and eliminated your code as the problem.
|
As I mentioned I tried to read small files and it worked, when I tried the same code on large files it didnt worked.
| Vitor wrote: |
Ok, your design is hosed. If you're taking a single input file it shouldn't end up in separate threads unless you're doing something quite specific and possibly Java related in your transformation. Certainly WMB won't understand how it got from 1 file to multiple threads and this will break the transaction model, causing the file to be left in the mqsittransit directory. |
I Know, but I had no choice because the problem of the large fie, that had me to separate the file to the entities and process each entity alone.
Actually I do use JAVA in my trasformation , but I dont think that this issue
requires separate threads. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 26, 2012 6:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If the problem is that, sometimes you get errors that cause the file to stay in the mqsitransit, then the solution is to separate the processing of the file from the transformation that sometimes throws errors.
I.e. your flow should be FileInput->MQOutput
MQinput->Java Transformation->MQOutput
You can take small steps to ensure that you can handle errors where a single record means that the whole file should be deleted.
Again, this whole design is very close in pattern to the Large File handling samples and patterns that are included with the Toolkit. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 26, 2012 6:48 am Post subject: Re: Overwrite file after a failure |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Omuch wrote: |
| From my experience and from IBM Consultants, when ypu parsed the large XML the broker just parsed it without any problem? |
Where these consultants from IBM or consultants who claimed knowledge of IBM products? I've never used anything other than the XMLNS/XMLNSC domains to parse XML, and never had any problems.
Before kimbert says anything, I used XMLNS either because the client insisted on it for consistency with existing code or in one memorable case because the whitespace was business significant. Don't laugh.
| Omuch wrote: |
| The XML file include repeated entities , becasue I failed to parse the whole file the option that I Had is to read it in parts, and to process each part separated. |
I don't remember the last time I saw an XML document that didn't have repeated entries. Even if you're processing each part separately (and I can see a design logic for that) you still shouldn't have separate threads in the way you're describing.
| Omuch wrote: |
| As I mentioned I tried to read small files and it worked, when I tried the same code on large files it didnt worked. |
That's not an answer to either point. How did you determine that the file size was the issue rather than the larger file had a data problem causing the broker abend? How did you determine that it was a problem with the broker handling of the file rather than your code running the broker out of memory for large files (assuming it was running out of memory)?
| Omuch wrote: |
| I Know, but I had no choice because the problem of the large fie, that had me to separate the file to the entities and process each entity alone. |
If you're insistent that you can't handle an XML document larger than 15Mb in WMB then I refer you to the advice further up this thread to use alternative transport mechanisms.
| Omuch wrote: |
Actually I do use JAVA in my trasformation , but I dont think that this issue
requires separate threads. |
Yet you have separate threads which WMB is not creating and this is breaking the transactional model. Is your Java experience principally with WAS or a similar server? Java written as if it's running in WAS rather than according to the rules laid down for a JCN will blow the broker out of memory quite quickly. Like if it has to process a lot of data from a large file.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 26, 2012 6:52 am Post subject: Re: Overwrite file after a failure |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Omuch wrote: |
| The XML file include repeated entities , becasue I failed to parse the whole file the option that I Had is to read it in parts, and to process each part separated. |
I don't remember the last time I saw an XML document that didn't have repeated entries. Even if you're processing each part separately (and I can see a design logic for that) you still shouldn't have separate threads in the way you're describing. |
If you instruct the FileInput node to output parsed record sequence, it will send each record in the file into a new message flow instance, which runs in a thread. If there are additional instances declared to the flow or the FileInput node, it's very easy to get different records in different threads.
That's usually the intent. |
|
| Back to top |
|
|
|
|
