| Author |
Message
|
| sudaltsov |
 Posted: Fri May 11, 2012 5:51 am Post subject: FileInput, End Of Data, Environment |
 |
|
Voyager
Joined: 02 May 2012
Posts: 82
|
There is FileInput node FI. There are two Compute nodes C1 and C2, connected to Out and End Of Data connectors of FI.
I am setting some Environment.foobar variables in C1. But in C2 the Environment is blank.
Is it me or the broker? What is the best way to keep some state between Out processing and End Of Data processing? Using SHARED variables does not look nice, it is not safe. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri May 11, 2012 5:56 am Post subject: Re: FileInput, End Of Data, Environment |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| sudaltsov wrote: |
| Using SHARED variables does not look nice, it is not safe. |
SHARED variables are extremely safe.
The documentation should be clear on what gets passed to End of Data - i.e. that it should or should not be creating a new instance of the message flow with a new Environment. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 11, 2012 5:59 am Post subject: Re: FileInput, End Of Data, Environment |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sudaltsov wrote: |
| Is it me or the broker? |
It's you. Re-read the documentation on the scope of the Environment tree.
| sudaltsov wrote: |
| Using SHARED variables does not look nice, it is not safe. |
Perhaps some more technical objection than "not look nice" would help us understand your postion.
Also what's not "safe" about SHARED variables? Is this the same line as some output nodes just don't support some parsers and you have to play around with some RCD nodes first?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Fri May 11, 2012 6:00 am Post subject: Re: FileInput, End Of Data, Environment |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
| mqjeff wrote: |
| SHARED variables are extremely safe. |
What I mean is that it is kept between the flow runs. That means the results of the previous flow execution can affect the current flow execution, if not properly cleared. The Environment does not have that issue. Am I wrong? |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Fri May 11, 2012 6:33 am Post subject: |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
From the document:
| Quote: |
| The End of Data message flow transaction is initiated only if this terminal is attached. |
Why is that a separate transaction?? I guess that is why Environment gets cleared after Out processing |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 11, 2012 7:26 am Post subject: Re: FileInput, End Of Data, Environment |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sudaltsov wrote: |
| What I mean is that it is kept between the flow runs. |
And it's kept safely between runs.
| sudaltsov wrote: |
| That means the results of the previous flow execution can affect the current flow execution, if not properly cleared. |
If it's not properly cleared thats's a problem with your code not the product.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 11, 2012 7:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sudaltsov wrote: |
| Why is that a separate transaction?? |
Why would it be the same one?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Fri May 11, 2012 7:28 am Post subject: Re: FileInput, End Of Data, Environment |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
| Vitor wrote: |
| If it's not properly cleared thats's a problem with your code not the product. |
With the Environment you do not have to do anything, it is cleared automatically between flow runs. |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Fri May 11, 2012 7:32 am Post subject: |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
| Vitor wrote: |
| Why would it be the same one? |
Because I am reading the same input file. So if "Out" connector provides me with individual records of the single file, it would be logical to consider N*"Out" + "End of data" as one transaction. Or, even better, provide the configuration property for handling that. Well, that would be logical IMHO - apparently IBM thinks differently. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 11, 2012 9:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sudaltsov wrote: |
| Because I am reading the same input file. So if "Out" connector provides me with individual records of the single file, it would be logical to consider N*"Out" + "End of data" as one transaction. |
If I have an input file with 2.5m records in it (and I do have such files) I don't want all 2.5m in the same unit of work.
| sudaltsov wrote: |
| Or, even better, provide the configuration property for handling that. Well, that would be logical IMHO - apparently IBM thinks differently. |
There's a place for such opinions. With enough votes, this opinion will become fact in WMBvn. Don't forget to post a link to gather votes from like-minded forumers (if that's a word).
In the meantime I guess you're stuck with using SHARED variables (or other mechanism of your choice) and ensuring your code cleans up after itself.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri May 11, 2012 9:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| sudaltsov wrote: |
| Vitor wrote: |
| Why would it be the same one? |
Because I am reading the same input file. So if "Out" connector provides me with individual records of the single file, it would be logical to consider N*"Out" + "End of data" as one transaction. |
That's entirely backwards.
If you want N*Out records to be one transaction, you read the whole file.
If you want N*Out records to be n transactions, you read the file by record and then use the end of file to know you're done. Unless you have something in the file that says there is N records in the file, then you can check to see if you've reached N yet.
It would make some sense if the node could determine that a given record was also the end of file, and signal this in the specific transaction for that record.
But that's generally a Turing Complete problem, so it is hopefully understandable that it's not solved here. |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Sat May 12, 2012 1:49 am Post subject: |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
| Vitor wrote: |
| I don't want all 2.5m in the same unit of work. |
Ok, you convinced me. It should be configurable.
| Vitor wrote: |
| There's a place for such opinions. With enough votes, this opinion will become fact in WMBvn. Don't forget to post a link to gather votes from like-minded forumers (if that's a word). |
Sorry, I am not sure I understand what place you are talking about. |
|
| Back to top |
|
|
| sudaltsov |
| Posted: Sat May 12, 2012 1:53 am Post subject: |
|
|
Voyager
Joined: 02 May 2012
Posts: 82
|
| mqjeff wrote: |
| If you want N*Out records to be one transaction, you read the whole file. |
Well, yes and no. I might want to handle every record separately - to split them on output, for example. Of course, there can be other ways todo it...
| mqjeff wrote: |
| It would make some sense if the node could determine that a given record was also the end of file, and signal this in the specific transaction for that record. But that's generally a Turing Complete problem, so it is hopefully understandable that it's not solved here. |
Yes, that is true. Indicating the last record would really be helpful and enough for my purposes. But they don't provide that. And I am not sure it is Turing Complete problem, to indicate the EOF scenario, for both delimited and fixed width cases. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun May 13, 2012 3:14 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sudaltsov wrote: |
| Sorry, I am not sure I understand what place you are talking about. |
I mean here. And I was serious about posting a link; votes count.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun May 13, 2012 4:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| sudaltsov wrote: |
| And I am not sure it is Turing Complete problem, to indicate the EOF scenario, for both delimited and fixed width cases. |
In the general case, it is exactly and only the Halting Problem. You only know that the file is "complete" if you know that the program WRITING the file has Halted.
In general, this is why using files for exchanging data is the absolutely worst choice you can make. Because files are open-ended and unterminated. With something like MQ or even HTTP, you work with discrete blocks.
Certainly, one can make some assumptions that if Broker has started to process the file, then the program that was WRITING the file is "done" at some level of meaning. But in order to indicate within the same transaction that covers the last record one would have to do a read-ahead and say "there is no more data after this, so this must be the last record". And that can be significantly more complicated than it might appear. It could also be just that easy - but without knowing what the code of the FileInput node looks like, I can't say.
If you want to do anything more than debate this in a public forum, you should either file an RFE or open a PMR. |
|
| Back to top |
|
|
|
|
