| Author |
Message
|
| wbi_telecom |
 Posted: Thu Aug 23, 2012 5:00 am Post subject: Shared variable changes? |
 |
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
We are using broker version 7.0.0.3 on AIX. We have a flow which processes large files (upto 8 GB) containing flat records and converts them into a single XML message.The flow uses file input node to read the file record by record in the flow. The flow maps each flat record into an XML block and stores it in a shared ROW variable with XMLNSC parser. At the End of File record, the XML in the shared variable is copied to the outputRoot (propagated to Fileoutput node) and the shared varialbe is initialised as follows
| Code: |
| SET Svar.XMLNSC = NULL; |
This flow has been working great for last couple of years when our broker was on Z linux. We recently moved our broker from Z Linux to AIX and this flows still works great for the first file but the next file takes a lot of time.
i.e. 4 GB file gets proccessed in 8 minutes in first run and takes close to 2 hours in the second run.
If we restart the flow or reload the execution group, the flow works fine. (I suspect because the shared variable gets initialized).
I looked at the memory utilization of the flow using flow statistics and the memory is released at the end of the flow when the shared varible is initialized. But still the next file takes a long time.
Is there a reason the flow is behaving differently on AIX than it behaved on Z linux? Is there a better way to initialize the shared variable that would work on AIX ? ( I tried doing SET SVar = NULL; as well but no success)
Cheers, |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Thu Aug 23, 2012 5:06 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
You should be managing your own cache data. Here are more options besides shared variable:
1. Singleton
2. solidDb
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Aug 23, 2012 6:34 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| WMB 8001 has a new global cache feature. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Thu Aug 23, 2012 7:25 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
Perhaps you can also consider to use fileOutput node with append option and put your processed segments straight to result file?
--
Marko |
|
| Back to top |
|
|
| McueMart |
| Posted: Thu Aug 23, 2012 7:26 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| Have you tried a user trace? If you enable user trace and take a trace of you reading in the first file, then take a trace of you reading in a second file, you should be able to compare them and see where the extra time is being taken. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 23, 2012 7:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm confused.
You're reading the file record by record, presumably to improve performance and reduce the memory footprint.
But then you're ignoring that and accumulating the entire contents of the file in memory, in a large message tree. |
|
| Back to top |
|
|
| wbi_telecom |
| Posted: Thu Aug 23, 2012 9:07 am Post subject: |
|
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
The reason we keep the content in a shared variable is because we want to send a single XML file out containing all the records in the flat file. Because of the complexity of the XML schema, we decided not to append the file in fileoutput node . We create the full XML in the message flow and validate it against the schema before sending it out. The flow does use memory while the XML is being built but it releases all of it as soon as we send the file out.
We had asked IBM if there is a better way to collect messages within the flow (because we tried the collector node and that brought the server down because it creates parsers which take up much more memory than the message size). But in broker 6.1 there was no other option but to use shared variable and so far it was working great for us.
Cheers, |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 23, 2012 9:38 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| wbi_telecom wrote: |
| shared variable |
This is your problem, not your solution. Find another way to store the data besides shared variable. Shared variable is not the only option and other options will work better.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 23, 2012 9:48 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| wbi_telecom wrote: |
| But in broker 6.1 there was no other option but to use shared variable |
This is false. This statement is not correct. Singleton and solidDb work with all modern versions of Broker, including 6.1.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 23, 2012 9:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Okay, again.
Why are you reading it record by record? to reduce the complexity of the transform?
You can accomplish that in other ways, by reading the whole file, moving it to Environment, and then processing individual records and reassembling in Environment.
And the copy environment to output. |
|
| Back to top |
|
|
| wbi_telecom |
| Posted: Thu Aug 23, 2012 10:19 am Post subject: |
|
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
Jeff,
Yes, to Reduce the complexity of transformation and the XML file is much easier to hold in memory than the flat file. We get files as big as 30 GB .They are fixed length files generated by a COBOL batch program.
but they produce much smaller XMLs .
LancelotInc
We have 30 interfaces built using this design which have been working in produciton for last 2 years. We have processed files of varying sizes daily without any issues. Why is shared variable suddenly a problem now? I will have to justify that to the upper management before I start changing a well established design.
Cheers, |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 23, 2012 10:23 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| wbi_telecom wrote: |
| Why is shared variable suddenly a problem now? |
Because it doesn't work with the increased file sizes?
I am agnostic to any solution. As long as it works.
Henry Ford once said, "You can have any color car you want, as long as it's Black."
You can have any solution you want, as long as it works.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Thu Aug 23, 2012 10:43 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
I'm just guessing, but one thing that came to my mind is that there could be some difference how Z linux and AIX handles their memory allocation and deallocation. Perhaps you could rise a PMR to find out more accurate answer than you can get from here.
--
Marko |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 23, 2012 11:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
there are several options other than a shared variable.
You should presume that the parser issue you experienced under 6.1 is fixed in 7.0, and so collector should be viable.
You can use a database table instead of collector.
You can use a FileOutput and append the records to a temporary file and then use FileGet or another flow starting from FileInput to read the concatenated file and then make sure it fits the proper xml schema.
You can emulate the collector node yourself and use MQOutput and MQget to assemble the pieces.
It's a little surprising that the XML file is smaller than the COBOL file.
There are still techniques you can use to only process the cobol file in small pieces, whilst still using a single instance of the message flow. You should review the Large Messaging samples. |
|
| Back to top |
|
|
| NealM |
| Posted: Thu Aug 23, 2012 3:06 pm Post subject: |
|
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
| Geez wbi_, everybody wants to rewrite your flow! I think the two best answers are (1) for you to use the usertrace (try it at "normal" level to compare the time spent per node first) and (2) file a PMR, you may be able to justify a sev 1. I assume you were at 7.0.0.3 on zLinux also where/when it behaved properly, before your switch to AIX. Since AIX is IBM's baby, I'm sure the PMR will get some attention. |
|
| Back to top |
|
|
|
|
