MQSeries.net :: View topic - Broker environment variable setup

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Broker environment variable setup

Goto page 1, 2 Next

Broker environment variable setup

« View previous topic :: View next topic »

Author

Message

nmaddisetti

Posted: Fri Mar 19, 2010 7:58 am Post subject: Broker environment variable setup

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi All,

I setup the variable MQSI_FILENODES_MAXIMUM_RECORD_LENGTH with 209715200 but still I am getting exception 'RecordTooLongException - length exceeds 104857600 bytes'

I tried mqsireportproperties command with couple of options but no luck to see expected result.

Can some one help me, how to make sure variable value applied to broker.

Thanks & Regards,
Venkat.

lancelotlinc

Posted: Mon Mar 22, 2010 10:17 am Post subject: Re: Broker Setup

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

Hi Venkat,

Per support docview sizes larger than 100 MB are possible but not supported.

A typical strategy to deal with very large files is to store them on the filesystem and send a reference to them in a message, rather than the file itself.

Perhaps if you provided more information about the solution you are trying to implement, we can help you come up with a better compromise. For instance, will your message flow be changing the information inside this large message?

HTH
Sirlancelotlinc
Rorqual qualified.

nmaddisetti

Posted: Mon Mar 22, 2010 10:37 am Post subject:

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi,

Thanks for the reply.
We are processing this big file and storing some data from this file into DB.
When it come to my question how can I make sure my variable value is set to broker?

Thanks & Regards,
Venkat.

lancelotlinc

Posted: Mon Mar 22, 2010 10:57 am Post subject:

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

Hi Venkat,

I would disregard that parameter by doing your database processing from a JavaCompute node using Java code. This way you are not hindered by this limit and can page through the large file using a file cursor.

Let me know if this works for you.

Best regards,
Sirlancelotlinc
Rorqual qualified.

nmaddisetti

Posted: Mon Mar 22, 2010 11:54 am Post subject:

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi,

Initially our solution is with JCN only but when we encountered 160 MB file it failed with the exception Outof Memory exception then by setting this variable MQSI_FILENODES_MAXIMUM_RECORD_LENGTH we are able to process the file.
And to keep the JCN solution to process the 160 MB file we need to increase JVM heap size to 1.5 GB and with this setting we are able to process 160 MB file but we may recevie much bigger files so then we have to increase JVM heap size to 2 GB or 3 GB...
Client people are not interesed to keep on increasing the JVM heap size then when I am looking at the file node it can handle more than 100 MB from 6.1.0.3 onwords and it will use less resources when compared to reading and parsing whole file using JCN.
with this information and based on the discussion in the below link I thought File input node is better solution comapred to JCN.
http://www.mqseries.net/phpBB2/viewtopic.php?p=259419&sid=1b79c111739af474f9cfe91e65c810f9

Any better thoughts from your side are appreciated.

Thanks & Regards,
Venkat.

lancelotlinc

Posted: Mon Mar 22, 2010 12:15 pm Post subject:

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

The letter of the law is yes, it can handle more than 100 MB. The practical limit might as well be just 100 MB.

The trick to the solution is not to try to read the whole file. Read only the part you want. Don't store the BLOB in memory, write it out to a temp file as you read it from the original file.

No need to increase JVM heap size when you are only paging through a file.

InputStream instr = new BufferedInputStream (new FileInputStream (fileName));

Allocate a buffer (20 Mb) when calling the read method. When you get to a part of the file you are interested in, open an output stream and start moving data to the temp file until you are end of that part you want.

Think in nibble-sized units of work. Sounds like you are trying to accomplish the whole task with one step. Break up the single step into smaller tasks.

Initialize variables and then use them repeatedly in a while loop. Reduces items that need to be garbage collected.

Instead of storing intermediate data in a big array and extending it every time data is retrieved, maintain data in a list, where each element contains only a piece of data. This can be references to filenames on disk rather than in-memory objects.

Your going to need to be creative here. Think outside the box. Neither human nor computers think in megabyte data elements. Both of us think in smaller units. Use the filesystem rather than heap memory.

Good luck.

mqjeff

Posted: Mon Mar 22, 2010 12:17 pm Post subject:

Grand Master

Joined: 25 Jun 2008
Posts: 17447

I'm not sure that anything lancelotlinc has said is strictly correct, and may be fairly misleading.

Even if you do use the SUPPORTED option to increase the file size by using MQSI_FILENODES_MAXIMUM_RECORD_LENGTH, and you do need to be at a later fixpack to use that (you should really be at 6.1.0.5 at least!), you will still need to increase the memory in use by Broker in order to represent that file data.

If you want to process an arbitrarily large file whilst maintaining a constant size of memory usage, you need to process the arbitrarily large file in constant size chunks - that is to say in RECORDS.

The FileInput node will handle this for you and allow you to propagate individual records from the file either by fixed length or by logical records in a message definition.

This is the preferred and best way to deal with large files.

If you need to reassemble all of the records after processing them, you can either use the FileOutput node's Finish File terminal to close a file when the last record has gone in, or you can use the Collection node to assemble all the records in one message. (that will take memory to do)

nmaddisetti

Posted: Mon Mar 22, 2010 12:52 pm Post subject:

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi Jeff,

Thanks for the reply.
We already have application in place which is file inputnode based with finetuned esql code to handle large files but record detection option is WHOLE FILE with ondemand parsing as we can't use any other option for record detection . only thing is this variable is not set on the box and recently added this variable to mqsiprofile file by our admin and after that when we dropped the file it is still saying RecordTooLong.
I think variable was not set properly.
and my request is for the command to know this variable is set or not.
I tried copule of options with mqsireportproperties but I did not see anything related to this variable.

Thanks in advance,
Venkat.

Vitor

Posted: Mon Mar 22, 2010 12:57 pm Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

nmaddisetti wrote:

record detection option is WHOLE FILE with ondemand parsing as we can't use any other option for record detection

I'm interested. What's the structure of the file that the FileInput node can't locate records but (presumably) your ESQL can? Or does the ESQL handle the entire 100Mb as a single object? Is the file really 100+Mb of unsegmented business data?

Slightly off topic to your original question I admit, but it's an interesting file structure & I hope you'll share.
_________________
Honesty is the best policy.
Insanity is the best defence.

nmaddisetti

Posted: Mon Mar 22, 2010 1:17 pm Post subject:

Centurion

Joined: 06 Oct 2004
Posts: 145

Hi Vitor,

I doent mean that esql is locating records may be my previous description is giving you this but my flow is picking the file with Whole File option and On Demand parsing and esql was written with while last move and next sibling and references were used to point to possible indepth parent.

And when it comes to data it is an XML report( called Audit report) for X12 835 file generated by another box called EDIFECS ( EDIFECS will do HIPAA validations) if there are more HIPAA errors in 835 file then we will see large Audit report.

This audit report will be having repeated structures but they are after 5 levels but we need to capture data from level 1 ,2,3,4 then level 5. ( for more information about the structure please refer to the link in the previous post) becasue of this levels we are not able to use parse record sequense as per my understanding.

Thanks in advance,
Venkat.

fjb_saper

Posted: Mon Mar 22, 2010 1:30 pm Post subject:

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

Well if your report is 100 MB long there is seriously something wrong about the application or the report.
The report should just highlight the problems, show some stats and point to a detail file (for those who need to look over complex original data).

Seriously, nobody reads a report that's more than 2 pages anymore.
You need a digest!

_________________
MQ & Broker admin

Vitor

Posted: Mon Mar 22, 2010 1:42 pm Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

nmaddisetti wrote:

And when it comes to data it is an XML report( called Audit report) for X12 835 file generated by another box called EDIFECS ( EDIFECS will do HIPAA validations) if there are more HIPAA errors in 835 file then we will see large Audit report.

I'm sure kimbert will be along in a moment to correct me, but I thought WMB could parse X12 & EDIFACT message formats, which are indeed the kind of nested record you describe.
_________________
Honesty is the best policy.
Insanity is the best defence.

kimbert

Posted: Mon Mar 22, 2010 2:53 pm Post subject:

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:

I'm sure kimbert will be along in a moment to correct me, but I thought WMB could parse X12 & EDIFACT message formats...

Nothing to correct in that statement. Message broker can indeed parse X12 and EDIFACT files.
However, nmaddisetti is attempting to parse an XML report file - one which seems to relate to some other X12/EDIFACT document. I think his problem is that the Parsed Record Sequence option, which would be ideal for his needs, only works if the 'Record' is the top-level element in a message definition. It can't deal with a *nested* repeating record.

Vitor

Posted: Mon Mar 22, 2010 3:07 pm Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

The light of understanding flows through me...

_________________
Honesty is the best policy.
Insanity is the best defence.

kimbert

Posted: Mon Mar 22, 2010 3:21 pm Post subject:

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

I think the most memory-efficient solution is to (mis)use the 'Delimiter' property on the FileInput node. On the other thread, Vitor correcty pointed out that you will sometimes get badly-formed XML documents. I think it is possible to work around that problem.
Here's your message format:

Code:

Let's imagine that you set the 'Delimiter' on the FileInput node to '<CHILD8>'. You will get the following sequence of 'records' in your flow:
1.

Code:

<ROOTTAG><Child1></Child1><Child2></Child2><Child3>/Child3><CHILD4><CHILD5><CHILD6><CHILD7>

Code:

Code:

...
Final Record:

Code:

</CHILD7>
</CHILD6>
</CHILD5>
</CHILD4>
<Child2></Child2>
</ROOTTAG>

I can think of several ways to make these snippets of XML well-formed. I presume that the documents always have the same overall structure, so for 1. you can append the ( fixed) XML from my 'Final Record' above, and get a well-formed XML snippet. You can then parse that using CREATE...PARSE DOMAIN 'XMLNSC' and extract the header info.
For 2...N-1 you will need to put <CHILD8> at the start and '</CHILD8> at the end. Then use CREATE...PARSE again to create a parsed tree for your flow to process.

You can safely ignore the final record, unless it contains useful data.

Now I'm waiting for someone to point out the fatal flaw in my cunning plan.

Display posts from previous:

Goto page 1, 2 Next

Page 1 of 2

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Broker environment variable setup

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP