ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB 10.0.0.23 - JSON Parser and BitStreams

Post new topicReply to topic
IIB 10.0.0.23 - JSON Parser and BitStreams View previous topic :: View next topic
Author Message
usagi83
PostPosted: Tue Apr 06, 2021 8:29 pm Post subject: IIB 10.0.0.23 - JSON Parser and BitStreams Reply with quote

Newbie

Joined: 06 Apr 2021
Posts: 3

Hello everyone, I am working on performance tuning an IIB application to reduce memory utilization, and I am trying to implement the IBM pattern below, except my input message is of type JSON. My input message is large, and consists of a variable length array of repeating JSON objects.

I am struggling to implement the suggestion to create a single output object, and then copy it as a BitStream to the output list. I believe the first question is the JSON parser compatible with the FolderBitStream option?

If it is, when the below code fires, it outputs items with a blob in them, which decodes to my JSON object, however after writing to the out MQ, the items are empty in the next downstream app and in the MQ message.


IBM Pattern: IBM Knowledge Center topic=message-large-output (I cannot post URLS yet)

Code Stub:
WHILE LASTMOVE (InItem) DO
--Create temporary item
CREATE FIELD Environment.Variables.JSON.Data.ItemList IDENTITY (JSON.Array)ItemList;

CREATE FIELD Environment.Variables.JSON.Data.ItemList.Item IDENTITY (JSON.Object)Item;

DECLARE refItem REFERENCE TO Environment.Variables.JSON.Data.ItemList.Item;

-- Maps the InItem into the temporary Item refItem
CALL CreateItemObject(refItem,InItem);

--Removes InItem from memory now that it is done, this section works properly and has already reduced memory utilization.
MOVE InItem NEXTSIBLING REPEAT TYPE NAME;
IF (LASTMOVE (InItem) = TRUE) THEN
DELETE PREVIOUSSIBLING OF InItem;
ELSE
DELETE FIELD InItem;
END IF;

--Create bitstream of the refItem
DECLARE ItemBitStream BLOB ASBITSTREAM(refItem OPTIONS FolderBitStream);

--Copy refItem to output array
CREATE LASTCHILD OF refOutItemList IDENTITY (JSON.Object)Item VALUE ItemBitStream;

--Delete temporary item
DELETE FIELD refItem;

END WHILE;
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 07, 2021 6:57 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8887
Location: US: west coast, almost. Otherwise, enroute.

With RAM so inexpensive why the effort to architect apps to use less?
_________________
Five out of four people have trouble with fractions. - Steven Wright.
Back to top
View user's profile Send private message
usagi83
PostPosted: Wed Apr 07, 2021 7:52 am Post subject: Reply with quote

Newbie

Joined: 06 Apr 2021
Posts: 3

The app receives and parses large JSON messages (200k-1/2MB), and when applied to 25 threads(For this app), combined with the rest of the load of the upstream/concurrent processes, manages to eat up enough of the server's 26 GB of ram to bring down the server.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 07, 2021 8:02 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8887
Location: US: west coast, almost. Otherwise, enroute.

Ummmm. 25 threads of .5MB = 12MB-ish concurrent IIB for this app. What is using the remainder of the 26GB of server RAM? What leads you to believe that this app is low-hanging fruit?
_________________
Five out of four people have trouble with fractions. - Steven Wright.
Back to top
View user's profile Send private message
usagi83
PostPosted: Wed Apr 07, 2021 8:23 am Post subject: Reply with quote

Newbie

Joined: 06 Apr 2021
Posts: 3

My understanding is that parsing the JSON into the message tree greatly inflates the memory requirements. This is required, as the input JSON must be mapped into a different JSON object to be sent to a down stream process.

We have isolated this app and observed its JSON parser taking up multiple gigabites of memory. Note this is without some of the performance tweaks present in the code stub, like removing the processed claim from memory when done with it. We are trying to complete the last suggestion in the large message pattern from IBM to ensure we have tuned the app as far as we can.

However, I am starting to suspect that even if we figure out the ASBITSTREAM copy, we may need to reparse the data for use downstream, simply moving the memory pain from one app to another.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 07, 2021 9:44 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8887
Location: US: west coast, almost. Otherwise, enroute.

usagi83 wrote:
My understanding is that parsing the JSON into the message tree greatly inflates the memory requirements. This is required, as the input JSON must be mapped into a different JSON object to be sent to a down stream process.

Yes, memory is required. "Greatly" presumes that you believe that memory is being wasted, and not used effectively and efficiently.

usagi83 wrote:
We have isolated this app and observed its JSON parser taking up multiple gigabites of memory. Note this is without some of the performance tweaks present in the code stub, like removing the processed claim from memory when done with it. We are trying to complete the last suggestion in the large message pattern from IBM to ensure we have tuned the app as far as we can.

However, I am starting to suspect that even if we figure out the ASBITSTREAM copy, we may need to reparse the data for use downstream, simply moving the memory pain from one app to another.

Back to my original suggestion, namely: buy more RAM. How much effort is your organization going to devote to a software solution when a hardware solution is an option?
_________________
Five out of four people have trouble with fractions. - Steven Wright.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Apr 07, 2021 6:08 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2223
Location: Melbourne, Australia

We had a struggle with IIB memory usage on Linux, where some flows process large messages (10 to 50MB). It seems that IIB allocates an additional memory buffer every time a slightly larger message comes through. We upped the server memory from 48GB to 96GB, restarted the IS's every week, and the issue went away. Memory is cheap, fixing complex underlying IIB behavioral issues is expensive.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 07, 2021 9:20 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8887
Location: US: west coast, almost. Otherwise, enroute.

What gbaddeley said.
_________________
Five out of four people have trouble with fractions. - Steven Wright.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB 10.0.0.23 - JSON Parser and BitStreams
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.