| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| IIB 10.0.0.23 - JSON Parser and BitStreams |
« View previous topic :: View next topic » |
| Author |
Message
|
| usagi83 |
 Posted: Tue Apr 06, 2021 8:29 pm Post subject: IIB 10.0.0.23 - JSON Parser and BitStreams |
 |
|
Newbie
Joined: 06 Apr 2021
Posts: 3
|
Hello everyone, I am working on performance tuning an IIB application to reduce memory utilization, and I am trying to implement the IBM pattern below, except my input message is of type JSON. My input message is large, and consists of a variable length array of repeating JSON objects.
I am struggling to implement the suggestion to create a single output object, and then copy it as a BitStream to the output list. I believe the first question is the JSON parser compatible with the FolderBitStream option?
If it is, when the below code fires, it outputs items with a blob in them, which decodes to my JSON object, however after writing to the out MQ, the items are empty in the next downstream app and in the MQ message.
IBM Pattern: IBM Knowledge Center topic=message-large-output (I cannot post URLS yet)
Code Stub:
WHILE LASTMOVE (InItem) DO
--Create temporary item
CREATE FIELD Environment.Variables.JSON.Data.ItemList IDENTITY (JSON.Array)ItemList;
CREATE FIELD Environment.Variables.JSON.Data.ItemList.Item IDENTITY (JSON.Object)Item;
DECLARE refItem REFERENCE TO Environment.Variables.JSON.Data.ItemList.Item;
-- Maps the InItem into the temporary Item refItem
CALL CreateItemObject(refItem,InItem);
--Removes InItem from memory now that it is done, this section works properly and has already reduced memory utilization.
MOVE InItem NEXTSIBLING REPEAT TYPE NAME;
IF (LASTMOVE (InItem) = TRUE) THEN
DELETE PREVIOUSSIBLING OF InItem;
ELSE
DELETE FIELD InItem;
END IF;
--Create bitstream of the refItem
DECLARE ItemBitStream BLOB ASBITSTREAM(refItem OPTIONS FolderBitStream);
--Copy refItem to output array
CREATE LASTCHILD OF refOutItemList IDENTITY (JSON.Object)Item VALUE ItemBitStream;
--Delete temporary item
DELETE FIELD refItem;
END WHILE; |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Wed Apr 07, 2021 6:57 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
With RAM so inexpensive why the effort to architect apps to use less?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| usagi83 |
| Posted: Wed Apr 07, 2021 7:52 am Post subject: |
|
|
Newbie
Joined: 06 Apr 2021
Posts: 3
|
| The app receives and parses large JSON messages (200k-1/2MB), and when applied to 25 threads(For this app), combined with the rest of the load of the upstream/concurrent processes, manages to eat up enough of the server's 26 GB of ram to bring down the server. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 07, 2021 8:02 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Ummmm. 25 threads of .5MB = 12MB-ish concurrent IIB for this app. What is using the remainder of the 26GB of server RAM? What leads you to believe that this app is low-hanging fruit?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| usagi83 |
| Posted: Wed Apr 07, 2021 8:23 am Post subject: |
|
|
Newbie
Joined: 06 Apr 2021
Posts: 3
|
My understanding is that parsing the JSON into the message tree greatly inflates the memory requirements. This is required, as the input JSON must be mapped into a different JSON object to be sent to a down stream process.
We have isolated this app and observed its JSON parser taking up multiple gigabites of memory. Note this is without some of the performance tweaks present in the code stub, like removing the processed claim from memory when done with it. We are trying to complete the last suggestion in the large message pattern from IBM to ensure we have tuned the app as far as we can.
However, I am starting to suspect that even if we figure out the ASBITSTREAM copy, we may need to reparse the data for use downstream, simply moving the memory pain from one app to another. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 07, 2021 9:44 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| usagi83 wrote: |
| My understanding is that parsing the JSON into the message tree greatly inflates the memory requirements. This is required, as the input JSON must be mapped into a different JSON object to be sent to a down stream process. |
Yes, memory is required. "Greatly" presumes that you believe that memory is being wasted, and not used effectively and efficiently.
| usagi83 wrote: |
We have isolated this app and observed its JSON parser taking up multiple gigabites of memory. Note this is without some of the performance tweaks present in the code stub, like removing the processed claim from memory when done with it. We are trying to complete the last suggestion in the large message pattern from IBM to ensure we have tuned the app as far as we can.
However, I am starting to suspect that even if we figure out the ASBITSTREAM copy, we may need to reparse the data for use downstream, simply moving the memory pain from one app to another. |
Back to my original suggestion, namely: buy more RAM. How much effort is your organization going to devote to a software solution when a hardware solution is an option?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Apr 07, 2021 6:08 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
We had a struggle with IIB memory usage on Linux, where some flows process large messages (10 to 50MB). It seems that IIB allocates an additional memory buffer every time a slightly larger message comes through. We upped the server memory from 48GB to 96GB, restarted the IS's every week, and the issue went away. Memory is cheap, fixing complex underlying IIB behavioral issues is expensive. 
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 07, 2021 9:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
What gbaddeley said.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
