ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportAdditional Instances - Stupid Question Time

Post new topicReply to topic
Additional Instances - Stupid Question Time View previous topic :: View next topic
Author Message
PeterPotkay
PostPosted: Thu Dec 20, 2018 4:33 pm Post subject: Additional Instances - Stupid Question Time Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7507

OK, this seems like a dumb question, but I gotta ask. Cuz someone asked me and I couldn't come up with a drop-the-mic answer.

Given that IIB will startup additional instances of your flow as needed (up to what you configured the Message Flow's Additional Instances property), and it shut downs idle threads when they are no longer needed, why not just set Additional Instances to 256 on every flow and be done with it? Why try to come up with the ideal # of additional instances? Why not max out the setting and let Mr. BIP handle it, flexing as needed when volume arrives, shutting down idle threads when volume slows down?

Then only thing I could think of is if the flow had connections to other systems like databases or HTTPS end points, flexing up to 256 could overwhelm that back end system with too many connections. But if they agree to allow up to 256 IIB connections, I assume this eliminates this as a reason. Any others?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
missing_link
PostPosted: Fri Dec 21, 2018 1:08 am Post subject: Reply with quote

Acolyte

Joined: 08 Jan 2004
Posts: 55

Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that
Back to top
View user's profile Send private message
gfrench
PostPosted: Fri Dec 21, 2018 2:10 am Post subject: Reply with quote

Acolyte

Joined: 10 Feb 2002
Posts: 68

You might not want a flow that processes big messages running in lots of threads and consuming vast quantities of memory. Especially as the memory is not released by execution group.

A big flood of large messages arriving at once into a big flow with unrestricted threads might see a huge increase in memory and then not get used again.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
PeterPotkay
PostPosted: Fri Dec 21, 2018 5:10 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7507

Another couple things I thought of since:

If your IIB Node doesn't have unlimited CPU capacity, a huge volume of rapidly arriving MQ messages on one or more queues would cause one or more flows to flex up to 256 instances each. All of them would be competing for CPU, potentially to the point of starving the system of CPU causing other unrelated innocent message flows on the same system to suddenly experience performance issues. Versus the situation if there was a more reasonable # of instances - in that case the messages would safely queue up and be processed without that flow or flows "taking over" the system.

For HTTP Input based flows, there is a setting that controls how many total HTTP connections the Broker will allow concurrently. If you allowed all your HTTP flows to flex up to 256 instances, it means the system would reach the total HTTP connections limit sooner, again impacting other innocent unrelated message flows sharing the same system.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Dec 21, 2018 5:12 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7507

missing_link wrote:
Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that


What happened there? The additional instances exacerbated a problem with contention on the database end, causing all instances to perform worse?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Dec 21, 2018 5:14 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7507

gfrench wrote:
You might not want a flow that processes big messages running in lots of threads and consuming vast quantities of memory. Especially as the memory is not released by execution group.

A big flood of large messages arriving at once into a big flow with unrestricted threads might see a huge increase in memory and then not get used again.


Excellent point. The Broker may shut down idle threads once they are no longer needed, but the memory they needed in the Execution Group while they were running wouldn't be released. The longer the system runs, the more Execution Groups on your system reach this state, potentially hitting the memory limits of the server. And all that assumes you don't run into issue the first time the first flow that needs a lot of memory doesn't blow up when 256 threads all at once want all that memory.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
missing_link
PostPosted: Fri Dec 21, 2018 5:53 am Post subject: Reply with quote

Acolyte

Joined: 08 Jan 2004
Posts: 55

[quote="PeterPotkay"][quote="missing_link"]Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that [/quote]

What happened there? The additional instances exacerbated a problem with contention on the database end, causing all instances to perform worse?[/quote]

Pretty much. I could see that from my own prior tests that more x number of instances started to slow the overall throughput. So if the db was already struggling, throwing more stuff at wasn't going to help!!
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Dec 21, 2018 6:05 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25485
Location: Ohio, USA

PeterPotkay wrote:
If your IIB Node doesn't have unlimited CPU capacity, a huge volume of rapidly arriving MQ messages on one or more queues would cause one or more flows to flex up to 256 instances each. All of them would be competing for CPU, potentially to the point of starving the system of CPU causing other unrelated innocent message flows on the same system to suddenly experience performance issues. Versus the situation if there was a more reasonable # of instances - in that case the messages would safely queue up and be processed without that flow or flows "taking over" the system.





We group flows for multiple applications on a single server (because we don't have unlimited PVUs any more than we have unlimited CPU) and we have a specific check built into our deployment pipeline that flags any flow where the additional instances are set to more than 20, and additional (manual) authorization is needed for the deploy to proceed.

Yes, there's a whole raft of stuff doing capacity planning for how many applications go on which server, when we need to divide one server into two and all of that creamy goodness. It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Dec 21, 2018 6:18 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7507

Vitor wrote:
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....


How so?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Dec 21, 2018 6:36 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25485
Location: Ohio, USA

PeterPotkay wrote:
Vitor wrote:
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....


How so?


Because we'll give everyone their own container and they can do what they want in it, while we all go outside and frolic with the unicorns.

Apparently containers in clouds fix everything.

According to the business.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Dec 21, 2018 7:16 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25485
Location: Ohio, USA

Vitor wrote:
PeterPotkay wrote:
Vitor wrote:
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....


How so?


Because we'll give everyone their own container and they can do what they want in it, while we all go outside and frolic with the unicorns.

Apparently containers in clouds fix everything.

According to the business.


Ok, that's unfair and proof you should never post angry.

Once we have our container-based cloud architecture in place it'll be easier to organize applications and control their resource usage. It'll also be easier to laugh at them when they set additional instances to 256 and blow themselves up.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Dec 21, 2018 3:18 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19898
Location: LI,NY

Vitor wrote:


Ok, that's unfair and proof you should never post angry.

Once we have our container-based cloud architecture in place it'll be easier to organize applications and control their resource usage. It'll also be easier to laugh at them when they set additional instances to 256 and blow themselves up.

No you won't. They'll blame it on the product not being stable as Kubernetes will constantly have to initiate a new container as the old one blew up...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportAdditional Instances - Stupid Question Time
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.