| Author |
Message
|
| PeterPotkay |
 Posted: Thu Dec 20, 2018 4:33 pm Post subject: Additional Instances - Stupid Question Time |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
OK, this seems like a dumb question, but I gotta ask. Cuz someone asked me and I couldn't come up with a drop-the-mic answer.
Given that IIB will startup additional instances of your flow as needed (up to what you configured the Message Flow's Additional Instances property), and it shut downs idle threads when they are no longer needed, why not just set Additional Instances to 256 on every flow and be done with it? Why try to come up with the ideal # of additional instances? Why not max out the setting and let Mr. BIP handle it, flexing as needed when volume arrives, shutting down idle threads when volume slows down?
Then only thing I could think of is if the flow had connections to other systems like databases or HTTPS end points, flexing up to 256 could overwhelm that back end system with too many connections. But if they agree to allow up to 256 IIB connections, I assume this eliminates this as a reason. Any others?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| missing_link |
| Posted: Fri Dec 21, 2018 1:08 am Post subject: |
|
|
Acolyte
Joined: 08 Jan 2004
Posts: 60
|
Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that  |
|
| Back to top |
|
|
| gfrench |
| Posted: Fri Dec 21, 2018 2:10 am Post subject: |
|
|
Acolyte
Joined: 10 Feb 2002
Posts: 71
|
You might not want a flow that processes big messages running in lots of threads and consuming vast quantities of memory. Especially as the memory is not released by execution group.
A big flood of large messages arriving at once into a big flow with unrestricted threads might see a huge increase in memory and then not get used again. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 21, 2018 5:10 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Another couple things I thought of since:
If your IIB Node doesn't have unlimited CPU capacity, a huge volume of rapidly arriving MQ messages on one or more queues would cause one or more flows to flex up to 256 instances each. All of them would be competing for CPU, potentially to the point of starving the system of CPU causing other unrelated innocent message flows on the same system to suddenly experience performance issues. Versus the situation if there was a more reasonable # of instances - in that case the messages would safely queue up and be processed without that flow or flows "taking over" the system.
For HTTP Input based flows, there is a setting that controls how many total HTTP connections the Broker will allow concurrently. If you allowed all your HTTP flows to flex up to 256 instances, it means the system would reach the total HTTP connections limit sooner, again impacting other innocent unrelated message flows sharing the same system.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 21, 2018 5:12 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| missing_link wrote: |
| Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that |
What happened there? The additional instances exacerbated a problem with contention on the database end, causing all instances to perform worse?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 21, 2018 5:14 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| gfrench wrote: |
You might not want a flow that processes big messages running in lots of threads and consuming vast quantities of memory. Especially as the memory is not released by execution group.
A big flood of large messages arriving at once into a big flow with unrestricted threads might see a huge increase in memory and then not get used again. |
Excellent point. The Broker may shut down idle threads once they are no longer needed, but the memory they needed in the Execution Group while they were running wouldn't be released. The longer the system runs, the more Execution Groups on your system reach this state, potentially hitting the memory limits of the server. And all that assumes you don't run into issue the first time the first flow that needs a lot of memory doesn't blow up when 256 threads all at once want all that memory.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| missing_link |
| Posted: Fri Dec 21, 2018 5:53 am Post subject: |
|
|
Acolyte
Joined: 08 Jan 2004
Posts: 60
|
[quote="PeterPotkay"][quote="missing_link"]Definitely back end limitations. At a previous client when trying to sort out performance issues (which were clearly database related) we upped the IIB flow instances, retested and the overall performance was worse. Always nice being proved right with things like that [/quote]
What happened there? The additional instances exacerbated a problem with contention on the database end, causing all instances to perform worse?[/quote]
Pretty much. I could see that from my own prior tests that more x number of instances started to slow the overall throughput. So if the db was already struggling, throwing more stuff at wasn't going to help!! |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 21, 2018 6:05 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| If your IIB Node doesn't have unlimited CPU capacity, a huge volume of rapidly arriving MQ messages on one or more queues would cause one or more flows to flex up to 256 instances each. All of them would be competing for CPU, potentially to the point of starving the system of CPU causing other unrelated innocent message flows on the same system to suddenly experience performance issues. Versus the situation if there was a more reasonable # of instances - in that case the messages would safely queue up and be processed without that flow or flows "taking over" the system. |
We group flows for multiple applications on a single server (because we don't have unlimited PVUs any more than we have unlimited CPU) and we have a specific check built into our deployment pipeline that flags any flow where the additional instances are set to more than 20, and additional (manual) authorization is needed for the deploy to proceed.
Yes, there's a whole raft of stuff doing capacity planning for how many applications go on which server, when we need to divide one server into two and all of that creamy goodness. It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 21, 2018 6:18 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....
|
How so?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 21, 2018 6:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| Vitor wrote: |
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....
|
How so? |
Because we'll give everyone their own container and they can do what they want in it, while we all go outside and frolic with the unicorns.
Apparently containers in clouds fix everything.
According to the business.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 21, 2018 7:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Vitor wrote: |
| PeterPotkay wrote: |
| Vitor wrote: |
It'll all be easier when we finish the private cloud and have Docker containers as far as the eye can see.....
|
How so? |
Because we'll give everyone their own container and they can do what they want in it, while we all go outside and frolic with the unicorns.
Apparently containers in clouds fix everything.
According to the business. |
Ok, that's unfair and proof you should never post angry.
Once we have our container-based cloud architecture in place it'll be easier to organize applications and control their resource usage. It'll also be easier to laugh at them when they set additional instances to 256 and blow themselves up.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 21, 2018 3:18 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
Ok, that's unfair and proof you should never post angry.
Once we have our container-based cloud architecture in place it'll be easier to organize applications and control their resource usage. It'll also be easier to laugh at them when they set additional instances to 256 and blow themselves up. |
No you won't. They'll blame it on the product not being stable as Kubernetes will constantly have to initiate a new container as the old one blew up... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
