| Author |
Message
|
| krypton |
 Posted: Fri Jun 04, 2010 6:25 pm Post subject: Increasing the instances of a Message Flow |
 |
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
We have a flow(with JMSInput Node) and which is picking the messages from back end queue, sometimes (5% of the cases), what is happening is that there has been a delay encountered by this flow and the application which is expecting a response would timed out. We can not increase the waiting time for the application as it is already on threshold.
Does, increasing the instances of that flow with(JMSInput node) would help in faster processing(we are not bothered about the order/sequence of the messages here).
My doubt here is, if a flow is encountering a delay(sometimes), how come the other instances (also that particular time) would not behave in the same way. Or is it possible that one instance is delayed then other instance might be faster to pick up the messages?
Please suggest something.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Jun 04, 2010 6:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Increasing the number of instances of the flow will indeed increase the throughput capability. However if you have a problem in the service replying to your request, pretty much all instances will be affected...
If you are making heavy use of dynamic temporary queues, talk to IBM for an optimized pooling.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| krypton |
| Posted: Fri Jun 04, 2010 11:16 pm Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
[b]fjb_saper wrote[/b]
| Quote: |
Increasing the number of instances of the flow will indeed increase the throughput capability. However if you have a problem in the service replying to your request, pretty much all instances will be affected...
If you are making heavy use of dynamic temporary queues, talk to IBM for an optimized pooling. |
Thanks saper, no we are not making use of any dynamic temporary queues, we have a simple flow which use JMSInput Node to picks up the messages and then converting to MQ header(using JMSToMQHeader node)and putting the message into the local MQ. Most of the time this happened within a time span of 1 second, but sometimes this flow takes as much as 10 seconds to put the message in queue. So, I was thinking may be if we run multiple instances of this flow then if one is busy (or sleeping), the other can pick that message.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Jun 05, 2010 5:36 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| krypton wrote: |
| So, I was thinking may be if we run multiple instances of this flow then if one is busy (or sleeping), the other can pick that message. |
Message Flows don't sleep and ignore new work, unless you coded them to do that.
If the message flow is too busy to take on new work, you have to find out why. Simply adding more instances may just have them all waiting in line for the resource that is keeping the 1st instance busy (lack of CPU, lack of I/O, lack of a backend resource like a DB, etc). Or you may find that adding more instances is the perfect solution.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| krypton |
| Posted: Sat Jun 05, 2010 6:08 pm Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
PeterPookay wrote:
| Quote: |
| If the message flow is too busy to take on new work, you have to find out why. Simply adding more instances may just have them all waiting in line for the resource that is keeping the 1st instance busy (lack of CPU, lack of I/O, lack of a backend resource like a DB, etc). Or you may find that adding more instances is the perfect solution. |
as it is happening very infrequently it is very hard to simulate this problem(I tried many times on my end and it never happened) but then sometimes when the test team is doing System Testing it happens(and i get a sev2 defect), well I can bet my luck on increasing the instances of a flow as there seems to be no other solution in sight.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Sat Jun 05, 2010 7:18 pm Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
this is strange, does it looks like some code issue here
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 06, 2010 1:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| krypton wrote: |
PeterPookay wrote:
| Quote: |
| If the message flow is too busy to take on new work, you have to find out why. Simply adding more instances may just have them all waiting in line for the resource that is keeping the 1st instance busy (lack of CPU, lack of I/O, lack of a backend resource like a DB, etc). Or you may find that adding more instances is the perfect solution. |
as it is happening very infrequently it is very hard to simulate this problem(I tried many times on my end and it never happened) but then sometimes when the test team is doing System Testing it happens(and i get a sev2 defect), well I can bet my luck on increasing the instances of a flow as there seems to be no other solution in sight. |
I'm siding with Peter on this one.
Some avenues you'd need to check:
- Have the DB folks check the most expensive queries hitting the table(s) you access in the flow right when it happens
- missing indexes on the DB will influence the behavior (page lock by another process missing an index)
- Verify batch processes accessing the DB and make sure they run fast and have no missing indexes
- check for batches that run regularly in prod but are "on demand" in your test / qa environment
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| prasad.edlabadkar |
| Posted: Wed Jun 16, 2010 5:36 am Post subject: |
|
|
Novice
Joined: 10 Feb 2006
Posts: 21
Location: Pune, India
|
| krypton wrote: |
[b]fjb_saper wrote[/b]
| Quote: |
Increasing the number of instances of the flow will indeed increase the throughput capability. However if you have a problem in the service replying to your request, pretty much all instances will be affected...
If you are making heavy use of dynamic temporary queues, talk to IBM for an optimized pooling. |
Thanks saper, no we are not making use of any dynamic temporary queues, we have a simple flow which use JMSInput Node to picks up the messages and then converting to MQ header(using JMSToMQHeader node)and putting the message into the local MQ. Most of the time this happened within a time span of 1 second, but sometimes this flow takes as much as 10 seconds to put the message in queue. So, I was thinking may be if we run multiple instances of this flow then if one is busy (or sleeping), the other can pick that message. |
Hi,
If a flow is taking upto 10 seconds, increasing number of instances may not help.
Try using support pack IS02 (http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24012457) to see which part of the message flow is taking more time. Also, you would like to take a look at VMStat and IOStat to see there is usual resource consumption. You can ask your system administrators to run both of them for a day and then analyze.
_________________
Thanks and Regards
Prasad Edlabadkar
Tech Lead - Websphere Business Integration |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Jun 16, 2010 6:46 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| prasad.edlabadkar wrote: |
Try using support pack IS02 (http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24012457) to see which part of the message flow is taking more time. Also, you would like to take a look at VMStat and IOStat to see there is usual resource consumption. You can ask your system administrators to run both of them for a day and then analyze. |
May I humbly suggest that IS02 is NOT the answer to Life, the Universe and all Broker Throughput problems. 
There are plenty of other ways to get to the bottom of Flow Throughput problems. Another useful tool is ODBC tracing but again probably not in production.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 16, 2010 7:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
This sounds more like an MQ problem we were having and involving catalog availability of MQ. Open a PMR with IBM and reference this thread.
If your flow is really as simple as you portray it, there is no way that the time should be in the seconds... It should be done in about 100 ms. Now the question comes to the size of the messages that could wildly influence that guestimate (for messages under 10 K).
And remember if you are really only touching the headers, make sure the domain is BLOB and the parsing is set on demand.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| krypton |
| Posted: Sun Jun 20, 2010 2:07 am Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
smdavies wrote:
| Quote: |
May I humbly suggest that IS02 is NOT the answer to Life, the Universe and all Broker Throughput problems.
There are plenty of other ways to get to the bottom of Flow Throughput problems. Another useful tool is ODBC tracing but again probably not in production. |
How ODBC trace can help regarding this, we are not at all using the database in our Flow.
Am I missing something here? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Jun 20, 2010 5:27 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| krypton wrote: |
How ODBC trace can help regarding this, we are not at all using the database in our Flow.
Am I missing something here? |
No you are not missing anything. I was merely suggesting that there are other tools that can be used to get to the bottom of flow throughput issues.
If your flow uses external (to broker) systems such as a Database then isolating the system that is at the root of the problem is one of the approaches to take. I used ODBC tracing to isolate a very badly performing SQLServer System as the point where the problems were located rather than broker and the instances of the flow.
We fixed the SQLServer and the network links to it ( Full Duplex and stuff like AutoNegociate) and the problems went away.
This sort of investingation is very important when using SAP Systems. They can have very widely different perfomance with almost identical BAPI's.
As you are not using any DB connections then you can safely ignore that part of my previous post.
In essence, I was merely suggesting that while useful, ISO2 can only be part of the arsenal of tools that you use to get to the bottom of performance problems.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Rahul999 |
| Posted: Sun Jun 20, 2010 5:07 pm Post subject: |
|
|
Centurion
Joined: 14 Mar 2007
Posts: 134
|
| Quote: |
No you are not missing anything. I was merely suggesting that there are other tools that can be used to get to the bottom of flow throughput issues.
If your flow uses external (to broker) systems such as a Database then isolating the system that is at the root of the problem is one of the approaches to take. I used ODBC tracing to isolate a very badly performing SQLServer System as the point where the problems were located rather than broker and the instances of the flow.
We fixed the SQLServer and the network links to it ( Full Duplex and stuff like AutoNegociate) and the problems went away.
This sort of investingation is very important when using SAP Systems. They can have very widely different perfomance with almost identical BAPI's.
As you are not using any DB connections then you can safely ignore that part of my previous post.
In essence, I was merely suggesting that while useful, ISO2 can only be part of the arsenal of tools that you use to get to the bottom of performance problems.
_________________
WMQ User since 1999
Message Broker User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon) |
ISO2 can show you graphically the throughput of a flow, we usually use it for performance testing for our flows. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jun 21, 2010 12:43 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Rahul999 wrote: |
ISO2 can show you graphically the throughput of a flow, we usually use it for performance testing for our flows. |
That is great where you can get a GUI connection to the systems under test. There are many environments where this is not permitted to UAT, pre-prod or Prod systems.
Hmmmm, just like the one where I'm working now...(plus a few others I've come upon in the past few years.
All I'm trying to say is that IS02 is but one tool to use out of many.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| prasad.edlabadkar |
| Posted: Tue Jun 22, 2010 12:51 am Post subject: IS02 |
|
|
Novice
Joined: 10 Feb 2006
Posts: 21
Location: Pune, India
|
| smdavies99 wrote: |
May I humbly suggest that IS02 is NOT the answer to Life, the Universe and all Broker Throughput problems.
|
It is not. I never said IS02 is the answer to the problem. But when you don't know which part of the flow is causing poor performance, it gives a good starting point. In one of my projects, I could determine that slower disk was a bottleneck by using IS02.
_________________
Thanks and Regards
Prasad Edlabadkar
Tech Lead - Websphere Business Integration |
|
| Back to top |
|
|
|
|
