| Author |
Message
|
| Kireeti84 |
 Posted: Fri Mar 04, 2016 10:37 am Post subject: large volume of messages sitting in the queue |
 |
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
Hi All, I need your help to troubleshoot the below issue.
We have a interface which runs in 2 brokers. both the brokers configuration is same. code runs in eg ABC in both the brokers. So the problem the flow running under broker 1 processes all the messages without any issue but the flow running under broker 2 is not able process the messages, so we have to restart the execution group ( under broker 2 ) every time in order to drain all the messages out. The message flow has a mq input node. please share you thoughts and ideas.
Thanks for the help.
Kireeti |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Mar 04, 2016 10:41 am Post subject: Re: large volume of messages sitting in the queue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Kireeti84 wrote: |
| please share you thoughts and ideas. |
- The code isn't really the same
- The messages being delivered to broker 2 have unprocessable ("poison") messages more often than those to broker 1
- messages being delivered to broker 2 are not being committed properly
- etc
- etc
What does the user trace from broker 2 say?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ganesh |
| Posted: Fri Mar 04, 2016 11:00 am Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Do the 2 execution groups have the exact number of flows and are all the flows the same? |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Fri Mar 04, 2016 11:12 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
Exactly the same. no difference. i just have one message flow running in the EG.
Thanks
Kireeti |
|
| Back to top |
|
|
| ganesh |
| Posted: Fri Mar 04, 2016 11:15 am Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Kireeti84 wrote: |
Exactly the same. no difference. i just have one message flow running in the EG.
Thanks
Kireeti |
Check the logs to see if there are any messages related to the faulting flow or execution group in question.
It could be as simple as the flows are not running or something caused a failure causing the EG to be in a hung state.
The logs should have some information. |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Fri Mar 04, 2016 11:47 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
- The code isn't really the same
Ans : Code is the same. The same bar file got deployed in both the brokers.
- The messages being delivered to broker 2 have unprocessable ("poison") messages more often than those to broker 1
Ans : If that's the case why the poison message goes to broker2 all the time ? we are having issues only with broker 2.
- messages being delivered to broker 2 are not being committed properly.
how to chcek this ?
- etc
- etc |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Mar 04, 2016 11:58 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The environment in broker 2 is not the same as in broker 1:
java configurable classloader
global jar files
eg jar files
etc ...
network/firewall conditions are not the same (connection to DB, connection to any other external resources).
Broker 2 would need the use of a different http proxy than broker 1 ?
Broker 2 not having the same client / server certs as broker 1...
Broker 2 not having the same memory / cpu / speed as broker 1
Lots of other reasons... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 04, 2016 12:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Kireeti84 wrote: |
- The code isn't really the same
Ans : Code is the same. The same bar file got deployed in both the brokers. |
Sure? I mean so really, really sure you don't need to actually check with the brokers what code they're running? Despite the code being different being a really probably reason for the 2 brokers behaving differently?
I admire your confidence
| Kireeti84 wrote: |
- The messages being delivered to broker 2 have unprocessable ("poison") messages more often than those to broker 1
Ans : If that's the case why the poison message goes to broker2 all the time ? we are having issues only with broker 2. |
Bad luck? I've seen weirder. I agree it's not a highly plausible scenario, unless the 2 code versions differ and broker 2 code can't handle a specific and common message type.
Again, I admire your confidence.
| Kireeti84 wrote: |
- messages being delivered to broker 2 are not being committed properly.
how to chcek this ? |
Ask the queue manager.
What turned up in the user trace?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 04, 2016 12:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
The environment in broker 2 is not the same as in broker 1:
java configurable classloader
global jar files
eg jar files
etc ...
network/firewall conditions are not the same (connection to DB, connection to any other external resources).
Broker 2 would need the use of a different http proxy than broker 1 ?
Broker 2 not having the same client / server certs as broker 1...
Broker 2 not having the same memory / cpu / speed as broker 1
Lots of other reasons... |
Hence my suggestion of a trace.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Mon Mar 07, 2016 6:16 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
| Thanks Vitor. My Admin team checked all your points and they confirmed both the brokers have same kind of configuration. User trace is not providing us any clue. FInally we opened up a PMR with IBM. Sent them the user trace and PI. Thanks everyone. meanwhile please share if there are any other troubleshooting techniques. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 07, 2016 6:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Kireeti84 wrote: |
| My Admin team checked all your points and they confirmed both the brokers have same kind of configuration. |
I wasn't asking about the configuration (and the same kind is not the same IMHO), I was asking about the deployed code base.
| Kireeti84 wrote: |
| User trace is not providing us any clue. |
If it's not providing any clue, that's a clue by itself.
| Kireeti84 wrote: |
| FInally we opened up a PMR with IBM. |
It's why you pay the license fee.
Please let us know what the outcome is.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Mar 07, 2016 7:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And you're sure there is no full queue on broker 2?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Mon Mar 07, 2016 8:21 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
@ vitor. the deployed code base is exactly the same. We re deployed the code last week and same bar file deployed in both the brokers.
Thanks,
Kireeti |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Mon Mar 07, 2016 11:45 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
@ fjb_saper - Yes no full queue.
we have done all the possible troubleshooting that an average developer or admin can think of  |
|
| Back to top |
|
|
|
|
