ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » MQ Channel slows down every 10 minutes

Post new topic  Reply to topic Goto page 1, 2  Next
 MQ Channel slows down every 10 minutes « View previous topic :: View next topic » 
Author Message
PeterPotkay
PostPosted: Tue Feb 22, 2005 12:19 pm    Post subject: MQ Channel slows down every 10 minutes Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

QMA on ServerA has a SNDR channel to QMB on ServerB.

Every 10 minutes, on the 10 minutes, for about 50 seconds, the XMITQ on QMA backs up, and then it opens up and all the messages flow fine until the next 10 minute mark.


I started up repeated pins from various places into ServerB, and the pings look fine <1ms until that 10 minute mark. Then the pings take >80ms.

So I think MQ is a victim.


WHat do I tell these people to look at. I keep saying find out why the pings take >80ms every 10 minutes for 50 seconds. Solve that and MQ will be fine.


We are stuck here and going nowhere. What do you guys recomend I suggest to the 30 people on this conferance call? Every 10 minutes, all our MQ messages into QMB get delayed for 50 or so seconds.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Tue Feb 22, 2005 1:19 pm    Post subject: Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

Is Transaction Vision doing active monitoring of MQ every ten minutes?
_________________
I am *not* the model of the modern major general.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Feb 22, 2005 1:29 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

TV is always on, recording everything that comes thru. I don't think it does anything only every 10 minutes.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Tue Feb 22, 2005 1:37 pm    Post subject: Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

PeterPotkay wrote:
TV is always on, recording everything that comes thru. I don't think it does anything only every 10 minutes.
It's typical in MQ monitoring for certain metrics to be gathered at regular interfals, like queue statistics, because MQ doesn't provide an event driven mechanism to gather them.

Granted, TV could have an entire suite of exits that allow it to bypass the "normal" mechanisms.

To determine if it's MQ or the network, you can try redirecting traffic from QMA to get to QMB by way of QMC - that uses a different network route.

I assume you've done things like reconcile your channel configuration against the problem (making sure your HBINT isn't ten minutes, that kind of thing).

And your ping tests do tend to rule out MQ - but it never hurts to be a bit paranoid.

The interval very much says to me that it's some kind of monitoring interfereance.

Also, your QM isn't using SAN disk space, is it? I remember some one a few months back who had weird issues like this that was finally nailed down to SAN implementation interference.
_________________
I am *not* the model of the modern major general.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Feb 22, 2005 1:41 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

That was me with the SAN, and no, this server has no SAN.


Just to be sure I stopped TV entirely in the QA environment, and still this occours. 2 10 minutes cycles already.

At 5 PM, we are stopping MQ completly on the server for 30 minutes to see if the behaviour goes away.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
csmith28
PostPosted: Tue Feb 22, 2005 1:43 pm    Post subject: Reply with quote

Grand Master

Joined: 15 Jul 2003
Posts: 1196
Location: Arizona

If you have friends in the Network group you could put a sniffer on ServerA and ServerB for a few minutes each and see what they can get by analyzing the contents of the buffer.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Feb 22, 2005 1:45 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

OK, the network guys have put the sniffer on an hour ago.

They see requests coming into the ServerB box (sniffer on the ServerB box),
but they don't see the 002 box responded(no TCP level acknowledgement).

Another thing, there are 2 boxes on our environment that are acting the
exact same way. One is the QA version of the other. Every other server is
OK.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Tue Feb 22, 2005 1:49 pm    Post subject: Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

Is there common hardware involved - either "same type" or "same equipment"? Like all three affected boxes are on the same switch/hub, or all three are the same brand of box or using the same brand and make of NIC, or etc?

Is there any monitoring of the box other than TV? Like something monitoring SNMP traps, or hardware utilization?
_________________
I am *not* the model of the modern major general.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Feb 22, 2005 1:50 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

Yup, we are on that path. For the life of us we cannot figure out what these 2 servers and only these 2 servers have in common that would cause something like this.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Feb 22, 2005 3:24 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

MQ shutdown, server rebooted, server back up with no MQ, and still the pings are slow on the 10 minute mark.



At least they stopped blaming MQ and now realize MQ is handling the situation as best it can (no messages lost, just delayed) given the lack of a stable connection.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bower5932
PostPosted: Tue Feb 22, 2005 6:58 pm    Post subject: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

PeterPotkay wrote:
At least they stopped blaming MQ and now realize MQ is handling the situation as best it can (no messages lost, just delayed) given the lack of a stable connection.


Isn't handling network problems for your application what WMQ is all about.
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
csmith28
PostPosted: Tue Feb 22, 2005 10:19 pm    Post subject: Reply with quote

Grand Master

Joined: 15 Jul 2003
Posts: 1196
Location: Arizona

Quote:
Isn't handling network problems for your application what WMQ is all about.


Indeed, as WMQ Admins we are expected to troubleshoot, application problems, hardware, firmware/driver problems, network problems, firewall problems,... you name it.

Like I said before, if there is an "M" and a "Q" in the error output and in some cases even if there isn't. The title of the ticket is "Problem with MQ" or some such.

That said, Peter what are you doing to resolve this issue?
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial.
Back to top
View user's profile Send private message
Michael Dag
PostPosted: Wed Feb 23, 2005 1:25 am    Post subject: Reply with quote

Jedi Knight

Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)

as usual the one that deals with all components (MQ...) get's blamed.
It's almost a fact of life we all have to live with.

Peter, do let us know what it turned out to be... 10 minutes = 600 seconds = 600000 milliseconds. 'something' must be doing something at these intervals...
_________________
Michael



MQSystems Facebook page
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
kevinf2349
PostPosted: Wed Feb 23, 2005 5:41 am    Post subject: Reply with quote

Grand Master

Joined: 28 Feb 2003
Posts: 1311
Location: USA

Peter

Do you use a security password synchronization product? I have seen ours slow things down from time to time.

Has this just started happening or has it been happening for a while but only just now been narrowed down?
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Feb 23, 2005 5:43 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

It's not the BATCHINT by any chance?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » MQ Channel slows down every 10 minutes
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.