| Author |
Message
|
| johanroux |
 Posted: Mon Nov 29, 2010 11:33 pm Post subject: MQ goes into "sleep mode". |
 |
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
Good day all, we have a problem at a customer where MQ V6 on zOS seems to go into a sort of sleep mode. There is a buildup of messages and MQ itself has very slow response on the both the panels and also to direct commands. It sometimes recovers by itself after 20 minutes or so. We haven't been able to find any issues in the MQ Setup. We finally found a possible match when we saw extremely slow response on the volumes used by the log files of the queue manager. From RMF we've determined that the connect time for the volumes seems to be the problem. This happens on various volumes at various times. The hardware guys have investigated and found no problems. We've even had a look at the actual VSAM log files to check for splits or anything else that could cause the slow response but found none. The log files are the only files on the actual volumes and are roughly 2.8GB in size. At peak load the files are switched every 15 minutes or so. Most of the queues used are persistent.
It doesn't appear to be related to load because the queue manager is not busy and the overall system is not busy - showing about 30% CPU usage. The biggest buildup of messages on any of the queues (not just 1 involved) was a few hundred messages. There was initially a problem with too little buffers which was resolved with first adding buffers and then applying PM22696 to resolve UK53282 which was previously applied. This resolved the issue with the buffers but not the initial problem.
The problem is already logged with the IBM labs but I would appreciate any thoughts or possible help with this. |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Tue Nov 30, 2010 1:02 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
do you share the volumes with other applications that may bring load to them? any problems with the log offload ?
_________________
Regards, Butcher |
|
| Back to top |
|
|
| johanroux |
| Posted: Tue Nov 30, 2010 1:14 am Post subject: |
|
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
| The log volumes are the only datasets on the actual volumes. 1 log dataset per volume. |
|
| Back to top |
|
|
| johanroux |
| Posted: Tue Nov 30, 2010 1:16 am Post subject: |
|
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
| They have no offload to ARCHLOG volumes although it is a production system. This is going to be rectified as part of an ongoing process. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 30, 2010 4:49 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Logs on Zos are beam lds, so no splits occur. Is this qmgr the only one in this lpar? If there is another qmgr, it is behaving in a similar manner?
Is this a new lpar? A new Zos instance?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| johanroux |
| Posted: Tue Nov 30, 2010 5:16 am Post subject: |
|
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
| It's not a new LPAR or zOS instance. This is the only QMGR on the LPAR. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 30, 2010 6:30 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Is this the first time anyone has used this qmgr?
Is this the first qmgr in this lpar? Is this a new version/release/modification-level of WMQ in this LPAR?
Did the system programmer that installed the WMQ software follow the steps in the System Setup manual?
Did the systems programmer perform the iSV (installation verification procedure)?
What has changed with this qmgr?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| johanroux |
| Posted: Tue Nov 30, 2010 6:49 am Post subject: |
|
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
| None of the above. The queue manager has been in use for years. Not a new release. Sizes of the log files changed around 6 weeks ago. No issues in the interim. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 30, 2010 6:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
So, in summary: this is an existing qmgr, in an existing LPAR, with an existing z/OS instance; in a zSeries server... and suddenly, only the qmgr is experiencing this slowdown?
If all other applications in the LPAR are experiencing a similar slowdown, I'd suspect that your sysprogs have under-provisioned the LPAR - taken away central storage and processors.
Is TSO response time (press ENTER key) sub-second? Is response time in the other ISPF panels sub-second? If yes, I'd expect sub-second response from the WMQ panels.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| johanroux |
| Posted: Tue Nov 30, 2010 7:31 am Post subject: |
|
|
Novice
Joined: 21 Apr 2010
Posts: 10
|
| Your summary is spot on. Except that it's only MQ that get's impacted. Panels, display commands to the qmgr. Messages building up etc. From what I've seen no other applications are impacted and MQ was changed to run at SYSSTC. No contention has been found on the system either. Like I said, this is a VERY strange one. Happened on Thursday and Friday last week and again yesterday. Nothing today. |
|
| Back to top |
|
|
| gouda |
| Posted: Tue Nov 30, 2010 7:37 am Post subject: |
|
|
Apprentice
Joined: 20 May 2001
Posts: 32
Location: Germany, Nuremburg
|
One option for search could be an EREP output, of course to be interpreted by the support team, what enforces a PMR.
The other is SupportPac MP1B. It will help you to find out latches.
Just generate SMF records for some minutes with 'start trace(a) class(3)' and let MP1B evaluate it.
Search in the printout for really big holes by "find '== Latch ' " .
Then look for the 'latch' section in the mp1bv7using.pdf .
It shows you where time is stolen by the latch class. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 30, 2010 9:06 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
While there was improvement in latch processing in v7, I doubt a lightly used qmgr like the one in the OP would experience such delays due to latch contention.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ramires |
| Posted: Tue Nov 30, 2010 10:13 am Post subject: |
|
|
Knight
Joined: 24 Jun 2001
Posts: 523
Location: Portugal - Lisboa
|
|
| Back to top |
|
|
| Gouldmar |
| Posted: Wed Dec 01, 2010 1:56 pm Post subject: |
|
|
Novice
Joined: 03 May 2005
Posts: 11
Location: Munich, Germany
|
If I read the above correct you have increased the size of the log files from one size up to 2.8gb in size, were the logs previously set to the default size as supplied in the sample install jobs? If not, what was the previous log size and how many active logs are defined in the queue manager?
How often does the queue manager perform checkpoints, is it only when the active log switches?
Do you see the issue of a stall as part of a log switch, before a log switch or after a log switch?
As part of the log resize, was the LOGLOAD parameter in CSQ6SYSP reviewed, if so what was it previously set to and what is it set to now? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 01, 2010 3:47 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
To be more specific, look at the SYSOUT from the qmgr in question for a day that the slowdown occurred.
What events/errors/messages just before and during the same time-period as the slowdown of 20 minutes? Post here.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
