ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Mainframe, CICS, TXSeries » MQ for z/OS QMGR stalls during startup after IPL

Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next
 MQ for z/OS QMGR stalls during startup after IPL « View previous topic :: View next topic » 
Author Message
bruce2359
PostPosted: Thu Mar 05, 2020 10:45 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

TXM751 wrote:
Pretty simple

Nothing about z/OS, MQ for z/OS, or QueueSharingGroups is pretty simple.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
TXM751
PostPosted: Thu Mar 05, 2020 10:56 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

Poobah .. you are not being helpful

I have multiple clients. I have many many QMGR's on many many LPAR's. I implemented Queue sharing more than 5 years ago

I am a SYSPROG with 35+ years of experience with CICS, DB2, MQ and every iteration of OS going back to before OS/390

I have tried everything to solve this problem .. looked in every log, turned over every rock ...

I posted on this page to see if anyone else had this problem .. clearly you have not and I'm not finding your input helpful ... If you haven't had this problem, then don't bother commenting on this post further .. you aren't helping me.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 05, 2020 11:33 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Sorry if I offended. I had not intended to do so. Your low post count lead me to believe, incorrectly so, that you were new to some or all of this.

Like Morag, I'm surprised that IBM would brush you off as you describe. Re-open the issue with IBM severity-1.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Thu Mar 05, 2020 2:35 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Have you got the sysprogs on board? Started a GTF trace for all the MQ events?
_________________
Glenn
Back to top
View user's profile Send private message
TXM751
PostPosted: Fri Mar 06, 2020 3:58 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

Reply to gbaddeley ...

First of all .. I am a SYSPROG with 35 years experience with MVS (or whatever you want to call it through the years), CICS, DB2, MQSeries, WAS etc. etc.. Second .. I have involved numerous other groups/sysprogs. Our first thought was that it was related to message flooding, but that has been turned off and it is not the cause of the problem.

We have investigated the Coupling Facility Structures. They aren't the problem. DB2 is not the problem.

Second .. The problem occurs immediately following an IPL .. MQ isn't even up yet and I haven't even gotten a chance to turn a GTF trace on. There really aren't any MQ events

I am going to follow IBM's most recent recommendations for Problem determination requirements (dumps, traces .. etc).

The question I have is - Has anyone else had this problem before

Symptoms are: It started after implementing Queue Sharing. It only affects 2 QMGR's that are members of 2 DIFFERENT QSG's, but the SAME DB2 Datasharing Group ... The QMGR's affected are on the same LPAR. The members of the 2 QSG's on the other LPAR have not been affected to date. The LPAR that is IPL'ed first is the one that is having the problem with it's QMGR's. The second LPAR is IPL'ed after the first one is up and those QMGR's have no problems.

The problem does not impact the members of either QSG on the second LPAR. The problem does not affect any other QMGR's in any of our other Queue Sharing Groups on any of our other LPAR's. I do not have this problem for a second client that I support who has a large number of QMGR's and Queue Sharing groups on Multiple LPAR's (I implemented Queue Sharing for this client more than 5 years ago).

A restart of the affected QMGR's always results in the QMGR's starting successfully.

We are going to attempt to delay the startup of the affected QMGR's by about 2 minutes this coming weekend to ensure that DB2 is fully up before the QMGR's start.

I am going to follow IBM's suggestions for diagnosis the next time this happens

Tom
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Mar 06, 2020 5:46 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Please excuse my zOS ignorance. I am seeing that you are willing to add some delay between db2 start and MQ start to see if it has a bearing on the problem.
I am going to ask maybe a meaningless question, but have you made sure that the Coupling Facility is up and fine before starting MQ?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
TXM751
PostPosted: Fri Mar 06, 2020 5:59 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

Reply to fjb_saper

If the coupling facility is not up and fine, then we have a MUCH MUCH greater problem than MQ not starting

But .. YES ... the CF is there (and running) and the required MQ (and DB2) CF Structures are defined and fine ...

I am beginning to suspect a timing issue .. some sort of ENQ problem that is causing MQ to lose it's mind or go to sleep

I am hoping that putting a 90+ second delay in the startup of MQ does the trick, but if it fixes the problem, it doesn't tell me where the problem was

Tom
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Mar 06, 2020 8:20 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

TXM751 wrote:
Poobah .. you are not being helpful
I posted on this page to see if anyone else had this problem .. clearly you have not and I'm not finding your input helpful ... If you haven't had this problem, then don't bother commenting on this post further .. you aren't helping me.

An unfortunate attitude for someone seeking free help from strangers.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
TXM751
PostPosted: Fri Mar 06, 2020 8:30 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

LOL .. you just couldn't resist eh poobah
Back to top
View user's profile Send private message
rekarm01
PostPosted: Fri Mar 06, 2020 9:19 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1415

Novice wrote:
Poobah .. you are not being helpful

Just a side note here: "Poobah" refers to a user's rank, not a user's name. It is based on the number of posts submitted.

To illustrate, TXM751's rank changed from "Newbie" to "Novice", after submitting 10 posts.
Back to top
View user's profile Send private message
TXM751
PostPosted: Fri Mar 06, 2020 9:26 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

Thanks rekarm01
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Mar 09, 2020 2:55 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Hi Tom. You have a very interesting case that I don't think anyone here has encountered. We are keenly interested in following the results of your investigation and ultimate resolution.
_________________
Glenn
Back to top
View user's profile Send private message
TXM751
PostPosted: Tue Mar 10, 2020 3:12 am    Post subject: Reply with quote

Novice

Joined: 04 Mar 2020
Posts: 14

Hi Glenn,

IBM says that it is message flooding, but we ruled this out by turning off message flooding. Despite telling IBM that it isn't message flooding, they have come back and told us it's message flooding again.

I will note that DB2 must be up for Queue Sharing members to start ... I noticed that there was a 90+ second pause during startup of the QMGR while it waited for DB2 initialization to complete. This is not unique to any of the QMGR's in any of our QSG's (they all have this pause), but just for giggles, we put a relationship in automation to wait for the DB2 initialization message before starting MQ ... I don't believe that this is the cause of the problem, but this past weekend we did not encounter the Hang issue ...

I will monitor for the next month or so ...

Tom
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Mar 10, 2020 5:12 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Sounds like you found your culprit
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Tue Mar 10, 2020 8:37 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

fjb_saper wrote:
Sounds like you found your culprit

Some background in IPL timing. As with everything z/OS, it's complicated.

At z/OS IPL, two members of PARMLIB "flood" the system with address space startup procs. IEACMDxx member is IBM-supplied, and it starts system support services address spaces. COMMNDxx member is used to start user support services address spaces. "Flood" is literal here, as no order is implied or imposed as to which address spaces complete their initialization ahead of others.

One of these address spaces is VTAM. VTAM provides networking services for both hardware devices and software (logical) network components. DB2 is one of the latter. The more network nodes it needs to initialize, the longer it will take to complete its initialization.

CICS, MQ and other user support address spaces initialize quickly, but they depend on VTAM having completed its initialization. Startup procs for these address spaces usually have as their STEP1 a "wait 90 seconds" process to allow VTAM to complete its initialization.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » Mainframe, CICS, TXSeries » MQ for z/OS QMGR stalls during startup after IPL
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.