ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » HA queue manager behaviour

Post new topic  Reply to topic Goto page 1, 2  Next
 HA queue manager behaviour « View previous topic :: View next topic » 
Author Message
emiranda
PostPosted: Fri Jun 21, 2019 10:23 am    Post subject: HA queue manager behaviour Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

Howdy!

We are using 8 MQ Appliances (9.1.0.1), configured as 4 pairs for HA. We've experienced a partitioned status on a number of the HA queue manager, in different appliances, at different times, but as there are no messages about this condition, the queue managers kept running for some time (its unclear for how long, as it's unclear how they came into partitioned). Eventually we realised the condition and resolved them as per documentation.

Days later, during an OAT test in one appliance, we noticed one channel went missing in one queue manager after the failover. We scratched our heads to understand what happened, and even questioned whether we'd created it in the first place. When the queue manager failedover back to its preferred appliance, the channel was there.

When we realised it, we ran some tests with simple putter/getter application and we got some duplicated messages and lost others.

Our conclusion was that, when/if a queue manager is in partitioned status and the appliance where it is running goes down for any reason, the queue manager will failover to the other appliance, regardless of its partiotioned status, compromising data integrity!

Did anybody test or came across this situation?
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sat Jun 22, 2019 12:03 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

That's because you are supposed to monitor and resolve such a partitioned condition and not let it go on until it bites you in the rear.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
emiranda
PostPosted: Mon Jun 24, 2019 1:08 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

fjb_saper wrote:
That's because you are supposed to monitor and resolve such a partitioned condition and not let it go on until it bites you in the rear.
Yeah, we know that, thanks. We also know that we can get really smart about scripting around the appliance in order to fix what seems not to have been delivered by IBM .

Back to the point:

emiranda wrote:
... when/if a queue manager is in partitioned status and the appliance where it is running goes down for any reason, the queue manager will failover to the other appliance, regardless of its partiotioned status, compromising data integrity!
Shouldn't the queue manager, knowing it is in a partiotioned status, refrain itself from starting? Or at least have an option where it would not start in such condition, as maybe, just MAYBE, business might think that integrity has more value than availability?
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Jun 24, 2019 4:23 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Well you said yourself the other node was not available. How should the system know that the queue manager is in a partitioned state when the other node is not available?

If you are switching manually you should be able to check first and resolve the partitioned state before switching...


_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
emiranda
PostPosted: Mon Jun 24, 2019 6:25 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

fjb_saper wrote:
If you are switching manually you should be able to check first and resolve the partitioned state before switching...

Absolutely - and so we did, and it works fine.

fjb_saper wrote:
Well you said yourself the other node was not available. How should the system know that the queue manager is in a partitioned state when the other node is not available?

Not really, maybe I was not clear... Simply speaking:
- On Appliance A, QM1 status is Running; HA status is Partitioned
- On appliance B, QM1 status is Running elsewhere; HA status is Partitioned
- Appliance A goes down - QM1 will failover to appliance B and run aged data.
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Jun 25, 2019 4:52 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Open a PMR and bring it up to IBM.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
emiranda
PostPosted: Tue Jun 25, 2019 7:19 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

fjb_saper wrote:
Open a PMR and bring it up to IBM.

So I did... and IBM's answer was
Quote:
Essentially, the current behaviour is suitable for Customers who prioritise service availability over individual messages.

Then I started to ask myself what's happened to the MQ I used to know? Where's all the good stuff about "providing a reliable once and once only message delivery system..."
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Jun 25, 2019 7:33 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

emiranda wrote:
Then I started to ask myself what's happened to the MQ I used to know? Where's all the good stuff about "providing a reliable once and once only message delivery system..."


*cough*

*cough*

Kafka!

*cough*

Welcome to the brave new world of streaming.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
emiranda
PostPosted: Wed Jun 26, 2019 1:06 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

Yeah, that's what I tought myself: is MQ changing?!? Then decided to share it here to get the community's view.

The latest MQv9.1 Datasheet opens saying "Flexible and reliable hybrid messaging solution across on premises and clouds..." and goes re-affirming reability, one of the paramount features of MQ, other 7 times... So I'd ask, Is MQ in the appliance any different from conventional MQ? Or it's just another "flavour" with some limitations (as no shell) and some extra functionalities (as built-in HA)? ... ???

IBM suggested me to open an RFE, but to my point, if it's just another MQ, shouldn't a data integrity issue be threated as a defect? I wonder...
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Jun 26, 2019 4:57 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

emiranda wrote:
So I'd ask, Is MQ in the appliance any different from conventional MQ? Or it's just another "flavour" with some limitations (as no shell) and some extra functionalities (as built-in HA)? ... ???


I would say it's just another "flavor" (sorry, US-English spell checking with fascist auto-correct settings) of MQ. I think where it's a culture shock is that there's only been 2 flavors for so long; z/OS and everything else. You get something like but not the same kind of "really?" thing with some of the fringe platforms (Non-Stop et al) but it's all the same MQ if you just breathe deeply and accept the platform quirks.

emiranda wrote:
IBM suggested me to open an RFE, but to my point, if it's just another MQ, shouldn't a data integrity issue be threated as a defect? I wonder...


I'm old enough to be cynical about IBM's description of a defect as a "previously unknown and undocumented feature".

My 2 cents; it's no worse than (for example) mismanaging the bootstrap datasets on z/OS. Someone used to MQ on z/OS will go "what? what?" when trying to describe them and doing it badly will cause message loss and/or non-functioning queue managers.

It should certainly be called out more clearly in the documentation, and in an ideal world (the one built by RFEs) not happen.

Other opinions are equally valid and may make more sense.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Jun 26, 2019 4:59 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

emiranda wrote:
Yeah, that's what I tought myself: is MQ changing?!? Then decided to share it here to get the community's view.


To be clear, my comment was directed at the New World Order of data streaming, where data may or may not arrive but will do so very quickly & the fact of it's non-arrival is treated with indifference by everyone from architects to developers.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jun 26, 2019 5:07 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The MQ product has always offered the opportunity for less than the highest QoS - in the form of non-persistent msgs. QoS is a choice.

MQ’s TT and mobile interfaces offer lower QoS solution opportunities, as well.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
emiranda
PostPosted: Wed Jun 26, 2019 8:38 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

bruce2359 wrote:
The MQ product has always offered the opportunity for less than the highest QoS - in the form of non-persistent msgs. QoS is a choice.

Exactly to my point: where is the choice to run at my desired QoS - DATA INTEGRITY?
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jun 26, 2019 9:15 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

emiranda wrote:
bruce2359 wrote:
The MQ product has always offered the opportunity for less than the highest QoS - in the form of non-persistent msgs. QoS is a choice.

Exactly to my point: where is the choice to run at my desired QoS - DATA INTEGRITY?

Your choice includes more robust and reliable platforms and failover technologies. If you chose the wrong ones, the configuration that fails to meet the requirements of the organization, then you can’t affix blame to MQ.

MQ for z/OS in Parallel Sysplex GDPS would improve your lot.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
emiranda
PostPosted: Thu Jun 27, 2019 3:06 am    Post subject: Reply with quote

Disciple

Joined: 21 Nov 2002
Posts: 196
Location: Dublin, Ireland

Yeah, we do have z/MQ as well, Sysplexed QSGs, GDPS is coming, plus all the good stuff around it... but again: if MQ on the appliances is just another flavo(u)r of the g'old MQ, where's integrity? It should be clearly stated the differences, which are not!

Actually, one of the VISP (Very Important Sales Point) highlighted to us was the "Built-In HA". Precisely to run, among others, a VIS (Very Important System) currently running on z/MQ - and I'm councious this could derail the discussion elsewhere, so let's leave it aside...

I've been working with MQ time enough, and I confess not to be most up-to-date on details of different platforms, but I'd say that MQ was never about availability, and always about reliability, hence the need for an external software, as VCS, PowerHA, MSCS, etc...

With the MQ Appliance, IBM brought the HA solution into the black-boxed-appliance, which sounds promissing, but them it'd make sense to take care of MQ's most precious value: ASYNCHRONOUS AND RELIABLE ONCE AND ONCE ONLY MESSAGE DELIVERY!

And excuse me if I sound too astonished, but IBM's answer in the PMR was that "Essentially, the current behaviour is suitable for Customers who prioritise service availability over individual messages."

Does it sound outrageously nutty or it is just in my head???
_________________
Warm Regards,
EM
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » HA queue manager behaviour
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.