| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Using NPMCLASS(HIGH) for saving NP messages |
« View previous topic :: View next topic » |
| Author |
Message
|
| orman |
 Posted: Fri Jan 30, 2015 12:01 am Post subject: Using NPMCLASS(HIGH) for saving NP messages |
 |
|
Apprentice
Joined: 08 Aug 2013
Posts: 40
|
Hey all
Our environment include Broker that connect to like 10 QM by channels.
Applications using to put message on their QM each other and those message get to Broker QM and than to the other application QMs.
I have found that a problem on our Production environment.
Not all of the applications put the message to be persistent - why? have no idea.
But those message are important and cannot be disappear because of shut-down of our servers.
The best is to make those application to write the messages as persistent, but we cannot do it nowadays.
So i decided to make a little search and I found NPMCLASS(HIGH) option in WMQ - Non-persistent Messages.
http://www-01.ibm.com/support/docview.wss?uid=swg21168546
| Quote: |
| NPMCLASS(HIGH) - This setting enables non-persistent messages on this queue to be retained across a queue manager restart. Non-persistent messages may still be lost in the event of a failure. |
More information on other sites:
http://www-01.ibm.com/support/docview.wss?uid=swg21293219
https://webspheremq.wordpress.com/category/ibm-technote/
I made some tests and it works pretty well.
So, it looks for me that this is a good solution to use this option in all of my business and important queues till we will fix the applications putting messages behavior.
I want to know your opinions and what are the risks and implications to use this one.
As I see, this NPMCLASS(HIGH) option it not that common, what makes it to look like a little unknown dark hole..  |
|
| Back to top |
|
 |
| exerk |
| Posted: Fri Jan 30, 2015 1:06 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
If, and I state if, the applications set messages persistence as per the queue definition you can change the queue definition to be persistent - DEFPSIST(YES) - which will make the messages persistent until such time as the application code can be changed.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jan 30, 2015 4:25 am Post subject: Re: Using NPMCLASS(HIGH) for saving NP messages |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| orman wrote: |
I want to know your opinions and what are the risks and implications to use this one. |
| orman wrote: |
NPMCLASS(HIGH) - Non-persistent messages may still be lost in the event of a failure. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 30, 2015 5:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
As I understand it, NPMCLASS really controls whether or not the queue manager throws away non-persistent messages (and perhaps markt will wander by and correct me). Traditionally, the queue manager would discard NP messages at startup, and NPMCLASS basically alters that behavior.
I don't believe NPMCLASS affects how messages are sent across the channel, so you can still lose NP messages that way, and I don't believe it affects how messages are written to q files, so again. |
|
| Back to top |
|
|
| Andyh |
| Posted: Fri Jan 30, 2015 6:37 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
NPMCLASS(HIGH) requires that NP messages are both saved at shutdown, and restored at startup. The way messages are handled during normal put/get processing is unchanged (NP messages may simply be buffered in memory, or may be subject to lazy writes to the queue file).
The typical gotcha here is that NPMCLASS(HIGH) is only effective if the queue manager shuts down in a controlled fashion ( it can be either a quiesce or an immediate shutdown, but it must complete normally). If the queue manager doesn't shut down in a controlled fashion then the NP messages may not be chained on disk and would not be restored on startup.
Hence a crash recovery would not result in NPMCLASS(HIGH) queues having their NP messages restored. |
|
| Back to top |
|
|
| orman |
| Posted: Fri Jan 30, 2015 11:46 pm Post subject: |
|
|
Apprentice
Joined: 08 Aug 2013
Posts: 40
|
| mqjeff wrote: |
As I understand it, NPMCLASS really controls whether or not the queue manager throws away non-persistent messages (and perhaps markt will wander by and correct me). Traditionally, the queue manager would discard NP messages at startup, and NPMCLASS basically alters that behavior.
I don't believe NPMCLASS affects how messages are sent across the channel, so you can still lose NP messages that way, and I don't believe it affects how messages are written to q files, so again. |
The risks I mean are not of losing messages-this one is already exists and I want to fix it.
I am not expecting the messages to be persistent, it will still stay not - persistent as i know
But, are there any other risks for my QM? In terms of the file system or log or something, I don't know
And, is the option is recommended to use? |
|
| Back to top |
|
|
| tczielke |
| Posted: Sat Jan 31, 2015 6:46 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| Can you explain more why you just can't change the application to use persistent messages? Saying that a Production application can not be changed is really an untenable stance to take. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Jan 31, 2015 6:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| orman wrote: |
I am not expecting the messages to be persistent, it will still stay not - persistent as i know
But, are there any other risks for my QM? In terms of the file system or log or something, I don't know
And, is the option is recommended to use? |
Recommended?
NPMCLASS(HIGH) is an option. It offers you the possibility of retaining non-persistent messages during qmgr restart for some local queues.
Your organization, and its business policies should determine if this option makes sense (recommended) for your applications.
Are your non-persistent messages important (like inquiries)? Or are they mission-critical (like $ funds transfers)?
Are the transactions that created your non-persistent messages reproducible?
There is very low risk to your qmgr from setting NPMCLASS(HIGH). Non-persistent messages are not logged.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Last edited by bruce2359 on Sat Jan 31, 2015 4:00 pm; edited 1 time in total |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Jan 31, 2015 2:55 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Yes, but what are the impacts to the QM and the MQ server? These are fair questions.
Does the QM take longer to shutdown when it has one or more NPMCLASS(HIGH) queues? If yes, what is the formula?
Does the QM take longer to start up when it has to restore those non persistent messages?
Where does the QM store them if the QM ends normally? Does it add them into the Linear logs if you are using Linear Logs? If yes, does it then remove them from the Linear logs upon start up?
If you are using circular logs, I assume there is no impact to the circular logs either way, but is that true?
Or do the logs not come into play in any scenario for "remembering" these NPMCLASS(HIGH) messages, and the QM is just writing to some file, maybe the q file?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Jan 31, 2015 4:24 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| PeterPotkay wrote: |
Yes, but what are the impacts to the QM and the MQ server? These are fair questions.
Does the QM take longer to shutdown when it has one or more NPMCLASS(HIGH) queues? If yes, what is the formula?
Does the QM take longer to start up when it has to restore those non persistent messages?
Where does the QM store them if the QM ends normally? Does it add them into the Linear logs if you are using Linear Logs? If yes, does it then remove them from the Linear logs upon start up?
If you are using circular logs, I assume there is no impact to the circular logs either way, but is that true?
Or do the logs not come into play in any scenario for "remembering" these NPMCLASS(HIGH) messages, and the QM is just writing to some file, maybe the q file? |
I'll yield to others, but it is my understanding that at shutdown (endmqm), all messages remaining in queue buffers (persistent and non-persistent) are written to queue-holding disk (file-system). Writing N-P buffers to disk might be a new behavior with the introduction of NPMCLASS queue option.
At qmgr restart, NPMCLASS(HIGH) queues will be bypassed by garbage collector, as other queues have their N-P messages deleted. There is no restoring of N-P messages.
IMS, endmqm options -c, -i, and -p, all result in the qmgr ending "normally," which includes offloading queue buffers, writing closing checkpoint, and so on. The -i and -p options might adversely affect applications.
I'll step back into the shadows, and await correction and/or flaming.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Andyh |
| Posted: Sun Feb 01, 2015 6:16 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
As stated earlier, NP messages may either be stored in queue buffers, or lazily (unforced) written to the appropriate queue file. Although the messages may be stored in the queue file, they are not normally linked together in the file and the only way of accessing the messages is through the in memory queue index. For example, an MQGET of a non-persistent message that was written to the queue file will involved file read activity, but no file writing.
For NPMCLASS(HIGH) queues, during a controlled shutdown any messages in queue buffers are written to the queue file, and the message linkages are also written (again to the queue file) for all NP messages (even those already written to the queue file). Finally the writes to the queue file are then flushed.
All of the 'extra' writing occurs during shutdown and is only to the queue file. There is no change to the way persistent messages are logged, and no additional log records are written w.r.t the NPMCLASS(HIGH) queues.
A completely successful orderly, or immediate shutdown would normally result in no loss of NPMCLASS(HIGH) messages, however an endmqm -p is a more abrupt affair and an endmqm -p may well result in NPMCLASS(HIGH) messages being discarded.
I hope I'm not too guilty of being the "flame thrower", perhaps I'm not finding the right balance when keeping some of the discussions honest while not correcting mistakes of little or no consequence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Feb 01, 2015 6:42 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Andyh wrote: |
A completely successful orderly, or immediate shutdown would normally result in no loss of NPMCLASS(HIGH) messages, however an endmqm -p is a more abrupt affair and an endmqm -p may well result in NPMCLASS(HIGH) messages being discarded.
I hope I'm not too guilty of being the "flame thrower", perhaps I'm not finding the right balance when keeping some of the discussions honest while not correcting mistakes of little or no consequence. |
I don't feel flamed at all, Andy.
As to the -p option: from http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.0.1/com.ibm.mq.amqzag.doc/fa15800_.htm
| Quote: |
| Use this type of shutdown only in exceptional circumstances. For example, when a queue manager does not stop as a result of a normal endmqm command. |
In an exceptional circumstance, there is no guarantee that any messages, persistent or non-persistent, will survive as we've come to expect from MQ.
| Quote: |
| This can give unpredictable results for WebSphere® MQ applications. |
Use of the -p option would most likely result in uncommitted UofW's being backed out at qmgr restart - and some messages "lost."
I recall needing to use the -p option (to deal with exceptional conditions) in v2 and v5; but far less since then. It seems to me that MQ at more current levels doesn't get its termination shoelaces tangled as it used to.
An update: In a POC test environment, I put test N-P messages to NPMCLASS(HIGH) queues, and lost no messages. In concept (the C in POC), NPMCLASS(HIGH) works. I freely admit that I have not tested this in a high-volume production environment. I would not expect (depend on) NPMCLASS(HIGH) to work as well in production.
I do not advise clients to use N-P messages or NPMCLASS(HIGH) if messages MUST survive qmgr restart.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Sun Feb 01, 2015 7:18 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| Andyh wrote: |
| I hope I'm not too guilty of being the "flame thrower", perhaps I'm not finding the right balance when keeping some of the discussions honest while not correcting mistakes of little or no consequence. |
It was helpful for me to have the extra detail. I really appreciate it when someone from the MQ Development team takes time to explain things more in detail. |
|
| Back to top |
|
|
| Andyh |
| Posted: Sun Feb 01, 2015 8:44 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
I'm not advocating the use of endmqm -p, however use of this option should not expose customers to losing persistent messages.
Simply killing all MQ related processes is even more abrupt than endmqm -p (which at least makes an effort to close things down in a controlled fashion), and even that action should have no adverse effect on persistent message integrity.
Inflight UOW's will be backed out (as they typically would for endmqm -i), however transactional integrity will be maintained and no committed messages should be lost.
Losing persistent messages as a direct consequence of using endmqm -p would be considered an APARable error. |
|
| Back to top |
|
|
| tczielke |
| Posted: Sun Feb 01, 2015 11:40 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| bruce2359 wrote: |
An update: In a POC test environment, I put test N-P messages to NPMCLASS(HIGH) queues, and lost no messages. In concept (the C in POC), NPMCLASS(HIGH) works. I freely admit that I have not tested this in a high-volume production environment. I would not expect (depend on) NPMCLASS(HIGH) to work as well in production.
I do not advise clients to use N-P messages or NPMCLASS(HIGH) if messages MUST survive qmgr restart. |
FYI - On Linux, I ran the following command from the shell prompt:
| Code: |
echo TESTMSG | amqsput TCZ.TEST1;for i in `ps -u mqm -o pid,args | egrep '(amq|runmq)' | cut -d ' ' -f1`; do kill -9 $i; done
|
which produced the results:
| Code: |
> echo TESTMSG | amqsput TCZ.TEST1;for i in `ps -u mqm -o pid,args | egrep '(amq|runmq)' | cut -d ' ' -f1`; do kill -9 $i; done
Sample AMQSPUT0 start
target queue is TCZ.TEST1
Sample AMQSPUT0 end
|
The command will first PUT the TESTMSG message on the TCZ.TEST1 queue that succeeded with a zero return code. Then it immediately kills the queue manager. When I restarted the queue manager, the TESTMSG message was missing on the TCZ.TEST1 queue. So this successfully recreates the scenario of how a PUT to a queue with NPMCLASS(HIGH) can cause message loss.
The key is killing the queue manager very quickly after the NPMCLASS(HIGH) PUT succeeds, but before the kernel thread that flushes lazy writes can execute. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
