| Author |
Message
|
| rjlfc |
 Posted: Wed Feb 09, 2011 3:26 am Post subject: Large Message Handling - Log File Pages Primary, Secondary e |
 |
|
Apprentice
Joined: 04 Apr 2008
Posts: 31
|
We are developing a PoC for an application (Websphere) to write large, transient messages to a queue and readin the message from a queue almost immediately.
We are using MQv7 on Windows 2003
Queue is non-persistent
Sending and Receiving apps using MQ Client connections
The QM is an existing one (not created for this prime use, it is shared)
All transaction log settings are default - Log files pages = 4096, Primary files 3, secondary files 2
Note the message will only be transferred once every few days (not frequently)
The QM, client channel and queue allow up to 100Mb messages. However, we only seem to be able to write up to 26Mb messages before getting errors as per below
I though the size of the logs were not relevant for non persistent messages? thanks
Error details :
AMQ7463: The log for queue manager XEGW_CEGW_DEVTEST is full.
EXPLANATION:
This message is issued when an attempt to write a log record is rejected
because the log is full. The queue manager will attempt to resolve the problem. |
|
| Back to top |
|
 |
| exerk |
| Posted: Wed Feb 09, 2011 3:48 am Post subject: Re: Large Message Handling - Log File Pages Primary, Seconda |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| rjlfc wrote: |
| ...Queue is non-persistent... |
No it's not - messages are persistent or non-persistent. Look in the manuals for what that attribute on the queue is for.
| rjlfc wrote: |
| ...we only seem to be able to write up to 26Mb messages before getting errors... |
Because your 'WebSphere' application (presumably you mean WAS) is putting "...large, transient messages..." on the queue, and maybe within a Unit-of-Work (UoW) that spans the available logs? Do you also see AMQ7465 and AMQ7466 message in the error log?
| rjlfc wrote: |
| ...I thought the size of the logs were not relevant for non persistent messages? |
Are you certain the messages are non-persistent?
| rjlfc wrote: |
| ...All transaction log settings are default - Log files pages = 4096, Primary files 3, secondary files 2... |
Increase the number of logs, bounce the queue manager, and try again.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| skoobee |
| Posted: Wed Feb 09, 2011 4:00 am Post subject: |
|
|
Acolyte
Joined: 26 Nov 2010
Posts: 52
|
You are right, logs are not used to store non-persistent msgs. However, for the log to become full the apps must be writing and reading persistent msgs.
Just because the queue is non-persistent, which I take to mean that the DEFPSIST attribute is NO, this only affectes the msg if the app sets the MD.Persistence to MQPER_PERSISTENCE_AS_Q_DEF. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Feb 09, 2011 4:06 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| skoobee wrote: |
| ...However, for the log to become full the apps must be writing and reading persistent msgs... |
Or using excessively long UoWs...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| rjlfc |
| Posted: Wed Feb 09, 2011 4:08 am Post subject: |
|
|
Apprentice
Joined: 04 Apr 2008
Posts: 31
|
many thanks fir this, really helpful.
So we've just determined that the app was actually setting the messages to persistence and not using the queue default. When we changed this, larger messages were fine.
So my next question is, although it seems the transaction logs are not affected by non-persistent messages (which makes sense) - if we had an application issue that caused these large messages to build up (i.e. increase the curdept of the queue), presumably the log files are not relevant in terms of limitations in this scenario, the limiting factors are disk space and the max number of messages on the queue?
E.g. to be able to cope with 100 non-persistent message of size 50Mb, we need sufficient disk space and max no. messages setting on the queue to be >100. The log file settings are not relevant in this case?
Thanks |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Feb 09, 2011 4:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
|
| Back to top |
|
|
| rjlfc |
| Posted: Wed Feb 09, 2011 9:02 am Post subject: |
|
|
Apprentice
Joined: 04 Apr 2008
Posts: 31
|
|
| Back to top |
|
|
| sunny_30 |
| Posted: Wed Sep 05, 2012 10:46 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
| does this mean that non-persistent messages are also stored in MQ transaction-logs before getting copied to the queue file ? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 05, 2012 11:28 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Non-persistent messages are not written to WMQ logs.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 05, 2012 11:38 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| bruce2359 wrote: |
| Non-persistent messages are not written to WMQ logs. |
Including those delivered as NPMCLASS(HIGH), when the queue manager shuts down normally or immediately?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 06, 2012 6:17 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Afaik, npmclass(high) only affects qmgr restart behavior - not purging non-persistent msgs.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Sep 06, 2012 7:41 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| bruce2359 wrote: |
| Afaik, npmclass(high) only affects qmgr restart behavior - not purging non-persistent msgs. |
The question is then - are non-persistent NPMCLASS(HIGH) messages stored only in the queue file, or in the queue file and logs?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 06, 2012 7:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| exerk wrote: |
| non-persistent NPMCLASS(HIGH) messages |
There's no such thing.
A given message is either persistent or non-persistent.
NPMCLASS only affects the behavior of the queue manager in relation to how it handles messages that are stored on the queue file.
Remember that in specific circumstances - a PUT that is executed while there is a waiting GET that matches - non-persistent messages may never be written *anywhere*, but merely handed off from one application to another. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Sep 06, 2012 7:53 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| mqjeff wrote: |
| exerk wrote: |
| non-persistent NPMCLASS(HIGH) messages |
There's no such thing. |
OK, bad terminology on my part, and having re-read the Info Centre in relation to JMS messages going to a queue with NPMCLASS(HIGH) I can see that they go to the queue file only but are not discarded on restart - the caveat being that all required conditions are met.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Andyh |
| Posted: Thu Sep 06, 2012 8:01 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Non persistent messages might be stored in memory only (i.e not spilled to disk), even in the case where a message cannot be delivered directly to a waiting MQGET. In the case where a non persistent message was stored in a memory buffer, any unconsumed messages would be written to disk at controlled queue manager shutdown for NPMCLASS(HIGH) queues.
Non persistent messages are never written to the recovery log at MQPUT time, regardless of whether these messages are inside or outside of syncpoint. A transaction involving only non persistent resources will not result in ANY data being written to the recovery log.
If persistent and non-persistent messages are stored on the same queue, in a linear logging environment, a media image of the queue might involve writing some non-persistent data to the recovery log, however these messages would not be recovered in the event of a media recovery operation (only the persistent messages would be recovered). |
|
| Back to top |
|
|
|
|
