| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| AMQ7469 and lost messages? |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Wed Jan 23, 2008 9:17 am Post subject: |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| jefflowrey wrote: |
| LouML wrote: |
| Am I correct in saying that if messages are getting rolled back and then lost then the app is not handling the rollback properly? |
It depends on which app had a transaction rolled back on it...
If it was the MCA, then it's likely handling the rollback properly - i.e. discarding the messages if non-persistent. |
Jeff, I'm not sure on this. If the non persistent mesage is on the XMITQ, MQ has got it and its safe, short of:
A. The QM restarting
B. Being shipped down a channel whose NPMSPEED is set to FAST and that channel puking.
If the channel tries to send it over to the other side and logs are invloved, then the channel speed must be NORMAL. If the channel then has to rollback, why would the message be destroyed? It was already on the XMITQ. It was part of a UOW and if that UOW is being rolled back why wouldn't the message just end up back on the XMITQ? Don't UOW protect persistent and non persistent messages just the same?
LouML, what's the channel speed please? Regarding the log size, ask yourself what costs more - disk space or outages due to puny logs. That normally gets the managers to agree to add space. I bet your server probably has gigs free anyway.
As for persistence, I don't back down for a second. Its up to the app to set. Either that or your standard is that ALL apps must use As Q Default and you are 100% responsible for setting the queues' defaults. As soon as you have some apps setting it themselves and some relying on you you have the oppourtunity for finger pointing. Its up to the apps to set I say. A, because they know better than you how important there message is, and B, the queue's default is just a suggestion. Its trivial for the app to override it.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| LouML |
| Posted: Wed Jan 23, 2008 9:27 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| jefflowrey wrote: |
Right, the real thing is.
DEFPSIST is just a suggestion.
The only thing that controls whether a message is persistent or not is the MQMD. If the MQMD says it's a non-persistent message, then it's a non-persistent message, regardless of what queue it is on. |
Okay, that's where my confusion lies. I thought that, if it wasn't specified at Put time, it gets the default from whatever queue it is currently residing on, rather than the default of the queue being Put to at Put time
Probably still not wording it correctly but I get it.
Thanks again all. |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Jan 23, 2008 10:08 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| PeterPotkay wrote: |
LouML, what's the channel speed please? Regarding the log size, ask yourself what costs more - disk space or outages due to puny logs. That normally gets the managers to agree to add space. I bet your server probably has gigs free anyway.
As for persistence, I don't back down for a second. Its up to the app to set. Either that or your standard is that ALL apps must use As Q Default and you are 100% responsible for setting the queues' defaults. As soon as you have some apps setting it themselves and some relying on you you have the oppourtunity for finger pointing. Its up to the apps to set I say. A, because they know better than you how important there message is, and B, the queue's default is just a suggestion. Its trivial for the app to override it. |
NPMSPEED(FAST)
| Code: |
bash-2.05$ echo "dis chl(CH.US.TO..THEM)" | runmqsc QM.US.01
5724-B41 (C) Copyright IBM Corp. 1994, 2002. ALL RIGHTS RESERVED.
Starting MQSC for queue manager QM.US.01.
1 : dis chl(CH.US.TO..THEM)
AMQ8414: Display Channel details.
CHANNEL(CH.US.TO.THEM) CHLTYPE(SDR)
TRPTYPE(TCP) DESCR( )
XMITQ(QM.US.01) MCANAME( )
MODENAME( ) TPNAME( )
BATCHSZ(50) DISCINT(6000)
SHORTRTY(10) SHORTTMR(60)
LONGRTY(999999999) LONGTMR(1200)
SCYEXIT( ) SEQWRAP(999999999)
MAXMSGL(4194304) CONVERT(YES)
SCYDATA( ) USERID( )
PASSWORD( ) MCATYPE(PROCESS)
CONNAME(mainframe(1414)) HBINT(300)
BATCHINT(0) NPMSPEED(FAST)
SSLCIPH( ) BATCHHB(0)
LOCLADDR( ) KAINT(AUTO)
MCAUSER( ) ALTDATE(2007-10-04)
ALTTIME(08.47.36) SSLPEER()
MSGEXIT( )
SENDEXIT( )
RCVEXIT( )
MSGDATA( )
SENDDATA( )
RCVDATA( )
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
bash-2.05$ |
|
|
| Back to top |
|
|
| JuulV |
| Posted: Wed Jan 23, 2008 10:45 am Post subject: |
|
|
Apprentice
Joined: 05 Sep 2007
Posts: 28
Location: Belgium
|
First some simple facts:
1. The messages MUST BE PERSISTENT; only persistent messages are ever written to the logs!
2. The logs are too small for the workload, as evidenced by the the AMQ7469 messages!
3. The applications suffering from this problem ARE MADE AWARE OF IT: if no log space is available, any activity (be it a GET or a PUT of a persistent message) requiring log space will fail with MQRC_RESOURCE_PROBLEM; if it is a custom application, I off course don't know if they check MQCC/MQRC after each call (BUT THEY MUST, not should!).
4. Changing the persistence for the XMIT queues doesn't matter a thing... As several replies already stated, persistence is an attribute of the message, not of a queue; unless some application (or exit) intervenes, the message will retain the persistence it had at the initial PUT, regardless of the setting of the Queue persistence attribute of any queue it travels through on its yourney.
5. Calls that receive MQCC_FAILED never happened what WMQ is concerned; if this is the initial PUT, there never was a message to begin with; if this was any other subsequent call, the message will always remain available somewhere in WMQ.
Now, a very stupid (but very basic) question: you state that the M/F reports over 32,000 messages missing... Does this message come from WMQ itself? Or is this the actual application that checks message content (E.g. some sequence number) and says that it misses messages?
If it is WMQ, the messages still must be available somewhere (see the simple facts above)...
If it's the application, the original PUTTING application probably doesn't check MQCC/MQRC, and your data are truly lost (WMQ never received them, and the application lost track of them)...
Regards,
_________________
Juul Vanparijs
Senior Developer
Cressida Technology Ltd |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Jan 23, 2008 11:14 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| JuulV wrote: |
| Now, a very stupid (but very basic) question: you state that the M/F reports over 32,000 messages missing... Does this message come from WMQ itself? Or is this the actual application that checks message content (E.g. some sequence number) and says that it misses messages? |
The mainframe back-office staff noted an End-of-Day breaksheet with hundreds of pages of breaks versus a usual days breaksheet with a page or so.
The back office support and the application manager worked together to manually figure out that there were blocks of missing messages. The app manager was able to resend all missing messages so in the end, everything turned out okay.
Something strange here is that the app manager states that each message is a single transaction. I would have thought they were being grouped causing a single transaction to span the logs. I guess with the logs so small this could still be the case.
Seems to me there are two problems
1 - Log sizes way too small
2 - Network glitch caused the loss of the non-persistent messages
| JuulV wrote: |
If it's the application, the original PUTTING application probably doesn't check MQCC/MQRC, and your data are truly lost (WMQ never received them, and the application lost track of them)...
Regards, |
I'm not a Perl programmer. It looks to me like the Put is done in a paragraph called sendMessage. Within that, the app checks to see if the Put fails.
| Code: |
sendMessage
if ( ! $OutputQ->Put( Message => $oMessage) ) {
$log->LogMsg (" Unable to put message onto $OutputQueueName"
. ": ".$OutputQ->Reason()." - ".MQReasonToText($OutputQ->Reason()));
$page->SendPages("Unable to put message onto ".$OutputQueueName
. ": ".$OutputQ->Reason()." - ".MQReasonToText($OutputQ->Reason()), "err");
return 1;
}
else {
$log->LogMsg (" Successfully put message onto $OutputQueueName");
return 0;
}
} |
However, when the sendMessage part is called, it does not look like it checks the return code.
| Code: |
| sendMessage($msg, 1); |
instead of
| Code: |
| if {sendMessage($msg, 1);} |
Anyway, they have been made aware that they need to make the change to the code and I know that I have to increase the log sizes over the weekend.
Thanks again to all. |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Jan 23, 2008 11:40 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
Now I'm confused. The queue that the app is putting to is the QR.EXECUTIONS.OUT, which has DEFPSIST set to YES. So, when the message was put, it should have been persistent.
| Code: |
bash-2.05$ echo "dis q(QR.EXECUTIONS.OUT)" | runmqsc QM.US.01
5724-B41 (C) Copyright IBM Corp. 1994, 2002. ALL RIGHTS RESERVED.
Starting MQSC for queue manager QM.US.01.
1 : dis q(QR.EXECUTIONS.OUT)
AMQ8409: Display Queue details.
DESCR( ) RNAME(QL.FLEXTRADE.IN)
RQMNAME(MQJP) XMITQ(QM.US.01)
CLUSTER( ) CLUSNL( )
QUEUE(QR.EXECUTIONS.OUT) ALTDATE(2006-02-02)
ALTTIME(18.48.22) PUT(ENABLED)
DEFPRTY(0) DEFPSIST(YES)
SCOPE(QMGR) DEFBIND(OPEN)
TYPE(QREMOTE)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
bash-2.05$ echo "dis ql(QM.US.01)" | runmqsc QM.US.01
5724-B41 (C) Copyright IBM Corp. 1994, 2002. ALL RIGHTS RESERVED.
Starting MQSC for queue manager QM.US.01.
1 : dis ql(QM.US.01)
AMQ8409: Display Queue details.
DESCR(MQJP Transmission Queue) PROCESS( )
BOQNAME( ) INITQ(SYSTEM.CHANNEL.INITQ)
TRIGDATA(CH.US.TO.THEM) CLUSTER( )
CLUSNL( ) QUEUE(QM.US.01)
CRDATE(2005-09-03) CRTIME(09.55.01)
ALTDATE(2008-01-23) ALTTIME(09.28.04)
GET(ENABLED) PUT(ENABLED)
DEFPRTY(0) DEFPSIST(NO)
MAXDEPTH(50000) MAXMSGL(4194304)
BOTHRESH(0) SHARE
DEFSOPT(SHARED) HARDENBO
MSGDLVSQ(PRIORITY) RETINTVL(999999999)
USAGE(XMITQ) TRIGGER
TRIGTYPE(FIRST) TRIGDPTH(1)
TRIGMPRI(0) QDEPTHHI(80)
QDEPTHLO(20) QDPMAXEV(ENABLED)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QSVCINT(999999999) QSVCIEV(NONE)
DISTL(NO) NPMCLASS(NORMAL)
DEFTYPE(PREDEFINED) TYPE(QLOCAL)
SCOPE(QMGR) DEFBIND(OPEN)
IPPROCS(1) OPPROCS(6)
CURDEPTH(0)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
bash-2.05$ |
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 23, 2008 11:55 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Right - but the PUT failed, so the message was never actually put.
And the reason the PUT failed is either a) the transaction was too big for the logs, or b) the transaction was too big for the logs at the time, because lots of other transactions or other big transactions were using up all the space.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 23, 2008 12:41 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| LouML wrote: |
| Now I'm confused. The queue that the app is putting to is the QR.EXECUTIONS.OUT, which has DEFPSIST set to YES. So, when the message was put, it should have been persistent. |
Only if the application is using the default and not setting non-persistence directly
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Jan 23, 2008 12:46 pm Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| Vitor wrote: |
| LouML wrote: |
| Now I'm confused. The queue that the app is putting to is the QR.EXECUTIONS.OUT, which has DEFPSIST set to YES. So, when the message was put, it should have been persistent. |
Only if the application is using the default and not setting non-persistence directly |
That is the case. They are not setting it at all.
Seems nobody in this firm is setting persistence in their apps. From the code I've been poring over all day, it's either omitted completely or it's set to MQPER_PERSISTENCE_AS_Q_DEF
This particular app is not setting anything.
Also, the NPMSPEED is set to fast on the channels on both sides, but since the messages now appear to be persistent, I guess this no longer applies. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 23, 2008 12:47 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
But the transaction was rolled back, so the PUT was never committed, so it never happened.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Jan 23, 2008 12:47 pm Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| jefflowrey wrote: |
Right - but the PUT failed, so the message was never actually put.
And the reason the PUT failed is either a) the transaction was too big for the logs, or b) the transaction was too big for the logs at the time, because lots of other transactions or other big transactions were using up all the space. |
Just spoke with the programmer and he says there were no messages in the log about failed puts. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 23, 2008 1:24 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| LouML wrote: |
| Also, the NPMSPEED is set to fast on the channels on both sides, but since the messages now appear to be persistent, I guess this no longer applies. |
In my opinion you need a clear and concise reason to have your channel speeds set to fast. Otherwise set them to normal to minimize message loss for non persistent messages. Personally I think the default should be normal, not fast.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| LouML |
| Posted: Fri Jan 25, 2008 11:15 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
We have plenty of room on the drive, so I'm going to make it 20 Primary pages and 5 Secondary. I'll do a queue manager restart this weekend and we'll see how it goes.
| Code: |
Current Proposed
LogPrimaryFiles=5 LogPrimaryFiles=20
LogSecondaryFiles=3 LogSecondaryFiles=5
LogFilePages=4096 LogFilePages=4096
LogType=LINEAR LogType=LINEAR
LogBufferPages=512 LogBufferPages=512
bash-2.05$ ls -la /MQHA/QM.jefomsapp.01/log/QM\!jefomsapp\!01/active/
total 41042
drwxrwx--- 2 mqm mqm 1024 Jan 22 15:53 .
drwxrwx--- 3 mqm mqm 96 Jul 5 2005 ..
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:35 S0000000.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:28 S0000001.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:09 S0000002.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:39 S0000003.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:10 S0000004.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:35 S0000005.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:28 S0000006.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:09 S0000007.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:39 S0000008.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:10 S0000009.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:35 S0000010.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:28 S0000011.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:09 S0000012.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:39 S0000013.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:10 S0000014.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:35 S0000015.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:28 S0000016.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 07:09 S0000017.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:39 S0000018.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 23 08:10 S0000019.LOG
bash-2.05$ |
|
|
| Back to top |
|
|
| LouML |
| Posted: Mon Jan 28, 2008 5:39 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| LouML wrote: |
We have plenty of room on the drive, so I'm going to make it 20 Primary pages and 5 Secondary. I'll do a queue manager restart this weekend and we'll see how it goes.
| Code: |
Current Proposed
LogPrimaryFiles=5 LogPrimaryFiles=20
LogSecondaryFiles=3 LogSecondaryFiles=10
LogFilePages=4096 LogFilePages=4096
LogType=LINEAR LogType=LINEAR
LogBufferPages=512 LogBufferPages=512 |
|
Well, I made the change to the qm.ini file over the weekend to go from 5 Primary and 3 Secondary to 20 Primary and 10 Secondary. The Queue Manager was restarted.
Today, I still see only 5 files. Should they have been immediately visible upon restart or will they only get created when needed?
| Code: |
bash-2.05$ ls -la /MQHA/QM.jefomsapp.01/log/QM\!jefomsapp\!01/active
total 41042
drwxrwx--- 2 mqm mqm 1024 Jan 22 15:53 .
drwxrwx--- 3 mqm mqm 96 Jul 5 2005 ..
-rw-rw---- 1 mqm mqm 4202496 Jan 28 07:48 S0000000.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 28 08:35 S0000001.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 28 07:25 S0000002.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 28 07:15 S0000003.LOG
-rw-rw---- 1 mqm mqm 4202496 Jan 28 08:23 S0000004.LOG
bash-2.05$ |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 28, 2008 5:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Is changing the number of logs like that supported now?
Back in my day, the number and type of logs was fixed at queue manager creation.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
