| Author |
Message
|
| superhero |
 Posted: Wed Jul 05, 2006 2:52 am Post subject: Messages in dead letter queue without reason code. Need help |
 |
|
Newbie
Joined: 16 May 2006
Posts: 6
|
Describing here the problem that we are facing in our system... We use Websphere MQ on AIX operating system. There are a number of messages found in the dead letter queue. For all of these messages, the dlq handling has become difficult since no where in the message can we see the reason code why they were put in the dlq. If any one has faced similar situation and can be of help, then they are most welcome.
Here is a sample message header from the dead letter queue: -
MQGET of message number 1
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : 'MQDEAD '
Priority : 0 Persistence : 1
MsgId : X'414D51204349414D5131312020202020D68B454420001D01'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ : ' '
ReplyToQMgr : 'CIAMQ11 '
** Identity Context
UserIdentifier : 'ciacli12 '
AccountingToken :
X'16010515000000CF8F3D27675B744EB282BE82F203000000000000000000000B'
ApplIdentityData : ' '
** Origin Context
PutApplType : '11'
PutApplName : 'C:\notes\NLNOTES.EXE '
PutDate : '20060421' PutTime : '15130339'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
**** Message ****
length - 308 bytes
...
..
.
Also some of the messages in the dead letter queue are such that the put time of those messages matches perfectly with my application - log file's "message recieved" timestamp. meaning that, my application recieved that message, but still it went to the dead letter queue ?
here is one such message header...
Message number 1164
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : 'MQDEAD '
Priority : 0 Persistence : 0
MsgId : X'414D512049424D4B503145342020202044554A5120037923'
CorrelId : X'000000000000000000000000000000000000000000000000'
Sequence Number : '0'
BackoutCount : 0
ReplyToQ : ' '
ReplyToQMgr : 'IBMKP1E4 '
** Identity Context
UserIdentifier : ' '
AccountingToken :
X'0000000000000000000000000000000000000000000000000000000000000000'
ApplIdentityData : ' '
** Origin Context
PutApplType : '7'
PutApplName : 'IBMKP1E4 '
PutDate : '20060505' PutTime : '11590982'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
Current Data Length : 856 bytes
**** Message Content ****
...
..
.
The problem has increased in magnitude since a lot of such messages have entered the dead letter queue, but still our applications work perfectly. Any help from anyone is most welcome! |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jul 05, 2006 2:58 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Are you saying there's no MQDLH structure in this message? The Format shows MQDEAD which indicates there should be one (and the Reason field therein). 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| superhero |
| Posted: Wed Jul 05, 2006 3:12 am Post subject: |
|
|
Newbie
Joined: 16 May 2006
Posts: 6
|
| Quote: |
| Are you saying there's no MQDLH structure in this message? |
Thats is what even I was expecting, but unfortunately what I pasted above is all that I can see. I used amqsbcg to browse the dead letter queue. Hope that amqsbcg does not modify the message headers while showing the contents. Apart from that I used another browsing tool made by my team mate and it showed same contents.
Thanks in anticipation. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 05, 2006 3:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
No, amqsbcg doesn't modify anything (having just run a quick test).
If it doesn't violate any confidentiality rules, can you post the entire message? Your previous post ends just as the DLH should have started. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| superhero |
| Posted: Wed Jul 05, 2006 3:47 am Post subject: |
|
|
Newbie
Joined: 16 May 2006
Posts: 6
|
| Quote: |
| If it doesn't violate any confidentiality rules, can you post the entire message? Your previous post ends just as the DLH should have started. |
Ok, if they want to fire me for this violation of confidentiality, its their wish but still I am committed to solve this issue 
This is a random message taken from the dlq. The next message follows it...
MQGET of message number 67
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : 'MQDEAD '
Priority : 0 Persistence : 0
MsgId : X'414D512049424D4B503145342020202044554A5120001302'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ : ' '
ReplyToQMgr : 'IBMKP1E4 '
** Identity Context
UserIdentifier : ' '
AccountingToken :
X'0000000000000000000000000000000000000000000000000000000000000000'
ApplIdentityData : ' '
** Origin Context
PutApplType : '7'
PutApplName : 'IBMKP1E4 '
PutDate : '20060501' PutTime : '01165790'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
**** Message ****
length - 856 bytes
00000000: 444C 4820 0000 0001 0000 0109 4942 4D4B 'DLH ........IBMK'
00000010: 5031 4534 2E4C 5044 2E49 4E49 5420 2020 'P1E4.LPD.INIT '
00000020: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000030: 2020 2020 2020 2020 2020 2020 4942 4D4B ' IBMK'
00000040: 5031 4534 2020 2020 2020 2020 2020 2020 'P1E4 '
00000050: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000060: 2020 2020 2020 2020 2020 2020 0000 0111 ' ....'
00000070: 0000 0333 4D51 5452 4947 2020 0000 0006 '...3MQTRIG ....'
00000080: 5255 4E4D 5154 524D 0000 0000 0000 0000 'RUNMQTRM........'
00000090: 0000 0000 0000 0000 0000 0000 3230 3036 '............2006'
000000A0: 3035 3031 3031 3137 3535 3737 544D 2020 '050101175577TM '
000000B0: 0000 0001 4942 4D4B 5031 4534 2E4C 5044 '....IBMKP1E4.LPD'
000000C0: 2E53 4552 5632 2E46 494C 4552 4356 2020 '.SERV2.FILERCV '
000000D0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000000E0: 2020 2020 4942 4D4B 5031 4534 2E4C 5044 ' IBMKP1E4.LPD'
000000F0: 2E53 4552 5632 2E50 524F 4345 5353 2020 '.SERV2.PROCESS '
00000100: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000110: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000120: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000130: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000140: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000150: 2020 2020 0000 0006 2F68 6F6D 652F 7061 ' ..../home/pa'
00000160: 6473 2F6C 7064 3130 2F62 696E 2F70 6164 'ds/lpd10/bin/pad'
00000170: 7346 7065 5265 6365 6976 6520 3220 2020 'sFpeReceive 2 '
00000180: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000190: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001A0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001B0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001C0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001D0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001E0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001F0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000200: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000210: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000220: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000230: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000240: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000250: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000260: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000270: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000280: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000290: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002A0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002B0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002C0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002D0: 2020 2020 2020 2020 7061 6473 6670 6520 ' padsfpe '
000002E0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002F0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000300: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000310: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000320: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000330: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000340: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000350: 2020 2020 2020 2020 ' '
MQGET of message number 68
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
...
..
. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Wed Jul 05, 2006 4:02 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
The reason code is x109, 265 = MQFB_APPL_CANNOT_BE_STARTED, and it is a trigger msg, put there when the trigger monitor cannot start the app.
See this topic
http://www.mqseries.net/phpBB2/viewtopic.php?t=22951&highlight=runmqtrm+aix
for a discussion of the cause of the problem.
If you have v6 or v5.3 CSD11 or above, the problem described by the URL is fixed - the trigger monitor silently ignores the incorrect return code from system().
However, it is possible that the msgs are caused by user error, if either the app genuinely cannot be found or started, or the app exits with a non-zero return code. In those cases, the trigger msg is legitimately put to the DLQ.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 05, 2006 4:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| superhero wrote: |
00000000: 444C 4820 0000 0001 0000 0109 4942 4D4B 'DLH ........IBMK'
00000010: 5031 4534 2E4C 5044 2E49 4E49 5420 2020 'P1E4.LPD.INIT '
00000020: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000030: 2020 2020 2020 2020 2020 2020 4942 4D4B ' IBMK'
00000040: 5031 4534 2020 2020 2020 2020 2020 2020 'P1E4 '
00000050: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000060: 2020 2020 2020 2020 2020 2020 0000 0111 ' ....'
00000070: 0000 0333 4D51 5452 4947 2020 0000 0006 '...3MQTRIG ....'
00000080: 5255 4E4D 5154 524D 0000 0000 0000 0000 'RUNMQTRM........'
00000090: 0000 0000 0000 0000 0000 0000 3230 3036 '............2006'
000000A0: 3035 3031 3031 3137 3535 3737 544D 2020 '050101175577TM '
000000B0: 0000 0001 4942 4D4B 5031 4534 2E4C 5044 '....IBMKP1E4.LPD'
000000C0: 2E53 4552 5632 2E46 494C 4552 4356 2020 '.SERV2.FILERCV '
000000D0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000000E0: 2020 2020 4942 4D4B 5031 4534 2E4C 5044 ' IBMKP1E4.LPD'
etc
|
Looks a lot like a dead letter header to me! 
There appears to be a reason code X'109', which I leave to your good offices to translate. Also appear to be trigger messages which a) could be a clue and b) might explain why the applications are still getting the data through.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 05, 2006 4:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| superhero wrote: |
00000180: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000190: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001A0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001B0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001C0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001D0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001E0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000001F0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000200: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000210: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000220: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000230: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000240: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000250: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000260: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000270: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000280: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000290: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002A0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002B0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002C0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002D0: 2020 2020 2020 2020 7061 6473 6670 6520 ' padsfpe '
000002E0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000002F0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000300: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000310: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000320: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000330: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000340: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
00000350: 2020 2020 2020 2020 ' '
|
But on the plus side, if they fire you for posting this as confidential, it's probably not a good place to work anyway. The paranoia would wear you down.... 
(Unless these are proprietory spaces...) 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| superhero |
| Posted: Wed Jul 05, 2006 5:05 am Post subject: |
|
|
Newbie
Joined: 16 May 2006
Posts: 6
|
hi Nigelg,
thanks a ton for that help. I just checked my MQ version and here are the details: -
->dspmqver
Name: WebSphere MQ
Version: 530.10 CSD10
CMVC level: p530-10-L050504
BuildType: IKAP - (Production)
Since it is CSD10, as per your note, I assume that the fix is not present with us.
| Quote: |
| However, it is possible that the msgs are caused by user error, if either the app genuinely cannot be found or started, or the app exits with a non-zero return code. In those cases, the trigger msg is legitimately put to the DLQ. |
But this does not seem to be the problem since I verified my application log files and found that every thing is perfect and no trace of error there. Since if the application would not have been triggered, then the logs would not have reflected an entry with the same timestamp as the one present in the DLQ for corresponding message. Now, I will ask the admin team to look up and if possible add these patches and hopefully it must solve our problem. And I will definitely get back to you guys whether it is solved or not
Vitor, I understand your frustration since I got angry on myself for not finding this out myself. Guess thats why they call me a "newbie". But seriously, I never knew that the reason code stays there... Can you please provide some link where I can study and learn that which bytes are reason code and what is the rest of stuff in DLH. Also once we get the x109 -> 265(decimal) how to map it with the error code ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 05, 2006 5:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| superhero wrote: |
| Can you please provide some link where I can study and learn that which bytes are reason code and what is the rest of stuff in DLH. Also once we get the x109 -> 265(decimal) how to map it with the error code ? |
You'll find all this in the application programming guide & reference (that's 2 separate books! ) which lays out not only the layout of the DLH and all the other headers which you can find in an MQ message. Basically the Format field describes the message so the MQMD in this case is set to MQDEAD so you know the message has a DLH on it. If you decode the DLH you'll find it has the actual format of your message in it, indicating no other header follows it (they can get stacked up).
These publications also contain all the reason codes possible for an applicatation. There's also a decription of triggering which you might find valuable in this instance; you might also want to try the System Admin publication for an insight into the internal and some of the functions available to control this sort of thing.
You'll find them all (in a convienient PDF format) at:
http://www-306.ibm.com/software/integration/wmq/library/library53.html
(I prefer PDFs - I can never get to the on-line version in a crisis. I like them on my hard drive where I know where they are!)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Wed Jul 05, 2006 5:21 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
If you cannot upgrade to CSD11 you can add the export of the env var in the shell before you start the trigger monitor, to circumvent the AIX bug.
| Code: |
export AMQ_SIGCHLD_SIGACTION=YES
|
The DLH structure, and all the published WMQ structures, are in the Application Programming Reference manual (APR), as are the values for the constants.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jul 05, 2006 5:22 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| superhero wrote: |
| Can you please provide some link where I can study and learn that which bytes are reason code and what is the rest of stuff in DLH. Also once we get the x109 -> 265(decimal) how to map it with the error code ? |
http://www.google.co.uk/search?q=MQDLH |
|
| Back to top |
|
|
| superhero |
| Posted: Wed Jul 05, 2006 8:29 am Post subject: |
|
|
Newbie
Joined: 16 May 2006
Posts: 6
|
|
| Back to top |
|
|
|
|
