| Author |
Message
|
| hari_k98 |
 Posted: Mon Dec 05, 2011 12:47 am Post subject: How to investigate Message loss |
 |
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
Hi
Iam using MS BizTalk to write to a local persistent queue.
The adapter used is MQ Series - Server Adapter(MQSC).
Another application running on Unix reads from the same queue.
We occasionally face the issue of message not received by the reading application.
There is no error messages in Biztalk or MQ server at that time.
I need establish that the message has been written successfully to the queue.
Can somebody let me know how to investigate this loss of messages?
Configuration Details
BizTalk Server 2006 R2
MQ Series Server Adapter (MQSC)
MQ Server 6.0 running on Windows 2003
Local Queue Persistent type
Thanks
Hari |
|
| Back to top |
|
 |
| exerk |
| Posted: Mon Dec 05, 2011 1:15 am Post subject: Re: How to investigate Message loss |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| hari_k98 wrote: |
| Iam using MS BizTalk to write to a local persistent queue. |
Queues are NOT persistent, messages are (or are not depending on the options set by an application)...
| hari_k98 wrote: |
Another application running on Unix reads from the same queue.
We occasionally face the issue of message not received by the reading application.
There is no error messages in Biztalk or MQ server at that time.
I need establish that the message has been written successfully to the queue. |
That is generally known as application logging/auditing...
| hari_k98 wrote: |
| Can somebody let me know how to investigate this loss of messages? |
Who says, or can prove, that messages are being 'lost'?
Obvious questions:
1. Is expiry being set on the messages?
2. Is the application explicitly setting non-persistence, thereby over-riding the queue attribute?
3. How have you definitively diagnosed that all messages have been written to the queue?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Mon Dec 05, 2011 1:33 am Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
Thanks for your reply.
From BizTalk Application I have a report which logs the timestamp of message entry.
The receiving application has confirmed that it has received the previous and next immediate messages.
When I checked the logs in Biztalk and MQ server there is nothing to suggest Biztalk applcation failed to write the message to MQ.
But the owner of the receiving application says he never received that particular message.
On daily basis Iam sending 2000 messages successfully acknowledged by receiving application.
Once in a 15 days I get a complain saying they have not received a specific message.
I suspect the receiving application is dropping message.
Now Iam looking for some trace or log at MQ side to say Biztalk has managed to write to the queue successfully. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Dec 05, 2011 2:16 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Does the receiving application have any failures?
Are they using MQGET with SYNCPOINT (and MQCMIT) to protect messages against application failure?
What else happens around the same time? Queue manager restart for example? |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Mon Dec 05, 2011 2:24 am Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
There is no queue manager restart or any network failure at that time.
Not able to get much info from receiving application. Their owner insists every thing fine with application.
Is there any application to read MQ Log files.
When opening with notepad I can locate the message but it is not complete. |
|
| Back to top |
|
|
| JasonE |
| Posted: Mon Dec 05, 2011 3:26 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
Try dmpmqlog.
If the log contains the message, that should mean it arrived as persistent.
So the thing you need to find out is whether it is it in the logs and committed. If it is, then the putting side was fine, and implies the getting side had a problem. If you note the spcidx and track that forward you will hopefully see someone get the message as well explaining why it isnt on the queue.
Personally if its an option I'd trace the whole operation hence the puts/gets (will be massive) but at least you'd be able to see the whole operations going on - identify the msgid getting 'lost' and see what happens to it. Only problem is trace is hard to read  Note: you can use strmqtrc -d -1 if you need to trace the whole user message (default is first and last 48 bytes) so if you need to see the 'data' to identify the acutal message dont forget this bit. |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Mon Dec 05, 2011 4:15 am Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
The complaint of missing messages comes after a week of actual message processing.
So Iam taking a daily back up of circular logs, by copying the folders.
Though the method is clumsy it seems the only way to prove message is written succesfully in to queue.
Atleast this will force the reading application to look into their logs for any error. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Dec 05, 2011 5:44 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hari_k98 wrote: |
| When opening with notepad I can locate the message but it is not complete. |
Are you saying that you open the MQ logs, and that you can see the 'lost' message, but that is incomplete?
On which qmgr did you see this? On the qmgr where the application created the incomplete message? Or on the qmgr where the message was to be consumed?
If the message was logged but is truly incomplete, then I'd be looking at an application failure.
MQ assured delivery means that messages are delivered by MQ without loss or duplication. It has been my experience that loss or duplication occurs due to incorrectly coded applications.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 05, 2011 1:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If the receiving application is not reading under syncpoint and explicitly doing a commit or rollback, there are chances that the message got received but had a problem processing and as such looks like it got lost... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Wed Dec 07, 2011 1:36 am Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
bruce2359,
"Are you saying that you open the MQ logs, and that you can see the 'lost' message, but that is incomplete? "
Sorry for not being clear, what I meant was the logs are not in readable format. The Actual message is complete.
The writing and reading happens from the same queue and queue manager.
The problem is occasionally reading application complains for message not received.But in actual I have my application logs saying message was delivered.
Now Iam pushing for the reading application owner to look into theire logs for any error/warning.
Thanks
Hari |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 07, 2011 5:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hari_k98 wrote: |
| The problem is occasionally reading application complains for message not received.But in actual I have my application logs saying message was delivered.i |
How does it know that one particular message is not delivered? By the MsgId? CorrelId? Something else?
If a message is persistent, it will be logged in the WMQ logs. Non-persistent messages are not logged. This presumes that the application did, in fact, successfully issue mqget/mqput API calls.
Included in the WMQ installation, and described in the WMQ System Administration documentation, is a dmpmqlog utility. It prints the log is human-readable format.
One of the ways to chase the mythical 'lost message' is to get-inhibit the queue; then run the app that mqputs the message to the queue; then display the message to make certain that it is correct. Display the log to verify that the message was, in fact, logged.
Next, run the app that mqgets the message into the queue. Display the log again.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Wed Dec 07, 2011 10:36 pm Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
Each being read by the application is stored in a database.
End user using one of fields in the message searches for the message.
They find a particular message is not in the database.
Whatever logs in the receiving side of the applcation does not show any error or warning.
I understand the dump utility needs the queue manager to be down for extracting the log.
We cannot afford any downtime during the week.
This has left me with the only option of copying the log folders and wait for the next occurrence of the issue.
Only concern is the log is not readable format and there is no time stamp in the log. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 07, 2011 10:51 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hari_k98 wrote: |
| Only concern is the log is not readable format and there is no time stamp in the log. |
The dmpmqlog utility will print the log in human-readable format. There is a time-stamp in GMT in all log entries.
How soon after a message is detected as 'lost' will you be notified? Minutes, hours, days, weeks?
Are you using circular or linear logs?
Let's say that the 'lost' message is in the logs. Will you be able to identify it? How will you be able to identify it?
Please, oh please, read about WMQ logs in the WMQ Systems Administration document or InfoCenter equivalent. The dmpmqlog utility documentation will show you how to print selected portions of the log.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hari_k98 |
| Posted: Wed Dec 07, 2011 11:52 pm Post subject: |
|
|
Novice
Joined: 13 Jul 2007
Posts: 14
Location: India
|
Earliest notification we have received so far is 4 days.Some times end user comes after 2 weeks also on missing messages.
It is the Circular log type.
Yes i can identify the message using a unique value inthe message which iam capturing in my app report before sending to MQ.
yes I will read the doc about dmpmqlog.
Thanks a lot. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 08, 2011 12:37 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Unless you have obvious error messages in your MQ error logs - I would suggest that you stop wasting your time trying to find a mysterious (and fundamental) bug in IBM's software.
Instead look at the application. Almost every MQ application I have ever inspected has been written badly. Because people assume MQ is simple and fail to grasp the nature of transactional design. Presumably this is not taught at University. |
|
| Back to top |
|
|
|
|
