ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Triggering on Linux is creating messages on the dead letter

Post new topic  Reply to topic
 Triggering on Linux is creating messages on the dead letter « View previous topic :: View next topic » 
Author Message
bduncan
PostPosted: Wed Apr 18, 2001 3:31 pm    Post subject: Reply with quote

Padawan

Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley

Problem Description: For some time now, we have been noticing that the SYSTEM.DEAD.LETTER.QUEUE on some of our machines would begin to fill up over time. Several times, this has actually filled up /var/mqm which caused the queue manager to break. It became clear that this was only occurring on those machines that had triggering enabled. This was supported by the fact that when the messages on the dead letter queue were browsed, they were all copies of the trigger messages that were supposed to be getting routed to the SYSTEM.DEFAULT.INITIATION.QUEUE. What threw us off at first was the fact that even though the trigger messages appeared to be going to the dead letter queue for some unknown reason, the triggered applications were still getting launched. Because the triggering was working, this problem took a low priority, and it didn’t become necessary to look at it again until the dead letter queues on several machines started getting very, very large. As an experiment, I ran the trigger monitor in the foreground so I could see exactly what happened when a message landed on a triggered queue. One point about the way MQ triggers applications - it always tries to pass a large string representing a Trigger Message Header Structure as a parameter to the program it launches. This is for those programs that need to know certain information from MQ to function properly. In our case, none of our programs required this information, but there was no way to turn this mechanism off. So what I observed while watching the trigger monitor was the fact that when a message would land on the triggered queue, the trigger monitor would indeed execute our program, but then the shell that the trigger monitor was running under would complain about a non-existent file or unknown command error, and this error always pointed at the Trigger Message Header. So it appeared that the command that the trigger monitor was passing to the shell to execute was corrupted somehow, and the shell was returning errors. When this occurs, the trigger monitor is designed to create a message describing the error and dropping it on the dead letter queue, which is what we observed.

Problem Solution: I called IBM support on this problem, since it seemed like other people must have run into this same problem. This turned out to be the case, and we saved a lot of time by not having to troubleshoot this ourselves. It turns out that the MQ Series trigger monitor program uses the system function to execute triggered applications. The system function takes a single string as a parameter, and the OS attempts to treat the string like a shell command. The people at IBM informed me that the way MQ constructs this string is a little tricky. It essentially takes three fields in the PROCESS structure we defined within MQ Series to trigger our application. These three fields are APPLICID, USERDATA, and ENVRDATA. APPLICID simply contains our command - we had it set to something like "csh /usr/db/feed/bin/autostartfeeds.csh &" and we left USERDATA and ENVRDATA blank. The trigger monitor will construct the system parameter by concatenating the fields in this order: APPLICID + USERDATA + Trigger Message Header + ENVRDATA. Notice that it drops the Trigger Message Header into the parameter. So our problem was occurring as follows. The system call would pass the concatenated string to the OS, and the OS would execute it like it was a shell command. It would begin to parse the string, and as it moved through the APPLICID field, it would get to the last character, a "&" and stop. The ampersand is telling the shell to execute the preceding command in the background. It also has a second, more subtle purpose - it acts much like a semicolon does in a piece of Perl code. It informs the shell that this is the end of a command. So the shell goes and executes our application without any complaints. But this wasn’t the end of our string. I mentioned that we left the USERDATA and ENVRDATA fields blank, but MQ automatically stuck the Trigger Message Header in between those two blank fields. So essentially the shell would consider this a second command (after correctly executing our application) and since this structure looks nothing like a shell command, the shell would return a syntax error to the trigger monitor, which would in turn create a message and drop it on the dead letter queue. This explains the odd symptom that our programs were being triggered correctly, but we were still getting messages on the dead letter queue. IBM support had a very simple fix for this problem. Take the ampersand out of the APPLICID field and throw it into the ENVRDATA field. This way, the ampersand comes after the Trigger Message Header, which means that the entire string will be treated as a single command by the system function.


_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator

[ This Message was edited by: bduncan on 2001-05-16 14:12 ]

[ This Message was edited by: bduncan on 2001-05-16 14:12 ]

[ This Message was edited by: bduncan on 2001-05-16 14:13 ]
Back to top
View user's profile Send private message Visit poster's website AIM Address
donthomas
PostPosted: Thu Nov 29, 2001 8:55 am    Post subject: Reply with quote

Novice

Joined: 28 Nov 2001
Posts: 23
Location: EDS, Pittsburgh, PA

Brandon,
I am having a similar problem on an NT server. Triggering is on and set to FIRST. A message comes in which satisfies the trigger evetn. A message is placed on the INITQ and the trigger monitor removes it and starts the application. After this, or before, I'm not sure which, I get a message on the dead letter queue for that qmgr saying that MQ was unable to start the application, even though the application has run successfully. Unlike your problem though our process is a straight path statement to a directory on the NT server. (i.e. d:appsapp.bat)There are no other parameters for the OS to misinterpret. Do you have nay ideas why this happening?
Back to top
View user's profile Send private message Send e-mail
StefanSievert
PostPosted: Thu Nov 29, 2001 3:23 pm    Post subject: Reply with quote

Partisan

Joined: 28 Oct 2001
Posts: 333
Location: San Francisco

Hi Don,
this might be a little different from the (unix specifc) problem that Brandon described. The trigger monitor starts a process by

rc = system(yourapplicid)

And then tests rc for zero. If it is non-zero, it will generate a DLQ message, saying that the application didn't run successfully. If you run the trigger monitor on NT in a DOS Window, you can see that messsage, too.
There are basically two solutions to that:
1) Modify your application to return(0) or exit(0) at the end of successfull processing or
2) prefix your APPLICID with the START command, e.g. START yourbatchfile.bat

You could also call a generic batchfile, which contains the line "START %2" and specify the command to be executed in the USERDATA attribute of your process definition. As Brandon pointed out correctly, the USERDATA attribute is the second argument that gets passed to the triggered APPLICID. Notice, however, that this will work only, if your APPLICID attribute has a maximum length of 255 bytes, that is 1 byte less than the maximum allowed length; and contains no blanks or is enclosed in quotes. Otherwise there would be no blank to seperate the command from the USERDATA field, thus %2 would return a different value.

Brandon, putting the & in the ENVDATA field is a neat solution. However, you could have solved your problem with the same approach on unix, I guess, by ignoring the command line arguments and start your application from within the script, adding the & to the end of the line. Right?!

I hope I could provide some additional thoughts!
Cheers,
Stefan

_________________
Stefan Sievert
IBM Certified * MQSeries

[ This Message was edited by: StefanSievert on 2001-11-29 15:24 ]
Back to top
View user's profile Send private message
donthomas
PostPosted: Fri Nov 30, 2001 9:58 am    Post subject: Reply with quote

Novice

Joined: 28 Nov 2001
Posts: 23
Location: EDS, Pittsburgh, PA

Stefan,
Thanks for your reply. We went with option #2 above. Try as we might we were not able to determine the value of the return code. It wasn't displayed in the DOS window when I ran the trigger monitor in the foreground. But, I think it's safe to assume that it was a value other than zero because when we changed the process definition to issue the START command instead of just passing the path to the batch file the problem went away. Thanks again for your help in getting this pesky issue resolved.

_________________
Don Thomas
412-203-3539
Back to top
View user's profile Send private message Send e-mail
donthomas
PostPosted: Tue Dec 04, 2001 11:07 am    Post subject: Reply with quote

Novice

Joined: 28 Nov 2001
Posts: 23
Location: EDS, Pittsburgh, PA

Stefan,
It appears that I spoke to soon. While using th .bat file and using the START command solved the problem of getting the message on the dead letter queue, it seems that using the .bat files is causing CMD.EXE processes to be started n the server. (Forgive my programming ignorance please). The application is long running, in that was started it will continue ro run and check the queue for data until a 10 minute period has pasesed without any data being found on the queue. What this seems to be doing though is leaving the CMD.EXE process generated by the .bat file running so that after several days I have 500+ of them in the task manager on the server. This seems to be having a very disruptive effect on all of the trigger monitors for that queue manager. After running for a couple of hours all of the INITIQ's that have a trigger monitor running against them start to throw messages on the dead letter queue with MQFB_265 APPL_CANNOT_BE_STARTED. Do you have an example of how a VB app can send a return code of zero to the OS so that maybe we can get around the problem of runmqtrm not accepting anything other than zero?
Back to top
View user's profile Send private message Send e-mail
StefanSievert
PostPosted: Tue Dec 04, 2001 3:07 pm    Post subject: Reply with quote

Partisan

Joined: 28 Oct 2001
Posts: 333
Location: San Francisco

Don,
I'll take the blame for that, I should have thought of it...
You need to code an EXIT statement at the end of your batch file. This will close the command window at the end of execution.

As to your VB question: I don't know how to return a completion code from a VB program. You might find a solution to that in the VisualBasic online help. But the EXIT command should do the trick.

Hope that solves your issue in a persistent way,
Stefan

NB. The disruptive effect you've seen is most likely to be caused by Windows running out of resources to start another CMD window.

_________________
Stefan Sievert
IBM Certified * MQSeries

[ This Message was edited by: StefanSievert on 2001-12-04 15:13 ]
Back to top
View user's profile Send private message
bduncan
PostPosted: Thu Dec 06, 2001 10:46 am    Post subject: Reply with quote

Padawan

Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley

Stefan,
You are correct, I could have put the '&' in the script itself to avoid this problem, but this solutions creates another problem. It means that any application I run needs to be wrapped in a script. In this example it is fine, because what I was running was already a script, but in some instances, I was triggering a binary file directly, in which case the solution I specified is the only way to get it to work unless I wrap it in a dummy script...


_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator
Back to top
View user's profile Send private message Visit poster's website AIM Address
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » Triggering on Linux is creating messages on the dead letter
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.