| Author |
Message
|
| SAFraser |
 Posted: Thu Oct 23, 2008 4:29 am Post subject: Garbage Characters Appear Randomly |
 |
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
We have a request message that comes from a (non-IBM) mainframe queue manager in a string, to a Solaris queue manager. WMB transforms it to XML, app picks it up and returns an XML response. WMB transforms the XML to a string and it's sent back to the mainframe. Very standard request-reply using a broker, I think.
Like this:
MF app puts request to qremote on MF qmgr.
Request received on qlocal on Solaris qmgr.
WMB transforms string to XML and puts onto another qlocal.
Client app retrieves request, returns response to Solaris qlocal.
WMB transforms XML to string and puts onto qremote to MF.
MF receives response in MF qlocal.
MF app consumes response message.
Several days ago, two (out of dozens) of mainframe applications that use this routine began aborting with format errors. When we re-run the programs, they abort again but never in the same place. In other words, a record that causes an abort on one run is processed successfully on another, with the program aborting on a different record.
We were finally able to reproduce the behavior in development. The message on which the program aborts has garbage characters appended on the end, like these, about 350 bytes appended to the end of the message:
†÷ Œ †ö “ H
I don't even know what some of these characters are!
I'm now writing the entire message contents of all four "stops" on the Solaris server to a text file via a trace node: string in, XML in, XML out, string out. All four look perfect for the "bad" message, absolutely no hint of garbage anywhere.
We're attacking this from many angles, and I thought getting some forum ideas would be one to try. Thoughts? Questions?
Last night when I called Console Ops to re-run the jobs, I asked them to dance around the console shaking their car keys and chanting "Error, be gone" as they kicked off the job. I have it on good authority that they did it. That's just how desperate we are. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Oct 23, 2008 5:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
350 bytes of garbage?
hrm... That sounds suspiciously like a header that's been added in the wrong place.
an amqsbcg output of the garbage would be helpful, if it's possible to get. |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Oct 23, 2008 6:24 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
| Great idea. I've just added it to today's task list. I'll post when I get something. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 23, 2008 7:25 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Non-IBM mainframe with MQ? Interesting - can you elaborate? Is it running z/OS? |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Oct 23, 2008 9:14 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
It's a Unisys machine. I believe that Unisys bought the HP-UX code from IBM and ported it; they market the product as MQS2200. It walks, talks and quacks just like IBM's version.
Current working theory is that the COBOL program which is consuming the message is grabbing more bytes from the message buffer than it should, that it is not properly recognizing the end of the message data. (That's where you were going, wasn't it, Jeff?)
More as the situation develops. Thoughts are welcome. Thank you. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Oct 23, 2008 9:18 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I was actually thinking that the Broker message flows were leaving an MQMD or HTTPHeader or etc. in the message tree , which was then getting serialized at the end of the message.
Failing to initialize the COBOL working storage area properly is also a good thought... but I figured you would have seen that one before...  |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Oct 23, 2008 9:49 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
COBOL is just generally a vast sea of mystery to me.
I wrote a full copy of the message to a file via WMB trace nodes everywhere I could: for the request message, the string received from the mainframe, the compute node's transformed XML; for the response message, the XML received from the client app, the compute node's transformed string. No garbage.
I also "trapped" the request and reply message as it made its way around it's transaction cycle. I even stopped the sender and trapped the message in the xmitq. Reviewing the amqsbcg output of all these revealed no garbage.
I predict that if we do an amqsbcg of the receiving queue on the mainframe, that will be clean, too.
The application programmers are chasing it at the moment. Stay tuned! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 23, 2008 3:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We had something like this (not quite) that had us baffled for a while.
We were receiving text messages with convert in CICS and all of a sudden their content turned up to be garbage.
Reason was we had upgraded the MF OS and there was a bug in which MQ let the CICS OS handle translation of CCSID but the some of the used pointers kept going bad. A PMR & PTF fixed it.
Just mentioning it to have you check you're not running into something equally crazy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Oct 23, 2008 6:22 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
| The prod instance of MQ was recently upgraded.... Thanks, we will definitely chase this. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 23, 2008 6:29 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| COBOL is just generally a vast sea of mystery to me. |
And it's not going away any time soon (defined: in my lifetime or yours).
I seem to recall Gartner estimating several million lines of new COBOL coded last year or the year before.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Oct 23, 2008 6:35 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
I may not know how to read COBOL code, but I know it's fast and it gets the job done.
Unlike java. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 24, 2008 1:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| SAFraser wrote: |
I may not know how to read COBOL code, but I know it's fast and it gets the job done.
Unlike java. |
You mean unlike java in it's early days! With the nio classes and fast processors these days you can write optimized java that gets the job done at quite a pace... Granted you'd need multiple jvms to do what happens in a single zOS instance but the speed argument is "passe".
Each language has its strengths and its weaknesses. Cobol does very good group change handling but poor individual String manipulations, or at least it used to.... Haven't played with that new object oriented cobol stuff yet....
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
