| Author |
Message
|
| kayoumt |
 Posted: Tue Sep 18, 2007 1:43 am Post subject: 2009 Errors |
 |
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
Hi,
I'm developping an MQ V6 application with very limited resources : Windows XP on IBM THINKCENTRE Pentium 4. The application has two parts : a single threaded C++ server reading on persistent request queue and sending "IAMLIVE" publish message (via MQRFH publish/subscribe API). The second part (the client) is a C# GUI (main thread and 3 lightweight threads listening on reply queues or sending MQRFH pub/sub messages).
Despite the slowness of the machine, all was working fine since couple months. I was able to run MQ V6 (evaluation version) and Base Broker, and MYSQL, and two Visual Studio .NET instances. And, all was working fine. Server, Client (and only the thread getting published message) were running perfect.
Problems started when I decided to start the three lightweight threads of the client ; running them concurrently with the main C# thread. Thread1 gets message from temporary published messages queue. Thread2 send messages by publishing them on SYSTEM.BROKER.DEFAULT.STREAM. Thread3 does nothing with MQ, but only C# GUI refreshes.
Note that the three threads functions are non-stop loops with 1 to 10 seconds sleep when there's no event to process.
- Trouble 1)
Thread1 reads empty messages after reading couple of correct messages. And, there's no information in the message header telling what the message is and where it comes from and no MQExecption is fired.
- Trouble 2)
Applications(server or client) did not want to connect to MQ. They try and automatically shut down.
- Trouble 3)
Application reply queues are temporary dynamic queues. Normally, after application shutdown, they all have to disappear. But, there's a reply queue that remains on queue manager and it is notified in th log file that "an object is damaged, then Queue manager is corrupted".
Trouble 4)
Many number 2009 errors in log file related to certain system queues :
SYSTEM.ADMIN.COMMAND.QUEUE
SYSTEM.CHANNEL.INITQ
SYSTEM.CLUSTER.COMMAND.QUEUE
SYSTEM.CLUSTER.REPOSITORY.QUEUE
I stopped, started, restarted queue manager, rebooted my machine, but the problem is still there. Do someone have an idea of what's wrong ?
It seems, I'll need to clean or reinstall MQ. But, I need to understand what caused this troubles.
Thanks in advance for any help. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Sep 18, 2007 2:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Do a search in the forum for 2009 errors. They're common, pernicious and do not necessarially have anything to do with your code or indeed your set up.
Aside from the advice you'll find in the many threads discussing this error, I'd start by checking the queue manager logs. You may find some illuminating errors there which may give you a starting point in your investigation.
Good luck!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kayoumt |
| Posted: Tue Sep 18, 2007 12:26 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
I did more investigations and found something that could help having help.
I rebooted my XP P4 512 Meg and restarted MQ.
1) I do no longer have 2009 errors.
2) My C++ application is able to connect to queue manager and open queues except TEMPORARY DYNAMIC QUEUES : completion code 2195, reason number 2.
It seems to be a maximum file open handles limitation due to memory limitation ??? Why do MQ suddenly raises this problem and never did it since couple months I'm running it ??? Seems something is corrupted ???
What MQ wrote in log files :
In "C:\Program Files\IBM\WebSphere MQ\errors\AMQERR01.LOG", it notifies that there has been an internal error.
In "C:\Program Files\IBM\WebSphere MQ\Qmgrs\QM_CADS6\errors\AMQERR01.LOG", MQ says that it started listeners, but failed to keep them running.
FDC dump :
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- mar. septembre 18 16:02:02 Est (heure d'été) 2007 |
| Host Name :- XPIBM (Windows XP Build 2600: Service Pack 2) |
| PIDS :- 5724H7200 |
| LVLS :- 6.0.0.0 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- AQ143000 |
| Application Name :- MQM |
| Component :- aqqAccessQHeader |
| SCCS Info :- lib/lqm/amqaqqca.c, 1.67 |
| Line Number :- 1213 |
| Build Date :- May 24 2005 |
| CMVC level :- p000-L050524 |
| Build Type :- IKAP - (Production) |
| UserID :- MUSR_MQADMIN |
| Process Name :- C:\Program Files\IBM\WebSphere MQ\bin\amqzxma0.exe |
| Process :- 00003236 |
| Thread :- 00000001 |
| QueueManager :- QM_CADS6 |
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 8709 |
| ConnId(4) App :- 2 |
| Major Errorcode :- arcE_OBJECT_DAMAGED |
| Minor Errorcode :- OK |
| Probe Type :- INCORROUT |
| Probe Severity :- 2 |
| Probe Description :- AMQ6125: Une erreur interne WebSphere MQ s'est |
| produite. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+ |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Sep 18, 2007 12:34 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
"arcE_OBJECT_DAMAGED" makes me sad. You may have a damaged/corrupted queue... possibly the default MODEL queue or the one that you're using as the MODEL queue.
You can try "strmqm -c" to recreate the default definitions. Be aware that this will revert any changes you have made to the default definitions.
_________________
I am *not* the model of the modern major general.
Last edited by jefflowrey on Tue Sep 18, 2007 12:42 pm; edited 1 time in total |
|
| Back to top |
|
|
| tleichen |
| Posted: Tue Sep 18, 2007 12:37 pm Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
| kayoumt wrote: |
...
2) My C++ application is able to connect to queue manager and open queues except TEMPORARY DYNAMIC QUEUES : completion code 2195, reason number 2. |
You mean completion code 2 and reason code 2195.. 
| kayoumt wrote: |
| ...It seems to be a maximum file open handles limitation due to memory limitation ??? Why do MQ suddenly raises this problem and never did it since couple months I'm running it ??? Seems something is corrupted ??? |
Whenever I've seen an application get a 2009 in the past, it has invariably been caused by an MQ call getting a bad reason code, but the application did not handle it and went on making subsequent MQ calls without regard to the prior error. For example, if you fail on an MQCONN, it makes no sense to attempt an MQOPEN. Or if you fail on an MQOPEN, then you cannot do an MQGET or MQPUT on that queue.
If the connection gets broken (i.e., at the IP level), subsequent MQ calls will also get a 2009 rc.
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| kayoumt |
| Posted: Tue Sep 18, 2007 3:56 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
"strmqm -c" did unfortunately not solve the problem. But, removing the queue manager and recreating it and recreating model queues made it work.
I'm still wondering what corrupted objects in queue manager ???
Additional information : I fixed a bug in my code. For closing connection to queue manager, I was calling Close instead of Disconnect. May be there's relation between 2195 an 2009 errors and this bug ? |
|
| Back to top |
|
|
| tleichen |
| Posted: Tue Sep 18, 2007 4:54 pm Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
| kayoumt wrote: |
"strmqm -c" did unfortunately not solve the problem. But, removing the queue manager and recreating it and recreating model queues made it work.
I'm still wondering what corrupted objects in queue manager ???
Additional information : I fixed a bug in my code. For closing connection to queue manager, I was calling Close instead of Disconnect. May be there's relation between 2195 an 2009 errors and this bug ? |
As I recall, 2195 is an "unexpected error". This typically results from a bad installation or one that has been corrupted in some way (usually the libraries, not the qmgr objects). That was probably the case and why the reinstall made it work. As for 2009s, see my earlier post.
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| kayoumt |
| Posted: Tue Sep 18, 2007 5:15 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
Tleichen,
Are you telling me that 2195 could occur again ???
I ask you the question, because I did not reinstall my system. I just removed the queue manager, and it had been hard to do it from the MQ Explorer ; for your information, I tried at least three times before getting queue manager completely removed.
Is there a way for making sure MQ installation is correct or not ??? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Sep 18, 2007 10:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kayoumt wrote: |
Are you telling me that 2195 could occur again ???
|
Sometimes bad things happen. One cause is bad install, another more common one is a "feature" in the software going off.
One way of mitigating this is to apply maintenance. The FDC says you're on plain v6; you should be on v6.0.2 at least.
Won't eliminate 2195, but will help.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kayoumt |
| Posted: Fri Sep 21, 2007 2:24 am Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
Hi,
My multithread c# (client) application still has trouble getting publish/subscribe replies from a temporary dynamic queue ; while the c++ application (server) also connected to MQ seems to be stable.
In the log file there is a mix of 2052, 2085, MQ Broker, TCP/IP ... errors. I did not be able to catch 2052 errors with my Visual Studio debugger.
I saw also something curious with 2033 reason code. Calling MQQueue.CurrentDepth raised 2033 error some times. This code is supposed to be returned when MQ GET is called on an empty queue. Did I have something to clean before calling CurrentDepth on the queue ?
In MQ Client log files. Il saw also "Duplicate AMQXCS2.dll found" in an FDC.
Are all these troubles related ? Any idea of what is rong in my very simple MQ V6.0 installation.
----- amqfqxaa.c : 679 --------------------------------------------------------
09/21/2007 04:23:26 AM - Process(5852.8) User(mqm) Program(amqfcxba)
AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the
dead-letter queue.
EXPLANATION:
The broker has written a message to the dead-letter queue
(SYSTEM.DEAD.LETTER.QUEUE ) for reason
'2052:MQRC_Q_DELETED'. Note. To save log space, after the first occurrence of
this message for stream (SYSTEM.BROKER.DEFAULT.STREAM ), it
will only be written periodically.
ACTION:
If the message was not deliberately written to the dead-letter queue, for
example by a message broker exit, determine why the message was written to the
dead-letter queue, and resolve the problem that is preventing the message from
being sent to its destination.
----- amqfqxaa.c : 679 --------------------------------------------------------
09/21/2007 04:26:16 AM - Process(4446.49) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3224 -------------------------------------------------------
09/21/2007 05:33:07 AM - Process(5852.8) User(mqm) Program(amqfcxba)
AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the
dead-letter queue.
EXPLANATION:
The broker has written a message to the dead-letter queue
(SYSTEM.DEAD.LETTER.QUEUE ) for reason
'2085:MQRC_UNKNOWN_OBJECT_NAME'. Note. To save log space, after the first
occurrence of this message for stream (SYSTEM.BROKER.DEFAULT.STREAM
), it will only be written periodically.
ACTION:
If the message was not deliberately written to the dead-letter queue, for
example by a message broker exit, determine why the message was written to the
dead-letter queue, and resolve the problem that is preventing the message from
being sent to its destination.
----- amqfqxaa.c : 679 --------------------------------------------------------
09/21/2007 05:34:50 AM - Process(4446.52) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3224 -------------------------------------------------------
09/21/2007 05:36:07 AM - Process(4446.53) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3224 -------------------------------------------------------
09/21/2007 05:36:27 AM - Process(4446.54) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3224 -------------------------------------------------------
09/21/2007 05:37:47 AM - Process(4446.55) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3224 ------------------------------------------------------- |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Sep 21, 2007 2:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Ok, try this:
1) Create a local permanent queue DEAD.LETTER
2) Alter the queue manager to use DEAD.LETTER as the system DLQ
3) Explain why you're using MQQueue.CurrentDepth
4) Retry your application
5) Check DEAD.LETTER, obtaining the reason code from the DLH of any messages present
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Sep 21, 2007 2:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- mar. septembre 18 16:02:02 Est (heure d'été) 2007 |
| Host Name :- XPIBM (Windows XP Build 2600: Service Pack 2) |
| PIDS :- 5724H7200 |
| LVLS :- 6.0.0.0 | |
Patch that qmgr soonest (au plus tot). 6.0.0.0 is not the stablest version.
Move right up to 6.0.2.2 !
Stable versions of V6 start at 6.0.1.1
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kayoumt |
| Posted: Fri Sep 21, 2007 1:29 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
[quote="Vitor"]Ok, try this:
1) Create a local permanent queue DEAD.LETTER
2) Alter the queue manager to use DEAD.LETTER as the system DLQ
3) Explain why you're using MQQueue.CurrentDepth
4) Retry your application
5) Check DEAD.LETTER, obtaining the reason code from the DLH of any messages present
[/quote]
CurrentDepth - Just For monitoring purposes. Isn't it the right member (or function) to call to findout how many messages are waiting in the queue ?
Dead Letter - I terminated writing some code for dumping DLH fields in a file. I'll also add in my queue manager creation script the commands below :
DEFINE QLOCAL(CADS6.DEAD.LETTER.QUEUE) LIKE(SYSTEM.DEAD.LETTER.QUEUE)
ALTER QMGR DEADQ(CADS6.DEAD.LETTER.QUEUE)
[quote="fjb_saper"]Patch that qmgr soonest (au plus tot). 6.0.0.0 is not the stablest version.
Move right up to 6.0.2.2 !
Stable versions of V6 start at 6.0.1.1 [/quote]
Yes ! Il était temps. I'm going to upgrade to 6.0.2.2 right now. |
|
| Back to top |
|
|
| kayoumt |
| Posted: Fri Sep 21, 2007 1:35 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
Sorry,
I tried to put in gray references to your posts. But, t did not work. I extracted below the message without references to yours.
CurrentDepth :
Just For monitoring purposes. Isn't it the right member (or function) to call to findout how many messages are waiting in the queue ?
Dead Letter :
I terminated writing some code for dumping DLH fields in a file. I'll also add in my queue manager creation script the commands below :
DEFINE QLOCAL(CADS6.DEAD.LETTER.QUEUE) LIKE(SYSTEM.DEAD.LETTER.QUEUE)
ALTER QMGR DEADQ(CADS6.DEAD.LETTER.QUEUE)
Stable version :
Yes ! I'm going to upgrade to 6.0.2.2 right now (il était temps). |
|
| Back to top |
|
|
| kayoumt |
| Posted: Fri Sep 21, 2007 6:07 pm Post subject: |
|
|
Voyager
Joined: 18 Sep 2007
Posts: 81
|
Hello,
Re-installed :
- re-installed V6 and added 6.0.2.2 on server (LINUX)
- re-installed V6 and added 6.0.2.2 on client (XP)
- defined local DLQ
But, always getting '2052:MQRC_Q_DELETED'.
See last queue manager log records and DLQ dump below.
----- amqccita.c : 3070 -------------------------------------------------------
09/21/2007 09:45:27 PM - Process(8657.21) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host 192 (192.168.2.10).
EXPLANATION:
An error occurred receiving data from 192 (192.168.2.10) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 104 (X'68'). Record these
values and tell the systems administrator.
----- amqccita.c : 3070 -------------------------------------------------------
09/21/2007 09:48:04 PM - Process(8646.9) User(mqm) Program(amqfcxba)
AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the
dead-letter queue.
EXPLANATION:
The broker has written a message to the dead-letter queue
(CADS6.DEAD.LETTER.QUEUE ) for reason
'2052:MQRC_Q_DELETED'. Note. To save log space, after the first occurrence of
this message for stream (SYSTEM.BROKER.DEFAULT.STREAM ), it
will only be written periodically.
ACTION:
If the message was not deliberately written to the dead-letter queue, for
example by a message broker exit, determine why the message was written to the
dead-letter queue, and resolve the problem that is preventing the message from
being sent to its destination.
----- amqfqxaa.c : 679 --------------------------------------------------------
STRUCT_ID : DLH
VERSION : 1
REASON_CODE : 2052
DEST_QUEUE_NAME : CADS6.192.168.2.10_2007.09.21_20.03.10.PS
DEST_QMGR_NAME : QM_CADS6
ENCODING : 546
CCSID : 1208
FORMAT : MQHRF
APPLI_TYPE : 26
APPLI_NAME : QM_CADS6
DATE : 20070922
TIME : 01461589
DATA : RFH � " � MQSTR MQPSCommand Publish MQPSQMgrName QM_CADS6 MQPSPubOpts NoReg MQPSQName CADS6.192.168.2.10_2007.09.21_19.59.14.PS MQPSTopic 2/PS/192.168.2.10_2007.09.21_19.59.14 <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<message>
<header>
<Session>192.168.2.10_2007.09.21_19.59.14</Session>
<Topic>2</Topic>
<TopicDesc>TOPIC_002_IAM_ALIVE</TopicDesc>
<Type>PUBLISH</Type>
</header>
<parameterlist>
<parameter name="ReplyQDepth" type="Xml">0</parameter>
<parameter name="SubscribeQDepth" type="Xml">0</parameter>
</parameterlist>
<body></body>
</message> |
|
| Back to top |
|
|
|
|
