| Author |
Message
|
| angelocook |
 Posted: Tue Sep 12, 2006 6:03 am Post subject: Recieving MQRC_Q_FULL Error |
 |
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
| I have just started receiving the MQRC_Q_FULL message in my applications that have been running perfectly fine for several months. I have 3 major queues in one queue manager, each of the queue has a max depth set at 9000000 and the default max message length of 4194304. The queue is set for persistance mode and the queues are located on a 13gb drive. When the queues are completly loaded with the work to be done, generally this is 400k message they are worked and move to another queue for logging. this has been working fine for at least a year. The only change is that more workers (threads) were added to handle the work loads. The messages being put on queue are small in nature and they are ASCII based from txt files which are 500mb and less then 500,000 messages. how am I reaching the FULL indicator? |
|
| Back to top |
|
 |
| kevinf2349 |
| Posted: Tue Sep 12, 2006 6:12 am Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
| Uncommited messages maybe? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Sep 12, 2006 6:13 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Did you confirm that the queue depth was less than 9000000?
Did you read the full description of MQRC_Q_FULL in the Messages manual, and follow all the suggestions it gives for handling this error?
Did you confirm that you are either not using Syncpoint or are making sure to always commit your PUTS and GETs?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 6:48 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
I have not looked in the messages manual yet, i forgot about that one. I will check those. As far as the uncommited messages, I under the impression that we coded for commit after put. I will also check on that. My main concer is that i may have too many clients connecting to the MQ server, the server has no errors or warnings about being full or anything.
Strange. |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Sep 12, 2006 7:03 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| angelocook wrote: |
| My main concer is that i may have too many clients connecting to the MQ server |
The number of clients connecting should not in itself be a problem. The number of messages is the problem if you get _Q_FULL.
It doesn't surprise me that MQ writes no errors - this is not a condition that in itself is a cause for concern.
You may wish to look into investigating this condition by using (a) queue depth events, or (b) API exits on MQPUT. |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 7:10 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
But what concerns me is that this just started to happen when I have more clients processing the messages. if I reduce the number of clients the problem goes away but I process longer. I feel that i may have staturated my MQ servers connection to the storage of the data. we are using ISCSI connection to the data drive on a SAN for the storage of the queues.
thanks for all of the quick posts and validations. I will continue to look into this. |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Tue Sep 12, 2006 7:18 am Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
| Quote: |
| I under the impression that we coded for commit after put |
| Quote: |
| But what concerns me is that this just started to happen when I have more clients processing the messages. |
Then I would definately check that the application is committing correctly.
Also check that the getting application is commiting after the GET too...and that it is doing a destructive get.
How is the reading(getting application) running? Does it run all the time? |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 7:47 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
this is the code I am using (.Net), this is just a rough skeleton of the code.
I am not sure of the extended options, but this is the general code minus the error checks etc..
<-- snippet -->
public void Connect(string QueueName)
{
_MessageQueue = _QueueManager.AccessQueue( QueueName, MQC.MQOO_OUTPUT + MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_INQUIRE + MQC.MQOO_FAIL_IF_QUIESCING );
}
public string GetMessage()
{
MQMessage retrievedMessage = new MQMessage();
//Set the get message options..
MQGetMessageOptions gmo =new MQGetMessageOptions();
//accept the defaults same as MQGMO_DEFAULT get the message off the queue..
_MessageQueue.Get( retrievedMessage, gmo);
return retrievedMessage.ReadString(retrievedMessage.MessageLength);
}
public void PutMessage(string MessageText, MQ_MessagePriority MsgPriority)
{
MQMessage mMessage =new MQMessage();
mMessage.Priority = (int)MsgPriority;
mMessage.ClearMessage();
mMessage.WriteString( MessageText );
//specify the message options...
MQPutMessageOptions mMsgPutOpt =new MQPutMessageOptions();
//accept the defaults, same as MQPMO_DEFAULT constant put the message on the queue
_MessageQueue.Put(mMessage, mMsgPutOpt);
} |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Sep 12, 2006 10:12 am Post subject: |
|
|
Guest
|
The Q_FULL message means exactly that: the queue has reached max queue depth.
Only two causes (and they are interrelated): the program(s) are MQPUTing messages to the queue and/or the the getting MQGETing isn't getting them (at all, or fast enough).
A possible technical solution: increase the max queue depth - this may help in the short-term. If the getting application is triggered by messages arriving in the queue, check trigger settings, check for trigger monitor error messages, trigger messages in the dead letter queue.
Your are correct in your assumption that more clients (running the PUTing application) will fill the queue faster. This may be good for business - unless the puting application has coding errors.
I'm assuming that your application architects said that 900000 is a reasonable number of messages to expect to be put to the queue. If not, the puting application has run amock. |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 10:34 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
I have opened a case with IBM on this issue. From what I see and have designed there is no way that any of the queue(s) are getting full.
6000 flat files (2 sets 3000 each)
first set contain one line entries (400000 total individual pieces of work)
second set contains 26 lines per piece of work (400000 total individual pieces of work)
total size 400mb
Process1:
reads file set1 puts each entry on Queue1 for processing
reads file set2 puts each group on Queue2 for processing
Process2: get message from Queue1, processes message and put results on Queue3 for logging
Process3: get message from Queue2, processes message and put results on Queue3 for logging
Process3: get message from Queue3 and logs into RDBMS for review
::REPEAT:: |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Sep 12, 2006 10:49 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
What does the CURQDEPTH of the queue show when the error occurs?
What does a BROWSE of the queue show when the error occurs?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 10:53 am Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
| i am going to monitor the processes this evening. currently there is little work to be done. our bulk or heavy loads start at 6pm-7:30pm. i have perfmon running so i can retain the stats. |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Tue Sep 12, 2006 11:40 am Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
Which queue is giving queue full?
Queue3 by any chance? |
|
| Back to top |
|
|
| angelocook |
| Posted: Tue Sep 12, 2006 12:24 pm Post subject: |
|
|
Newbie
Joined: 12 Sep 2006
Posts: 9
|
from the psuedo code posted before, its Queue3 and that process can handle 3000 per second and has never been above 4000 messages.
this is the strange part. I have spent alot of time designing and planning this, and in the year this has been functioning only in the last 4 days did i start getting this queue full error.
My IBM tech does not think its network traffic or number of clients (which went from 5 to 9, the only other change) the volume of work is a constant.
we deal with flights, a set number (3000 +-) and forcast for 1 year (365 days), do the math, max entries 1095000 +-, the kicker is we only forecast about 1/3 of those which gives me my average 400k entries. we have done a full forecast in the past processing all 1095000 with no issues.
I am stumpted. I am preparing to do trace logging for the IBM team to look into, of course it will be my luck that the problem wont manafest itself. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Wed Sep 13, 2006 12:06 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
| Quote: |
What does the CURQDEPTH of the queue show when the error occurs?
What does a BROWSE of the queue show when the error occurs?
|
You did not answer these questions.
IBM support will certainly ask the same questions before even considering looking at a trace.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
|
|
