| Author |
Message
|
| RatherBeGolfing |
 Posted: Fri Feb 02, 2007 12:44 pm Post subject: AMQRFCDA.DAT filesystem error AND AMQ6037 MQ Storage problem |
 |
|
Centurion
Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA
|
We're running MQ 5.3 with FixPack 7 (  yeah I know) on Win2003 platform and, over the past few weeks (not really sure when it started) have been seeing the following errors appear in the QMgrs\Errors\ file on 2 different machines.
-------------------------------------------------------------------------------
2/2/2007 13:56:16
AMQ6037: WebSphere MQ was unable to obtain enough storage.
EXPLANATION:
The product is unable to obtain enough storage. The product's error recording
routine may have been called.
ACTION:
Stop the product and restart it. If this does not resolve the problem see if a
problem has been recorded. If a problem has been recorded, use the standard
facilities supplied with your system to record the problem identifier, and to
save the generated output files. Contact your IBM support center. Do not
discard these files until the problem has been resolved.
----- amqrsrva.c : 437 --------------------------------------------------------
-------------------------------------------------------------------------------
2/2/2007 14:39:28
AMQ9516: File error occurred.
EXPLANATION:
The filesystem returned error code 6 for file 'E:\Program Files\IBM\WebSphere
MQ\QMGRS\SSWBIAP1\@ipcc\AMQRFCDA.DAT'.
ACTION:
Record the name of the file 'E:\Program Files\IBM\WebSphere
MQ\QMGRS\SSWBIAP1\@ipcc\AMQRFCDA.DAT' and tell the systems administrator, who
should ensure that file 'E:\Program Files\IBM\WebSphere
MQ\QMGRS\SSWBIAP1\@ipcc\AMQRFCDA.DAT' is correct and available.
----- amqrcdfa.c : 499 --------------------------------------------------------
2/2/2007 14:39:28
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'SSWBIAP1.CLIENT' ended abnormally.
ACTION:
Look at previous error messages for channel program 'SSWBIAP1.CLIENT' in the
error files to determine the cause of the failure.
----- amqrmrsa.c : 467 --------------------------------------------------------
After about 1/2 hour the client is able to re-establish his connection, drain the queue, and all is well again.
There are no FDC files created as a result of this. I have 55 Gb free space on my 64 Gb drive where the product is installed and where the logs are written. We use Circular logging with 2048 LogFilePages, 20 primary and 40 secondary - currently there are 30 allocated. There is 24 Gb memory on the box, with over 17 Gb free.
We have maxchannels set at 200 and maxactivechannels set at 2000.
I can't imagine what kind of "storage" Websphere was unable to obtain. And I don't see anything odd looking with the amqrfcda.dat file.
Nothing new has been introduced into the server in months (except application logic and microsoft security patches).
We do NOT see these errors on our Win2000 servers running MQ 5.3 with CSD06....
Does anyone have a clue as to what may be going on here?
_________________
Cheers,
Larry
MQ Certifiable |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Feb 02, 2007 12:56 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I'd say that you're running out of kernel or system resources to start another network session.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 05, 2007 12:49 am Post subject: Re: AMQRFCDA.DAT filesystem error AND AMQ6037 MQ Storage pro |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RatherBeGolfing wrote: |
| We're running MQ 5.3 with FixPack 7 ( yeah I know) on Win2003 platform |
I'm guessing from the world-weary "I know" moving your 5.2 to CSD 12 is not as easy as it looks from my perspective....? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RatherBeGolfing |
| Posted: Mon Feb 05, 2007 8:26 am Post subject: |
|
|
Centurion
Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA
|
Yes, any kind of software upgrade here is like pulling teeth. We've been keeping our fingers crossed that we'd be okay with V5.3 - CSD7, until we could get to v6. Thank goodness the EOS date was set for this year, that will usually buy us an upgrade ticket.
Knock on wood, this error hasn't seemed to affect any applications - at least none that are complaining to us.
_________________
Cheers,
Larry
MQ Certifiable |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Mon Feb 05, 2007 9:06 am Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
How long has the queue manager been running?
We had some strange issues with long running queue managers (months) that we resolved by scheduling a monthly restart of the queue manager.
I am not sure what CSD levels were involved but mostly our Windoze Qmgrs are at at least CSD7 with the majority at CSD11. I believe the issue was only with a CSD7 queue manager.
Of course this may be a total red herring but I thought I would mention it just in case. |
|
| Back to top |
|
|
| RatherBeGolfing |
| Posted: Mon Feb 05, 2007 9:32 am Post subject: |
|
|
Centurion
Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA
|
Thanks for the comments, but because we're running Windows (Win2003), we're applying security patches every month - and that almost always results in a reboot of the box. The QMgr therefore rarely runs continuously for more than about 4 weeks.
_________________
Cheers,
Larry
MQ Certifiable |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Feb 06, 2007 4:40 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| RatherBeGolfing wrote: |
| Thanks for the comments, but because we're running Windows (Win2003), we're applying security patches every month - and that almost always results in a reboot of the box. The QMgr therefore rarely runs continuously for more than about 4 weeks. |
Are quotas being set on the directories?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| RatherBeGolfing |
| Posted: Tue Feb 06, 2007 7:26 am Post subject: |
|
|
Centurion
Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA
|
Nope, we're not using dirquota on these particular servers.
_________________
Cheers,
Larry
MQ Certifiable |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Feb 06, 2007 8:28 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
One of our Win Admins suggests you run Perfmon to identify any physical and virtual memory issues.
Any errors in Event Viewer showing physical/virtual memory errors?
Trying to find a link to a MS white-paper that gives good information regarding Perfmon helping in identifying problems, I'll post it when I can find it again
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| RatherBeGolfing |
| Posted: Tue Feb 06, 2007 9:36 am Post subject: |
|
|
Centurion
Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA
|
Thanks for the posts, however this problem appears to be correlated to a bug within a WBIMB flow that is trying to insert a row to a SQL database, and failing. Our WBIMB developers are looking into the offending SQL statement. Based on time correlations, I'm pretty confident this is the root cause of my "storage" problem in MQ.
_________________
Cheers,
Larry
MQ Certifiable |
|
| Back to top |
|
|
|
|
