| Author |
Message
|
| srvm |
 Posted: Fri Jun 17, 2005 6:59 am Post subject: 2102 for Clients in bind mode |
 |
|
Apprentice
Joined: 18 Aug 2004
Posts: 43
|
We have a strange problem with our Queue Manager. We are running three QMgr's on the same box and one of the queue manager (linear logging) run out of disk space and after stopping it (some processes had to be killed). We started the qmgr ( cleaned ipcs by shutting down all the qmgrs bfr restart).
When the qmgr came up first it gave false return code to application running in bind mode 2059. Application in the client mode could connect and amqsbcg also connected in bind mode. Other QmGr's are functioning properly.
A restart of queue manager and start of bind mode apps started giving 2012 (Env Error).
The Kernel parameter are all per the recommendation of QB Guide. FD=1024 as soft and hard limit. I also did plimit and things seems to look right for all the qmgrs ( all have the same info for p limit 8MB of stack size).
When the bind mode app starts there aren't any errors logs or FDC's generated.
Env: Solaris 2.8 MQ 5.3 CSD07 (Using OAM Service for auth)
Any pointer/help would be higly appreciated.
Thank you in advance.
R
Correction: Error Code is 2102 : MQRC_RESOURCE_PROBLEM |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Jun 17, 2005 10:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
How much free file space do you have, for the qmgr and for the logs ?
What is the %tage of the free space vs total available space for the qmgr.?
Thanks
 |
|
| Back to top |
|
|
| srvm |
| Posted: Fri Jun 17, 2005 11:54 am Post subject: |
|
|
Apprentice
Joined: 18 Aug 2004
Posts: 43
|
We have log and data resides on SRDF devices.
Total (kb) Used Free
17408000 4320593 12270060 27% /data/part1/mqm
Free space varies anywhere from 27% to 53%. I have three qmgrs residing under the same directory (qmgrs/qm1, qmgrs/qm2, qmgr/qm3).
The log/active logs also resides under the same partition (log/qm1, log/qm2, log/qm3). Currently, I run linearlog cleanup every 1 hour. I do not see % go below 23%. When clean up logs is done I can see allmost 67-73% available.
The /var/mqm has 45-50% available.
There was one incidence of Sunday night when rcdmqimg was hung and we run out of space. Since then we are not able to get this QMgr to run properly. Other QmGr's are functioning properly. No FDC's are created when we get 2102.
R |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jun 17, 2005 6:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
It might still have to do with space.
Usually MQ requires more than 20% of free space.
Now you will need all the linear logs to start the qmgr correctly and possibly as well those for media recovery + 20%. This may be quite an order... Then check your qmgr and try and restart it.
To avoid the big out of space I dimension the file system to over 63 files of max size and set the qmgr to 2 primaries and 50 secondaries... I.E. if need be I can still up the secondaries to get more space and run rcdmqimg without running out of physical space. Then I should be able to archive old logs and thus free space. We run the archiving process every night.
Let us know how you fared.  |
|
| Back to top |
|
|
| srvm |
| Posted: Sat Jun 18, 2005 4:43 am Post subject: |
|
|
Apprentice
Joined: 18 Aug 2004
Posts: 43
|
Hi fjp,
Thank you for the reply.
We have primary and secondary setup otherway around. 60 Primary + 2 Secondary.
As far as recovery is concerned we were able to start the queue manager. However, bind mode app can not connect. Are you suggesting we have not completely recovered from the out-of-disk event yet? But, I do not see any FDC or error messages indcating that QmGr could not recover.
Now, I do not have logsas we run rcdimg and cleanmqlogs manytimes since then. In prod we run rcdimg once a day, but in test env (due to healy load testing etc) we tend to run it frequently to keep disk free space.
I 'll try reversing the log # as u suggested and report u the result.
R |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 18, 2005 3:26 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Note that it is prudent to not set the number of log files to the max.
This way if there is a problem the queue manager stops working before the max is reached. This allows you to expand the number of logfiles restart and run rcdmqimg to offload old logs before you're stuck in an irreversible manner. Then stop the qmgr and set down the total number of logs again.
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Jun 18, 2005 4:40 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If you stop the QM, all the transaction are ended, and when you start up, you have a fresh start, whether you had the logs already at the max or not to begin with, right?
So why not just make the logs as big as possible to begin with?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 19, 2005 5:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well if you make the logs and push them to the max what happens when they are all full and:
log needed to start qmgr = S00000.LOG
log needed for media recovery = S00000.LOG
How are you going to be able to run rcdmqimg that is going to write to the logs ?
This is why it is prudent to set your max #of logs to 52.
So when you stop the qmgr and restart it with a max logs of 62 you can try and free old logs...
When done reset the lower limit ....
One more thing... when running with multiple qmgrs you can run in a reduced file space if not all your logs are primaries.
NOT RECOMMENDED for production as if all your qmgrs need the max you will run into space issues...
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Jun 19, 2005 10:07 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
So I was only thinking in circular tersm, where these media images are not a factor.
I admit I am no expert in this area, so this is not a rhetorical question.
If you gave the Linear Logged QM 52 logs, and they filled up, you would not be able to run the record media command right? If you wanted to up the logs to 63, you would have to stop the QM, and when you restarted it, all transactions would be freed up, meaning checkpoints would have anvanced, meaning the QM would have plenty of active log space after the restart.
This would be the same if you bounced the QM with 52 logs, or 63 logs.
I still don't understand the benefit of artificially limiting your logs.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 19, 2005 7:20 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Your hypothesis is that you would have enough space to write the log using rcdmqimg...(with minimal log size running rcdmqimg can create 2 files....for an empty qmgr(no messages)..)
Now imagine some big transaction at the beginning long running etc...
At the end only small transactions so the place freed by shutting down the qmgr is peanuts...
Running rcdmqimg you would need to write at least one additional log file before being able to free one or more.....
Honestly I've never hit the wall but I like to be prepared...especially in production... that's why we run rcdmqimg and archive old logs every day
Thanks for thinking with me... |
|
| Back to top |
|
|
|
|
