| Author |
Message
|
| obriencm |
 Posted: Sat Oct 02, 2004 3:47 am Post subject: Channels "INITIALIZING" on MQSeries Server v5.3 CS |
 |
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
I have just moved a prod system to new hardware (win2k cluster, running MQSeries v5.3 CSD07, with 5 Qmgrs) and I am now having issues where I can only start aprox half of my channels, the rest go into an "INITIALIZING" state. All the channels used to run on my old system (win2k cluster, MQSeries v5.2 CSD05, with 5Qmgrs). I have ensured both the MaxChannels and MaxActive Channels are set to the same level on both. Is ther some other setting I need to check? or in the newer version of MQ is there a limitation to the number of active channels you can have?
Any/all help would be greatly appreciated.
C. |
|
| Back to top |
|
 |
| csmith28 |
| Posted: Sat Oct 02, 2004 5:04 am Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
By Default MaxChannels and MaxActiveChannels is 200 unless specified in your qm.ini file.
Have you looked at our AMQERR0*.LOG's?
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| obriencm |
| Posted: Sat Oct 02, 2004 5:17 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
I have the max channels set to different values on each of the Qmgrs. 3 of them are set to the default, the other 2 are set to Max Channels: 1000 and Max Active Channels: 500. There is nothing showing up in the error logs except that the channel is starting.
C. |
|
| Back to top |
|
|
| obriencm |
| Posted: Sat Oct 02, 2004 5:37 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
A little further information, it appears here to us, that perhaps we have hit some limit (but we can't find where) because if we stop a running channel we can then start one of the initialising ones.....
C. |
|
| Back to top |
|
|
| csmith28 |
| Posted: Sat Oct 02, 2004 5:42 am Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
| obriencm wrote: |
A little further information, it appears here to us, that perhaps we have hit some limit (but we can't find where) because if we stop a running channel we can then start one of the initialising ones.....
C. |
Hmmm 
Something fishy there but I don't know what. MQ on Microsoft platforms is a little bit out of my eliment.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Oct 02, 2004 12:05 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
What is the total # of channels you want running on this one server between all 5 QMs?
What does task manager show for amqrmppa procceses? Is it possible you are hitting some O/S limit for max procceces/threads?
JasonE?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JasonE |
| Posted: Mon Oct 04, 2004 2:41 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
There's no o/s limit for threads/processes (well, good luck trying to hit one without running out of storage first!). However, there is a desktop heap issue, where the o/s launches a process (and returns a process id) but never starts it as it fails in the loading phase. However, whether you are hitting it or not is another issue - How many qmgrs on this machine? How are you starting the channels (specifically how do they show up in the task manager - threads of amqrmppa or seperate eg runmqchl processes).
A good way of 'proving' whether its desktop heap (and for security problems, but obviously not needed here) is to launch the qmgr from the command line using strmqm, and manually run the listener etc and see if you still hit the limit. If you do, its not desktop heap. If you dont, it probably is. |
|
| Back to top |
|
|
| obriencm |
| Posted: Wed Oct 06, 2004 12:31 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
Thanks for that, from our investigations here and from a PMR open with IBm, it does indeed look like that may be the issue, detailed below are our reg settings:
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16
We have 5 qmgrs running on the system. IBM have suggested we increase the last value above to 1024. However to do this will require a system reboot. |
|
| Back to top |
|
|
| JasonE |
| Posted: Wed Oct 06, 2004 12:57 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
You can easily prove it - the interactive user has a huge desktop heap compared to background desktops. So run the channel (or even the whole qmgr + listeners etc) from the cmd line and confirm they start fine. However, if you do this - dont sign off, or your channel / qmgr will be killed  |
|
| Back to top |
|
|
| obriencm |
| Posted: Wed Oct 06, 2004 1:09 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
We have carried out this test and it did indeed work. So we are now waiting for a suitable change window, when we can carry out the needed reboot of the systems for the change to take place. Would you suggest increasing to 1024, or a higher value such as 2048 so as to ensure going forward we do not have a reoccurance of this issue?
C. |
|
| Back to top |
|
|
| JasonE |
| Posted: Wed Oct 06, 2004 1:16 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
My experience with playing with that value is that its a little trial and error. 1024 is a value which has been used by people in the past successfully.
Are you running your channels as threads or processes - ie if you look in task mgr what are the majority of the amq* processes called. There is also the option of tuning the system, but increasing the desktop heap should resolve the situation as well. |
|
| Back to top |
|
|
| obriencm |
| Posted: Wed Oct 06, 2004 1:24 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
| Or channels seem to be running in threads of amqrmppa, given this do you still think increasing to 1024 should resolve our issue. Again, we have been able to start additional channels from the command prompt. |
|
| Back to top |
|
|
| JasonE |
| Posted: Wed Oct 06, 2004 1:39 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
Desktop heap will definitely help and should be your first thing to try.
Ways of tuning include changing the number of threads per amqrmppa process, per agent etc (eg go from 64 to 128 - see in supportpac mp7a) but a global and simple solution is desktop heap. The more threads you use, the less processes (ie less heap) BUT you then suffer from more thread contention (its a balance!).
This is why 5.3 changed to using the amqrmppa concepts - it scales much better. At 5.2 if you wanted 5000 channels, you had 5000 threads to the listener for example, and getting a process wide lock is 'expensive'. At 5.3, with max 64 threads by default, thread locks are cheap. There are very few cross process locks in this scenario, and so this change made it faster to handle more channels. |
|
| Back to top |
|
|
| obriencm |
| Posted: Wed Oct 06, 2004 1:42 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2002
Posts: 64
Location: Ireland
|
THanks a million for your help in this matter, I will update this thread, after we manage to bounce the system to make the registry changes (unless you know a way to force these changes on the fly  ) |
|
| Back to top |
|
|
| JasonE |
| Posted: Wed Oct 06, 2004 1:44 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
| Nope - its definitely reboot time to get that taken, as its read during system initialization. |
|
| Back to top |
|
|
|
|
