| Author |
Message
|
| CHF |
 Posted: Fri Jun 25, 2004 10:49 am Post subject: [SOLVED]More Instances of Execution Server |
 |
|
Master
Joined: 16 Dec 2003
Posts: 297
|
On Distrubed platforms, If you have more load on Executon server, Does Admin Server brings up more instances of Execution Server?
Please Share your ideas.
Thanks
CHF  |
|
| Back to top |
|
 |
| vennela |
| Posted: Fri Jun 25, 2004 11:22 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| Quote: |
If you have more load on Executon server, Does Admin Server brings up more instances of Execution Server?
|
NO
But if you have enought hardware resources then you can set so that workflow can have more than one exec servers |
|
| Back to top |
|
|
| jmac |
| Posted: Fri Jun 25, 2004 11:23 am Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
CHF:
NO, only on Zos with Workload manager is this dynamic. The general rule of thumb is to set your Execution Server number of Initial Instances equal to 2* the number of processors. So for example: if you have an 8 CPU system you would want 16 ES.
Hope this helps
Venny beats me again 
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| CHF |
| Posted: Fri Jun 25, 2004 12:10 pm Post subject: |
|
|
Master
Joined: 16 Dec 2003
Posts: 297
|
I totally agree with you guys.
I am familiar with WLM and Workflow on OS/390, and with WLM you will have the option to specify the maximum number of address spaces to be started, and also the maximum number Execution Server instances per one address space can also be specified. With this, depending on the load on Workflow, WLM starts up additional address spaces eventually Execution Server instances.
 I saw the similar kind of behavior in Windows too. I know this is not possible in Windows. Give me your ideas on the following scenario:
My environment:
I have a stand alone installation and setup on
Windows XP professional
WF 3.4 SP5
WMQ 5.3
DB2 8.1
Number of initial execution server instances running = 2
I did the follwing test on my machine:
I put 3000 ProcessTemplateCreateAndStartInstance messages onto EXEXMLINPUTQ using a Java program. I was monitoring the number of Execution Server Instances using FMCAUTIL. After sometime, the initial 2 instances were stopped (because of Internal Error), and again immediately Admin Server brought back the 2 instances. This happened repeatedly for sometime. After that I am surprised to see that there are 5 instances started and running. I did not do anything after that and waited till all the messages are processed, and again still 5 instances were running.
Do you guys have any idea on Why Workflow behaved like this on my Windows setup?
I will do the similar test again and let you know in the mean time.
Thanks
CHF |
|
| Back to top |
|
|
| jmac |
| Posted: Fri Jun 25, 2004 12:42 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
| CHF wrote: |
| I was monitoring the number of Execution Server Instances using FMCAUTIL. After sometime, the initial 2 instances were stopped (because of Internal Error), and again immediately Admin Server brought back the 2 instances. This happened repeatedly for sometime. After that I am surprised to see that there are 5 instances started and running. I did not do anything after that and waited till all the messages are processed, and again still 5 instances were running. |
I have never seen this type of behaviour, and it does surprise me. I do not believe I have seen any documentation that indicates that this type of "load adjustment" is done in any environment outside of zOS. Please post the results of your follow up testing.
Thanks
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| manoj |
| Posted: Fri Jun 25, 2004 4:01 pm Post subject: |
|
|
Master
Joined: 30 Jan 2002
Posts: 237
Location: Virgina
|
CHF,
I have seen this scenario sometimes...what i believe is when the execution server started "disappearing" and when ES instances come back (admin server automatically starts the execution server) the exact count (say 2) is sometimes is not maintained.. This may be because the ES instances are repeatdly crashed ...
But i have never seen this scenario when the system is stable....
_________________
-manoj |
|
| Back to top |
|
|
| CHF |
| Posted: Mon Jun 28, 2004 5:52 am Post subject: |
|
|
Master
Joined: 16 Dec 2003
Posts: 297
|
I think you are correct manoj! Anyways I will repeat the test today and will post the results.
Thanks
CHF |
|
| Back to top |
|
|
| CHF |
| Posted: Tue Jun 29, 2004 5:12 am Post subject: |
|
|
Master
Joined: 16 Dec 2003
Posts: 297
|
I repeated the test 2 times with 2 different scenarios (see below).
1) In the first scenario, it did NOT start any new ES instances, and there were only 2 instances at the end of the test.
2) In the second scenario, it did start new ES instances, and there were 7 instances at the end of the test.
Environment:
Windows XP Professional
WMQ 5.3
WMQ WF 3.4 (SP5)
DB2 - 8.1
Here's the model info:
I have total 5 activities in my model, A, B, C, D, E.
The first activity A is a No-Op activity, and it branches to B and C.
B-->D, and C-->E.
B and C are synchronous UPES activities.
Test 1:
Number of initial Execution Server Instances = 2.
First I defined the Queue depth for the UPES Q of activity B to be 5000.
Next I put 3000 messages (Create and Start Instance) onto EXEXMLINPUTQ.
Nothing odd happened in this case, and WF started all 3000 instances, and all 3000 instances are in running, and 3000 messages (Activity Impl Invoke) were sent to UPES Q of B.
At the end of the test there were only 2 instances.
Test 2:
Number of initial Execution Server Instances = 2.
First I defined the Queue depth for the UPES Q of activity B to be 1000.
Next I put 3000 messages (Create and Start Instance) onto EXEXMLINPUTQ.
The following happened in this test:
1) First 1000 instances started smoothly, 1000 messages were sent to UPES Q of B.
2) Next the initial 2 ES instances are stopping and starting continuosly, and sometimes the messages are going to the HOLD Q. This happened for about 5 minutes.
3) Surprisingly after 5 minutes , it started 3 instances and died immediately, this happened like a cycle for 5 more minutes. And after that it's actually starting and stopping 7 instances.
4) Next I increased the UPES Q depth to 5000 again. Immediately after increasing the depth, it again run smoothly with 7 instances constantly working.
On the whole it started around 2700 Process instances, and the remaining 300 messages are placed in HOLD Q. At the end of this, there were 7 instances running.
Any ideas?
Thanks
CHF |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Wed Jun 30, 2004 7:40 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
Hmm, CHF:
I personally have NOT seen this kind of behavior before. However, I am curious to know if you have the "Workflow Runtime Threshold" value activated on the system? Also are there any indications in the error log files perhaps pointing to reasons why EXEC Servers might be "disappearing" in Manoj's terms. ? Is it that there are 5 execution servers always even if you have 2 as the INIT number?
At first glance it would seem like a bug, or perhaps even the possibility of a correlation with the WRT.
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| vennela |
| Posted: Wed Jun 30, 2004 7:56 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| praveenchhangani wrote: |
| Also are there any indications in the error log files perhaps pointing to reasons why EXEC Servers might be "disappearing" in Manoj's terms. ? |
That from the above scenario is because the UPES Q is full and he would get a Q full error. And I remember Volker Hoss mentioning that it would cause the exec server to abend. |
|
| Back to top |
|
|
| CHF |
| Posted: Wed Jun 30, 2004 9:16 am Post subject: |
|
|
Master
Joined: 16 Dec 2003
Posts: 297
|
There is no entry in the error (fmcerr) log. In the fmcsys log, the only entries are related to the Execution server stop and starts because of the Inernal error.
_________________
CHF |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Wed Jun 30, 2004 12:06 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
Vennela:
I'm not sure what Volker what was referring to here(You and Volker are very knowledgeable so I am following your thoughts on the Q full aspect as that makes sense from the application's standpoint, but surely if one were to have a backout queue installed, this should not be a problem right? MQSeries is very good about this. Infact if your backout queue itself were to fill up, you could setup another backout queue for your backout queue. Beyond that point, I think it's safe to say you are pretty much done for. Your thoughts?
CHF: Congrats and the test there buddy.
Thanks,
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| hos |
| Posted: Thu Jul 01, 2004 1:13 am Post subject: |
|
|
Chevalier
Joined: 03 Feb 2002
Posts: 470
|
Hi,
the messages on the hold queue and the log entries in the system log are
indeed related to the queue full problem of the UPES queue.
Once again: the server considers a put() failure as a fatal error and terminates (note: not an abend). The administration server starts a new server instance in order to keep the number of running execution servers as specified. This can lead to a kind of 'JoJo' playing (which in fact
certainly does not increase the server's processing capacity). Nevertheless this is a basic design constraint to ensure data integrity and avoid any loss of messages.
Now back to the original issue of this discussion:
While playing JoJo with the servers some terminating execution servers get ouf of the control of the administration server without actually terminating. They get into an intermediate state where they even cannot be deleted (even with e.g. taskmanager on WIN2K). If you use the admin utility you can see that the admin server is not aware of these additional execution servers so they do not count for the number of running instances.
You can see this behaviour since the latest ServicePacks of MQ. Seems that the execution server wants to terminate but MQ does not allow this(?)
This problem is related to server termination due to fatal errors and has nothing to do with workload management. |
|
| Back to top |
|
|
| CHF |
| Posted: Thu Jul 01, 2004 4:27 am Post subject: |
|
|
Master
Joined: 16 Dec 2003
Posts: 297
|
All the info here is very knowledgable and helpful. Thanks to all of you guys.
_________________
CHF |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Thu Jul 01, 2004 9:34 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
| hos wrote: |
Hi,
the messages on the hold queue and the log entries in the system log are
indeed related to the queue full problem of the UPES queue.
Once again: the server considers a put() failure as a fatal error and terminates (note: not an abend). The administration server starts a new server instance in order to keep the number of running execution servers as specified. This can lead to a kind of 'JoJo' playing (which in fact
certainly does not increase the server's processing capacity). Nevertheless this is a basic design constraint to ensure data integrity and avoid any loss of messages.
Now back to the original issue of this discussion:
While playing JoJo with the servers some terminating execution servers get ouf of the control of the administration server without actually terminating. They get into an intermediate state where they even cannot be deleted (even with e.g. taskmanager on WIN2K). If you use the admin utility you can see that the admin server is not aware of these additional execution servers so they do not count for the number of running instances.
You can see this behaviour since the latest ServicePacks of MQ. Seems that the execution server wants to terminate but MQ does not allow this(?)
This problem is related to server termination due to fatal errors and has nothing to do with workload management. |
Great. Thanks for the clarification there Volker.
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
|
|
