| Author |
Message
|
| anveshita |
 Posted: Tue Nov 29, 2005 8:39 pm Post subject: Question on system performance |
 |
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
This is a weird observation. We have noticed that the Workflow system performance is going down whenever we instantiate Process instances into Workflow. This is happening immediately affter the instances are put into Workflow ( XML interface) and after sometime the performance seems to be okay. We are hardly putting 3000-4000 instances into Workflow at oneshot. I guess this should be piece of cake in terms of load.
Re-starting of the Workflow servers has been the shortcut. So as soon as we instantiate the process instances, we have scheduled a stop/start of the Workflow servers. I believe this is too rude a way. Let me know if something needs to be done to improve the performance that would elimainate restarting of the workflow servers.
Any suggestions?
Also I would like to know, in the real world how many instances one would instantiate into Workflow. Ours is a small application and I would like to know what would be the max volume in terms of your setups.
Thanks |
|
| Back to top |
|
 |
| vennela |
| Posted: Tue Nov 29, 2005 8:42 pm Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| Quote: |
| We are hardly putting 3000-4000 instances into Workflow at oneshot. I guess this should be piece of cake in terms of load. |
I think that is too much. I would definitely expect workflow to degrade in terms of performance for such heavy loads. |
|
| Back to top |
|
|
| jmac |
| Posted: Wed Nov 30, 2005 3:56 am Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
You should have a look at supportpac WP01. Starting an instance is THE most expensive thing in MQWF
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed Nov 30, 2005 9:06 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks for the responses vennela and JMAC,
Ours is a batch process and the instantiations are carried out all at once. If you think that 3000-4000 process instances are too much,based on your experience/observation can you tel me what would be the ideal load. I was thinking Workflow would be able to handle process instances in terms of millions ( in case of banks etc not sure where I heard). May be I am wrong!! |
|
| Back to top |
|
|
| fidelio |
| Posted: Wed Nov 30, 2005 9:48 am Post subject: |
|
|
Apprentice
Joined: 14 Sep 2005
Posts: 45
Location: AttainBPM
|
| Most cases where large volumes of concurrent processes are running, the process starts are staggered over time. Processes that start with a UPES are also more resource intensive since sending the message for the initial step will be included in the process start transaction. If it fits your business model, it is better to pace the process starts over time so that the servers have some CPU cycles to process other actions. |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed Nov 30, 2005 7:11 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks fidelio
So if I understand correctly:
1. 3000-4000 would cause severe downgrade of performance.
2. It is better to split the load if possible.
Here are my questions:
1. Why the perfformance improved when we re-start the servers?
2. How in the real world people use the Workflow? No body use batch programs for instantiation? If so how are they managing. I am not having exposere to any other production implementation besides ours.Hence I would like to know the best practice for this. |
|
| Back to top |
|
|
| jmac |
| Posted: Wed Nov 30, 2005 7:34 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Read and understand supportpac WP01
| anveshita wrote: |
1. 3000-4000 would cause severe downgrade of performance.
|
This would be 11400 to 15200 BWU just for these process starts. Look at the processor you plan on running on and see if it can deal with such a load. Find the processor that can deal with such a load and convince your mgmt to buy it.
| anveshita wrote: |
2. It is better to split the load if possible. |
If it can be spread out it might be better. If the instances can all be started off peak it would be best.
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| hos |
| Posted: Thu Dec 01, 2005 12:09 am Post subject: |
|
|
Chevalier
Joined: 03 Feb 2002
Posts: 470
|
Hi,
| Quote: |
| 1. Why the perfformance improved when we re-start the servers? |
There is no valid answer for this. If you really see a performance improvement by re-starting execution servers I could only think of a DB deadlock/timeout situation that gets resolved in this scenario. Are you sure that you do not just increase the number of running execution servers? Check your system log and error log for any problems during process instance creation. How do you measure performance improvement? Is it the time until a user can 'work' with a newly created process instance? Or is it the time until your batch job returns?
| Quote: |
| 2. How in the real world people use the Workflow? No body use batch programs for instantiation? |
Customers DO use the XML interface to create process instances but they rarely use a bulk creation. Those who use bulk creation have to analyze their process model in depth (see WP01) and tune their system (MQ, DB, MQWF) to avoid the trouble that you face. Be aware that tuning of a high performance MQWF system is not a piece of cake and reqires a deep understanding of the underlying concepts ==> read the documentation! |
|
| Back to top |
|
|
| anveshita |
| Posted: Thu Dec 08, 2005 7:47 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
First of all my apologies and I did not check the forum for a while.
Hos,
| Quote: |
| There is no valid answer for this. If you really see a performance improvement by re-starting execution servers I could only think of a DB deadlock/timeout situation that gets resolved in this scenario. Are you sure that you do not just increase the number of running execution servers? Check your system log and error log for any problems during process instance creation. How do you measure performance improvement? Is it the time until a user can 'work' with a newly created process instance? Or is it the time until your batch job returns? |
This is very puzzling. I have instantiated 3000 process instnaces. The instaintaion program successfully put the messages into the Workfflow queue. There were two execution servers running at the time. The work instances got created with in few minutes. This I have verified by checking the out-of box client. Some of the process instances would create two work items at the first activity and some create one work item at the first activity. I have set my self as one of the users who would process the work item through the custom web client we have got. When the work item is pulled from the client the time taken for checking out and cheking in of the work item was taking anywhere from 6 to 10 sec. At this point I have verified that there were no errors on the logs. Having confirmed that there were no errors, I have asked the system admin to restart the workflow servers. Then I have tested again the process has become pretty fast say 1-2 sec. I have specifically asked the DBAs to check for deadlocks and much to my surprise no deadlocks were found.
So I am not sure why the system performance is getting improced upon restarting of the servers.Any thing I am missing here?
| Quote: |
Customers DO use the XML interface to create process instances but they rarely use a bulk creation. Those who use bulk creation have to analyze their process model in depth (see WP01) and tune their system (MQ, DB, MQWF) to avoid the trouble that you face. Be aware that tuning of a high performance MQWF system is not a piece of cake and reqires a deep understanding of the underlying concepts ==> read the documentation!
|
So very few people use bulk creation and tuning is a must for the bulk creation.If you/any one in the forum know/s of any users who use bulk creation, please let me know what is the volume of process instances which users use.
Given that we have no tuning in place, I would like to change our bulk instantiation to say "metered" instantiation. For example, the program would put instatiation XML and sleeps for "X" sec. Then reads the next process instantion XML and and sleeps for "X" sec.If this would be the right way for us instead of bulk instantiaion, I would like to know what could be the right value for "X". Please share your ideas. |
|
| Back to top |
|
|
| hos |
| Posted: Fri Dec 09, 2005 4:39 am Post subject: |
|
|
Chevalier
Joined: 03 Feb 2002
Posts: 470
|
Hi,
I would suggest a different, more sophisticated approach:
split your work in bulk processing via XML in one workflow system and client application work in a different system. This requires a multi system setup. The advantage is that you can have one pool of execution servers that exclusively handle your batch requests and another pool that handles your interactive (client) requests. So the 'interactive' execution servers are not blocked by cascading bulk XML requests. Of course this requires that your clients explicitely connect to the 'interactive' MQWF system.
The 'meter' would be the number of execution servers that you are running in each pool. |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Dec 13, 2005 7:25 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| Quote: |
Hi,
I would suggest a different, more sophisticated approach:
split your work in bulk processing via XML in one workflow system and client application work in a different system. This requires a multi system setup. The advantage is that you can have one pool of execution servers that exclusively handle your batch requests and another pool that handles your interactive (client) requests. So the 'interactive' execution servers are not blocked by cascading bulk XML requests. Of course this requires that your clients explicitely connect to the 'interactive' MQWF system.
The 'meter' would be the number of execution servers that you are running in each pool. |
I think I mislead you. We have the batch instantion that puts the XML messages into the Workflow. The model ( Process) instace created by the XML instantiaon message has both automatic and manual activities. I am not sure how multi system setup could help us in this case. Probably I might have mis-understood.
I have asked
| Quote: |
Given that we have no tuning in place, I would like to change our bulk instantiation to say "metered" instantiation. For example, the program would put instatiation XML and sleeps for "X" sec. Then reads the next process instantion XML and and sleeps for "X" sec.If this would be the right way for us instead of bulk instantiaion, I would like to know what could be the right value for "X". Please share your ideas. |
does the above solution help us? |
|
| Back to top |
|
|
| hos |
| Posted: Wed Dec 14, 2005 12:15 am Post subject: |
|
|
Chevalier
Joined: 03 Feb 2002
Posts: 470
|
Hi,
my answer was based on the assumption that your problem is the system responsivness to other tasks while processing the bulk XML requests. The second system would allow users to process interactive work during this time period. If this is not your problem, you should be more precise when describing the problem (maybe you do not have a problem at all ?) |
|
| Back to top |
|
|
| anveshita |
| Posted: Sun Dec 18, 2005 8:47 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks Hos, sorry if my earlier posting was confusing.
Here is what happens in our system. In our Workflow system, instantiation of process instances is done through batch. That means we at regular intervals ( 6-8 times a day) get 100 to 5000 process instances created (as an XML) into workflow. Each process instance comprises of manual as well as automatic activities. Now about the probelm we are facing. In the first run let us say we have create 100 process instances and users are working on the process instances. Now while the users are working on the process instaces created in the first run, let us say our batch job created 5000 process instances. At this point users who are working on the work items created by the first job are expereincing very poor performance by the Workflow system and taking considerable time to check-in/check-out work items. I have observed that, if I stop and start Workflow servers imediately after instantiating 5000 process instances( 2nd batch), the performance seems to be okay.
Now the questions I have are:
1. Why the system performance is improving by stopping/starting of the Workflow servers? Please remember the system performance is getting seriously affected immediatly after the instantiation of the process instances and I understand from the replies I got in this post that the creation of process instances is the hardest task for the servers. But how is stopping and starting helping to improve the performance?
2. Has anyone had the similar setup of 6-8 batch instantiation cycles and experienced similar situation?
3.Does starting additional execution server immediately after instantiating huge number of process instances (5000-6000) help?
4. Does having an intermediate program that would collect all batch feeds into single file and create the process instances at a regular intervals help?
5.I was thinking Workflow to be a robust product which can handle millions of transactions. Given one does the system tuneup, in the real world what is the max. throughput( process instances/per day) one could achieve so far? |
|
| Back to top |
|
|
| hos |
| Posted: Sun Dec 18, 2005 11:40 pm Post subject: |
|
|
Chevalier
Joined: 03 Feb 2002
Posts: 470
|
anveshita,
all your questions have already been answered in this thread!
1. There is no valid answer for this. Depends on how you define 'performance'
2. Yes this is a common pattern and yes, the performance/priority problems are known. My proposal exactly targets for a solution of this problem.
3. Starting additional execution servers before the bulk load may help. Use WP01 to decide the proper amount of Execution Servers.
4. Sure. If you can distribute the bulk job over a long time period this would help.
5. It is a robust product. You don't see any outages or data loss - do you? WP01 tells you the throughput you can expect for your machine, given that the components are properly tuned.
That's all that can be said to your post. Good luck! |
|
| Back to top |
|
|
|
|
