| Author |
Message
|
| sshaker |
 Posted: Mon May 19, 2003 11:47 am Post subject: large work items |
 |
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
hi
i was wondering if any of us encountered or experienced a scenario where large number of workitems (say.. 50k and up) in a worklist.. workitems joining the list frequently and users working on them at a similar frequency.. that means.. there is an inflow and outflow at a same rate and at any point the number of pending work items are going to be in the range of 50k upwards.. what are the performance implications in such scenarios.. if we add filter/sort criteria...what will be the implication.. if we add further complexity by using global container values to filter/sort..
any thoughts/views on this issues.. or experiences are welcome..
thnx in advance
regards
shaker
_________________
shaker |
|
| Back to top |
|
 |
| praveenchhangani |
| Posted: Mon May 19, 2003 12:28 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
Shaker,
We had encountered issues similar to the one you mentioned. What we noticed was that, anything above 50k (we had like 100 or so at one point in time) was that there was a performance degradation. The workflow webclient was functioning just fine, but it was very slow and users were complaining about response time. After we performed some load tests and made sure that the we were performing regular runstats on the db we then started to a slight improvement. Upon reducing the number of worklists to a much lower workitem number say (25), things were looking much better.
Having too many work items on a worklist is not a good idea performance wise, and when you come to think about it half the time users don't like to see more than 20-25 on their list anyways. Also it is not a good idea to have too many people assigned to one role, instead the notion of a group worklist (a virtual workflow user id) is much more recommended.
It is also recommed to have filters and also worklist thresholds for improved accessibility/webclient performance. This has to do with the amount of stuff that needs to be queried, the more the slower.
Another thing to think about while we are talking in this regard, is the worklist refresh feature. Unless absolutely required, it must turned off because this chews a large chunk from your performance bucket.
We are not making use of global containers at this point, however, my impression is that it is a good idea to make use of these as it helps performance wise.
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| sshaker |
| Posted: Mon May 19, 2003 12:50 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
thnx praveen
we are having a virtual user and the end user gets only one work item at a time. but, because of sort/filter criteria, and continuous updating of worklist.. when the end user requests next work item, it is going to resort/refilter the list .. that is my understanding.. (pl let me know if that is wrong).. in that case.. how the database performance is going to impact operations..
regards
shaker
_________________
shaker |
|
| Back to top |
|
|
| jmac |
| Posted: Mon May 19, 2003 1:43 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Sshaker:
Are you saying that you have 50K workitems on the Virtual Users worklist?
If so this is way too many.
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| sshaker |
| Posted: Mon May 19, 2003 2:10 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
yes John.. if not 50k.. it will be 20k.. !! How is it going to impact the performance? any thoughts?
regards
shaker
_________________
shaker |
|
| Back to top |
|
|
| jmac |
| Posted: Mon May 19, 2003 2:20 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Why are all 50K (20K) workitems on the Virtual users worklist? Or are you saying that there are more workitems than this in your system, this is just what is on the Virtual users worklist.
I hope you are running this on a BIG box.
I would try to come up with a way around having to issue a query that is going to yeild 50K workitems. You definitely need threshold. Have you ever looked at IBM's performance Guidelines. If not go get it and have a look.
http://www-3.ibm.com/software/integration/support/supportpacs/individual/wp01.html
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| sshaker |
| Posted: Mon May 19, 2003 3:02 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
yes john, we looked at the support pac ... !!
actually the threshold will be jsut 1, as all the users will need only one workitem at a time.. but all the workitems for the group of users are sorted (again..based on a complex logic).. if these users are requesting top most work item from the sorted list of say 20k workitems.. what will be the impact on peroformance..
regards
shaker
_________________
shaker |
|
| Back to top |
|
|
| jmac |
| Posted: Mon May 19, 2003 3:12 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Shaker:
Is there not some way that you can cut down this number of workitems. Perhaps you could have multiple layers of virtual users.
It is my opinion that this is going to be a strain on the workflow engine. Realize that this is just my opinion. You might want to post this question to the IBM newsgroup to see if one of the developers will answer your query.
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Mon May 19, 2003 6:55 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
John,
Are you saying that even 20 - 25 work items is overkill? My understanding is that even if you have somewhat of a large worklist(not too large) when you set filters, it helps performance wise. However, if you show all the workitems at once (no filter), then that would affect speed and access time.
From what you are saying, I am getting the idea that even 20-25 is quite a bit. Please let me know if I have interpreted this in a way other than what you were shooting for?
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| jmac |
| Posted: Tue May 20, 2003 4:55 am Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Praveen:
My understanding is 25K workitems not 25. There is no problem at all with 25 workitems
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Tue May 20, 2003 5:16 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
Oh, ok. I understand what you are saying John. Thanks for the clarification.
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| sshaker |
| Posted: Thu May 22, 2003 11:22 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
hi john
we are already talking to IBM.. but little hope till now.. as this is part of the problem !! will update on how we are going about.
in this regard, one basic question, what happens when a user tries to get an item from a persistent list.. will that call 'refresh' the database and get the 'current' suitable record .. or will that get a current record when the list was created..
i tried with a simple code.. where i created a persistent list.. displayed no of records.. and parallely.. i created some process instances.. and the count was increasing.. that means.. the call to query work items is always current.. does this mean.. everytime a query is sent to the persistent list.. the database contents .. workitems.. get refreshed.. before served?
if yes, what is going to be the impact on db performance.. some db gurus believe that the db will take care of optimizing.. i am not really sure.. anybody around who has some experience with such scenaios..
regards
shaker
_________________
shaker |
|
| Back to top |
|
|
| jmac |
| Posted: Thu May 22, 2003 11:33 am Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Shaker:
I believe that the query agains the Persistent list will go to the DB and get all the items that are currently there.
IF you are using a Virtual User technique, consider the following:
You have a workitem server that your users issue a request and it returns them the next workitem.
This server then does a query to get the workitems with a threshold of maybe 100 (depends on your environment, but I think you will get the idea).
Now each time a request for WI comes to the server it serves off the top one.
When there are no more items, it does the query again and gets the next 100.
The potential problem with this is that it could delay processing of workitems IF you are using a scheme other than FIFO, for instance if you were using Priority. In this case you would not always get the NEXT high priority item.
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| sshaker |
| Posted: Thu May 22, 2003 1:30 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 185
|
hi john
thnx ..
certainly we are not using FIFO..
| Quote: |
if we add filter/sort criteria...what will be the implication.. if we add further complexity by using global container values to filter/sort..
|
and the filter/sort criteria is very complex.. which does not depend on the keys we are talking about (FIFO, priority etc.) but on data container values..
and i would like to understand more about ur 'server' .. what do u mean by a server.. are u talking about a java application which will store 100 records or so.. or are u referring to something else..
can u pl elaborate on various components and their interactions in ur post viz. workflow database server, virtual user, wi server, user...
what happens when the first 100 are consumed and the server is querying again for next 100.. does it refresh the database again at that time?
if i get it right what ur telling.. i may end up an acceptable solution for my problem..
regards
shaker
_________________
shaker |
|
| Back to top |
|
|
| jmac |
| Posted: Thu May 22, 2003 1:40 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
Shaker:
| sshaker wrote: |
| and the filter/sort criteria is very complex.. which does not depend on the keys we are talking about (FIFO, priority etc.) but on data container values.. |
I certainly hope you are not retrieving container values in order to do a sort. This will have terrible performance, espeically if you are talking about 25K items
| Quote: |
| and i would like to understand more about ur 'server' .. what do u mean by a server.. are u talking about a java application which will store 100 records or so.. or are u referring to something else.. |
This is exactly what I mean... A little Java app that serves up Workitems.
When it initializes it gets say the first 100 workitems via queryWorkitems. It has a getWI() method that returns the next item in the array of workitems, and checks to see if its time to get more workitems. Of course you will need to take care that things are synchronized properly
GOOD LUCK
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
|
|
