| Author |
Message
|
| SANTYP |
 Posted: Sun Aug 07, 2011 10:26 pm Post subject: |
 |
|
Centurion
Joined: 27 Mar 2007
Posts: 142
|
Hi ..
I have question,
I do have a flow.. which will extract the data from one database table and generates a csv file...
I made that flow to run multithreaded.. i given 3 instances.. which means it can actullay process.. 4 msgs at a time..
but in some cases when one of the thread not responded.. its not processing the other messages as well..
explaing about the actual scenario.. I have a job scheduler which will acutomatically post the mesages on to the queue.. which are processing by the flow (Mq input -- java compute-- mqoutput) .. but some times one of the thread not responded due to some reason.. the flow not picking up the other messages even though it is multi threaded..
here what I am preassuming was if not thread is haulted by some reason... why other threads of the flow are not active(or not processing ) the messages..?
plz help me to understand..
Thanks in advance..
Regards,
Santy |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Aug 08, 2011 4:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| SANTYP wrote: |
| plz help me to understand.. |
Help me to understand. You describe a flow with MQInput at one end and MQOutput at the other. But start off talking about databases and csv files? Is all that in the JCN?
In the absense of other details, I would theorise that as you say:
| SANTYP wrote: |
| some times one of the thread not responded due to some reason |
the reason is a database delay or deadlock, or a file contention, and the other threads are waiting for the same reason.
But that's just a theory.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Aug 08, 2011 4:33 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
To add from what Vitor has said,
Simply adding extra instances to a flow is NO GUARANTEE that you will get even 1% more throughput.
If you are using a DB then you have to start to understand how the DB (and they all differ to some extent) handles row and page locks.
Then you may need to change the SQL (not ESQL) to limit the locking with a read on a table. In some DB's it is easy to do a read with no locks at all. Others, it is not so easy.
Then you may want to specifically limit the number of rows returned in a select.
finally if all that fails you may need to enable ODBC trace to see if there is for example a 'detaildeadlock' (from DB2) reported that the ODBC driver hides from the Broker user.
This all takes time, lots of time, patience and importantly expertise. Deep level broker expertise is often required to get the most out of a particular flow/flows.
btw,
Vitor's rate is now $550/hour plus expenses. 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Aug 08, 2011 4:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| Vitor's rate is now $550/hour plus expenses. |
Payable weekly in advance because I've got an honest face & you can trust me. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bloomy |
| Posted: Tue Aug 09, 2011 6:30 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2009
Posts: 61
Location: Bloomington, IL USA
|
| Quote: |
I made that flow to run multithreaded.. i given 3 instances.. which means it can actullay process.. 4 msgs at a time..
|
Vitor or Smdavies can correct me if I am wrong here but as per my experience and from my observation. Adding 4 additional instances will not necessarily process 4 messages all the time. if single thread can process all the four messages the message flow will not spawn the other 3 instances unless otherwise needed. I observed this from the flowstats reports I have generated where we were running a perf test. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 09, 2011 6:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bloomy wrote: |
| Quote: |
I made that flow to run multithreaded.. i given 3 instances.. which means it can actullay process.. 4 msgs at a time..
|
Vitor or Smdavies can correct me if I am wrong here but as per my experience and from my observation. Adding 4 additional instances will not necessarily process 4 messages all the time. if single thread can process all the four messages the message flow will not spawn the other 3 instances unless otherwise needed. I observed this from the flowstats reports I have generated where we were running a perf test. |
I think the OP was indicating that he's provided the capacity for 4 simultanious messages. But yes, setting additional instances does not mean all those threads will always be running.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| SANTYP |
| Posted: Tue Aug 09, 2011 6:00 pm Post subject: |
|
|
Centurion
Joined: 27 Mar 2007
Posts: 142
|
| Vitor wrote: |
Help me to understand. You describe a flow with MQInput at one end and MQOutput at the other. But start off talking about databases and csv files? Is all that in the JCN?
In the absense of other details, I would theorise that as you say:
|
This flow will be used for multiple functionalities, 1) it will extarcts data from data base and generates files, 2) inserts into another database by calling stored procedure, 3) converts database exrtacted message into xml and puts into a Queue.
| Quote: |
the reason is a database delay or deadlock, or a file contention, and the other threads are waiting for the same reason.
But that's just a theory. |
I verfied at the database level at the same time there were no deadlocks,
and I was confused at a point where the flow running on 4 instances, if one instance waiting for data to process why another instace carry with the next message.. ? is that how multithreading works in MB..? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 09, 2011 7:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| SANTYP wrote: |
This flow will be used for multiple functionalities, it will - extract data from data base and generates files,
- inserts into another database by calling stored procedure,
- converts database extracted message into XML and puts into a Queue
.
| Quote: |
the reason is a database delay or deadlock, or a file contention, and the other threads are waiting for the same reason.
But that's just a theory. |
I verified at the database level at the same time there were no deadlocks,
and I was confused at a point where the flow running on 4 instances, if one instance waiting for data to process why another instance carry with the next message.. ? is that how multi-threading works in MB..? |
Yes and no. There is no need for the database to show a dead lock.
It could just be a bottle neck that mandates a synchronized access to some DB records. As the result the 2nd thread will have to wait until the previous one (first) is done (and committed?) and so on...
So no dead lock, just a bottle neck.
However there is another matter here that makes me wonder... Usually interactions with the file system are frowned upon because they slow down considerably compared to other means of storage ...
Can you be more specific as to the files being generated and the data retrieved? For instance if the node just does data retrieval you could remove it from the transaction to allow the read to be committed on the DB. This might free up the same rows of the DB for a read from the next instance.
There is also such a thing as passthrough with uncommitted read at DB level.... and finally if the relevant rows being read are fairly static, they could be added to the db cache to speed things up...
Hope this helps some.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 09, 2011 8:58 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
To add to what FJB said,
Plese re-read what I said earlier about the scope of the locks you have on DB Reads and look at their duration.
Does this simple DB read lock data that the other threads want? How long are you implicitly keeping the lock in place? Could it be until that thread completes?
As I said, this is not elementary stuff and certainly not for the faint hearted. It will take TIME. Do not rush. Certainly do not rust to conclutions. Understand the differences between Burst and steady state processing. Don't Panic. Be logical.
{All the stuff that does not get taught in Universities these days (sigh)}
Ok?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Aug 10, 2011 4:45 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Seems like you have alot going on with these flows. I would consider being more granular in my design. Break these flows up into much smaller flows, that do one step each flow rather than your current design which seems to tackle many steps at once.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
