| Author |
Message
|
| udayakumarisampalli |
 Posted: Wed Dec 13, 2006 4:55 am Post subject: Message Flow not able to process all messages in Queue |
 |
|
Novice
Joined: 20 Jul 2006
Posts: 19
Location: India
|
i have implemented the Message flow using the TimeoutNotification node as follows
TimeoutNotification node connected with MQGetNode and then MQGet-->Compute node--->MQoutput node and this MQoutnode connected with the previous MQGetnode.
my idea is that this TimeoutNotification node triggers the flow at every 10 so MQ Get node pick up the message from Queue 'AA' and process it. as MQGet will take one message at a time, but i need all the messages should be processed once the Timer node triggers after a given intervel so i have connected MQoutput node to MQGet node so that one message is process control again come to MQ Get and so on, so loop repeats till end of all messages in the Queue.
currently my problem is that, what i have observed is that after 40 messages it is reading from the Queue and sending it to output queue it encoutring some problem and not processing the rest of messages.
as this complete loop is undersingle transaction all message are again back to Input Queue.
if i make transction property to "NO" in both MQGet and MQOutput node i am missing some messages say if i send 60 messages i am getting 58 only to output. i am facing this issue if number of messages exceed 40. if it is less than that it is processing fine. ( no problem with message becsue i am sending all similar message for testing)
i think it is becasue all this loop is running in single thread for all messages it can not able to process more than 40. i am not sure.
please tell me what is the problem i need to test for 1000 messages. means my flow needs to process 1000 message every time after timer node trigers after given intervel.
please provide me with some kind of inputs |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Dec 13, 2006 4:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Why are you using MQGet rather than MQInput?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 13, 2006 5:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Or is this the outcome of this bizarre timed processing you were posting about before?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Dec 13, 2006 5:09 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
What, exactly, is the error you get when you have more than 40 messages?
| Quote: |
| what i have observed is that after 40 messages it is reading from the Queue and sending it to output queue it encoutring some problem and not processing the rest of messages. |
?
Yes, if you do these with Transaction property to AUTOMATIC or YES, then you could run into issues with processing a large number of messages or a large message itself. This is because the transaction takes up space in the MQ Log.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| udayakumarisampalli |
| Posted: Wed Dec 13, 2006 9:15 pm Post subject: |
|
|
Novice
Joined: 20 Jul 2006
Posts: 19
Location: India
|
hi,
this is extension to what i have posted previously for timly processing of messages. i can not use MQInput node as it will not have the input terminal.
Jefflowery,
i did not getting any error but after the 40 messages the flow stops processing the messages. as this complete processing is in single transction the message which were put in the Ouput Queue were also back to the Source Queue.
i thought like this complete loop is running on single thread due to increase in the size it is stoping at this number.
please help me processing all the message in the queue without failure.
can some one tell how to clrear this MQLog at run time for smooth processing of the message by the flow |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 14, 2006 12:14 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| udayakumarisampalli wrote: |
| can some one tell how to clrear this MQLog at run time for smooth processing of the message by the flow |
The log is the queue manager log, and it's automatically cleared down. I'm not aware of a tool to clear it, and can't see how such a tool would be safe to use.
IMHO you have 2 options:
1) Ensure that the queue manager is not processing any other messages when your flow is running (i.e. there are no transactions in flight & hence the log is empty)
2) Define the log to have enough space to process the maximum number of inflight transactions.
Option 2) is of course the most logical and easiest to implement. My choice would be Option 3):
3) Use a more traditional messaging model and process the messages as they arrive on the queue rather than in batches.
I imagine you've already discounted this due to bizarre business requirements per pro your previous post.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| udayakumarisampalli |
| Posted: Thu Dec 14, 2006 1:50 am Post subject: |
|
|
Novice
Joined: 20 Jul 2006
Posts: 19
Location: India
|
hi Vitor,
i could understand u r options. but they may not be usefull for me.
i have running this flow now i need to process all the messages in queue at each intervel.
i made the trnsaction to 'NO' in both MQGet and MQOutput and sending the persistent messages. now for every 40 messages i am missing one message. say if i am sending 82 messages i am getting 80 messages in the output which i could say as at 41 message thread is getting snached and procssing from 42 message i am missing this 41 message.
i need to all the messages. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 14, 2006 2:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
You should check the errors log to see if a message is being thrown about "resource shortage", "log full", etc, etc, or if there's any other problem being reported.
You should also trace the flow to prove there's not some bizzare logic error knocking this message out, or a broker error of some kind being thrown that prevents the message being put.
I still don't get the requirement you're running under - why batch processing of messages like they're file records? 
Still, to each his own.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| udayakumarisampalli |
| Posted: Sun Dec 17, 2006 3:29 am Post subject: |
|
|
Novice
Joined: 20 Jul 2006
Posts: 19
Location: India
|
hi,
i have used the user Trace to identify for the missing message. i found a strange behaviour from the TimeoutNotification node i.e
as this node is triggering after the given time interval say after 3 minutes at that point of time when the message flow processed the 40 messages the TimeoutNotification node again firing becasue of this flow starting from the bigining and missing the one message without sending it to output Queue. this is only happneding when it is processing the messages. once it has processed all the messages it is not firing till it completes the timeinterval.
and also when it triggered and processing messages by the time it reaches 40 messages it is just completing the 1 or 2 seconds, i dont know why TimeoutNotification node is Triggering again. can any body have any idea on this. do i need to set any properties.
one more this i have observed is that when i given the time interval as 10 minutes, it is triggering when it processed around 300 messages i.e after 3 to 4 seconds when it start processing the messages. i found this behaviour only when it is processing the messages rest of the times it is working perfectly.
any kind of inputs are awaited  |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sun Dec 17, 2006 5:31 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It sounds to me like there is a bug in your flow downstream of the Timeout node, that is causing an exception that gets propagated all the way back to the Timeout node. But it only happens on a certain message.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| udayakumarisampalli |
| Posted: Sun Dec 17, 2006 9:14 pm Post subject: |
|
|
Novice
Joined: 20 Jul 2006
Posts: 19
Location: India
|
hi Jefflowrey,
may i know in detail what does you mean when you said 'bug in Flow downstream of the timeoutNode'
thank you
regards
Uday |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Dec 18, 2006 4:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I mean your code is generating an error somewhere, and this is causing the error to get pushed back up to the Timeout node.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bijesh |
| Posted: Mon Jul 16, 2007 12:51 am Post subject: |
|
|
Acolyte
Joined: 30 Jan 2007
Posts: 66
|
| Do you have any updates on this problem??? |
|
| Back to top |
|
|
| bijesh |
| Posted: Mon Jul 16, 2007 3:24 am Post subject: |
|
|
Acolyte
Joined: 30 Jan 2007
Posts: 66
|
Hi All,
Am also facing the similar problem which udayakumarisampalli wrote. the flow I've written does not process all the messages available in queue using MQGet node. Am making use of MQGet node to get the messages from the queue as the flow need to be triggered at a ceratin time interval.
Any thoughts on this problem ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 16, 2007 3:29 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bijesh wrote: |
| Am making use of MQGet node to get the messages from the queue as the flow need to be triggered at a ceratin time interval. |
I think most of the previous comments apply regarding why you're doing this. If you have a scheduled time constrant, why not use a proper scheduling package to control the flow and stick an MQInput node on the front?
Are you getting the same problem, or a similar one? What error are you getting? What symptoms are you seeing?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
