| Author |
Message
|
| kash3338 |
 Posted: Sun Feb 08, 2009 8:19 pm Post subject: Doubt regarding reading all messages from queue one by one |
 |
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
Hi,
I have a doubt regarding MQInput/MQGet nodes.
There is a senario where in i want to trigger my Message flow at one point of time in a day and at that point of time i need to read all the messages present in the queue one by one. For example, say i want to trigger my flow daily at 5pm and assume there are 100 messages in queue at that point, i need to read all those 100 messages one by one starting at 5pm.
How do i do that? Should i loop the message flow or is there any property available in MQInput/MQGet nodes?
My Message flow : MQInput/MQGet ---> Compute ----> MQOutput
Version of MB : 6.1
Regards,
Kashyap. |
|
| Back to top |
|
 |
| MQEnthu |
| Posted: Sun Feb 08, 2009 9:32 pm Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
| Quote: |
For example, say i want to trigger my flow daily at 5pm and assume there are 100 messages in queue at that point, i need to read all those 100 messages one by one starting at 5pm.
|
You can achive this by shceduling the flow using timer nodes. So that flow can be triggered at any specified time.
| Quote: |
| My Message flow : MQInput/MQGet ---> Compute ----> MQOutput |
MQGet is to read the messages from queue in the middle of the flow. Not at the begining of the flow. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Feb 08, 2009 11:41 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Tip:
If you are reading lots of messages using MQGet I suggest you read some of the recent posts in this forum about how to do it without using excessive resources in the ExecutionGroup.
There are many ways to trigger a flow at a set time.
I'm probably from the 'old school' in that if I were doing this, I'd trigger the flow externally using something like a cron job that sends a simple MQ message using amqsput (or similar). This way, i can:-
1) easily test it during development
But more importantly,
2) trigger it manually if required due to operational purposes. eg, there is a system outage at 16:55 and is not restored until 17:25.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 09, 2009 1:20 am Post subject: Re: Doubt regarding reading all messages from queue one by o |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kash3338 wrote: |
| i need to read all the messages present in the queue one by one. |
That's the only way you can read messages! Even if you use the MQInput node (much the better method) then you still get the messages one at a time.
I therefore fail to understand your question. Please clarify it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Feb 09, 2009 1:33 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
It's architecturally better to process messages as they arrive on the queue and each one is processed by the message flow independently.
That is how WMB is meant to work.
Using Timers and MQGET loops is turning an event driven broker into a sort of batch FTP job, which is not really ideal.
Message flow processing should be atomic and not aware of other messages either before or after the one being processed.
If you must group messages - use the MQ message grouping feature.
Why turn MQ into FTP? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 09, 2009 4:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
This is likely end-of-day processing in a retail kind of environment.
Read the messages as the arrive on the queue and process them immediately.
Store them in a database, indexed so that they can be retrieved "in order".
Write a flow that is triggered using any of the methods described to read the records from the database. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 09, 2009 4:29 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
This is likely end-of-day processing in a retail kind of environment.
Read the messages as the arrive on the queue and process them immediately.
Store them in a database, indexed so that they can be retrieved "in order".
Write a flow that is triggered using any of the methods described to read the records from the database. |
This method is pretty straightforward to implement provided you get over the hurdle of casting the Properties & the MQMD folders as BLOBS so that the can go into the DB as a single column (1 for each). The incoming message should be read as a blob but bear in mind that if the messages are large (>100kb) you may have to use a dedicated Large Object Tablespace in your DB.
Also note the recent posts here about message propagation loops and execution group resource conservation (on the replay side)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 09, 2009 4:32 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| smdavies99 wrote: |
| This method is pretty straightforward to implement provided you get over the hurdle of casting the Properties & the MQMD folders as BLOBS so that the can go into the DB as a single column (1 for each). |
This assumes that the business processing needs information that is in the MQMD or Properties. Which I'd design it not to do, and prefer it not to do.
I'd design the database to match the business data and business requirements, and store it that way. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Feb 09, 2009 8:46 pm Post subject: Re: Doubt regarding reading all messages from queue one by o |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| Vitor wrote: |
| kash3338 wrote: |
| i need to read all the messages present in the queue one by one. |
That's the only way you can read messages! Even if you use the MQInput node (much the better method) then you still get the messages one at a time.
I therefore fail to understand your question. Please clarify it. |
Actually I trigger my flow using a timer node (Say at 5pm daily). Once the flow is triggered, i need to read the messages from the queue one by one and process it. But what is happening is, once the flow is triggered, the flow reads the first message from the queue and once its processed, its not picking up the second message. How do i do that? |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Feb 09, 2009 8:52 pm Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| zpat wrote: |
It's architecturally better to process messages as they arrive on the queue and each one is processed by the message flow independently.
That is how WMB is meant to work.
Using Timers and MQGET loops is turning an event driven broker into a sort of batch FTP job, which is not really ideal.
Message flow processing should be atomic and not aware of other messages either before or after the one being processed.
If you must group messages - use the MQ message grouping feature.
Why turn MQ into FTP? |
Actually my requirement is to trigger the flow only at say 5pm daily. Till that time(5pm) many messages might fall in the queue and my flow should not process them until 5pm.
Once the flow starts at 5pm, i need to read each and every message separatly(one by one) and process it separatly. Now what is happening is, once the message flow is triggered, only the first message is read and it does not pick up the subsequent messages. |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Mon Feb 09, 2009 8:59 pm Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
| Quote: |
| Now what is happening is, once the message flow is triggered, only the first message is read and it does not pick up the subsequent messages. |
Please explain how you are using the timer node ...
_________________
-----------------------------------------------
It is good to remember the past,
but don't let past capture your future |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 09, 2009 11:07 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
A simple way to do the triggering is to have the flow is a stopped state then is a batch/cron script that is scheduled to run at 17:00, it starts the flow.
A similar script would run some time later to stop the flow if the depth of the input queue was = 0.
In reality, there are many ways to trigger a flow from outside of Broker. These methods help to keep what you re doing inside broker SIMPLE.
You could spend a lot of time trying to make Broker do this reliably in all business conditions but following KISS rules should make for a very reliable solution.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Feb 10, 2009 12:21 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| You can also control this by using Get Inhibited on the queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 10, 2009 1:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kash3338 wrote: |
| [ Now what is happening is, once the message flow is triggered, only the first message is read and it does not pick up the subsequent messages. |
Does the triggered flow start with an MQGet node (which only reads one message) or an MQInput node (which reads the queue one message at a time until the queue is empty)?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
