| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Fire Flow When All Messages in MQ are Processed (MQ Empty) |
« View previous topic :: View next topic » |
| Author |
Message
|
| ReemRashwan |
 Posted: Mon Aug 23, 2021 5:18 am Post subject: Fire Flow When All Messages in MQ are Processed (MQ Empty) |
 |
|
Novice
Joined: 09 Mar 2021
Posts: 12
|
TL;DR I need to monitor a queue when its empty after being filled with messages, how can I do that?
Understanding The Application
Hi, I have the following flows in my application
Flow 1: HTTP Input -> some nodes -> HTTP Request (get list of data from other system) -> Compute Node (to split data list into one message at a time) -> MQ Output Node (Name: APPIN)
Flow 2: MQ Input (APPIN) -> some heavy processing -> send data to other system.
Flow 3: HTTP Input -> Nodes to collect some statistics from other systems that is dependent on flow 2 output.
I have a cron job that starts the application using http node in flow 1. Flow 2 may take one or two hours dependening on incoming data from other systems even with multithreading. Flow 3 is called on demand whenever statistics are required
The Question
How can I monitor my queue when its done processing all message that came from other systems, say I have 2000 messages, I need to fire flow 3 only when all of these messages are processed (either succeeded or failed after retries). Any ideas? |
|
| Back to top |
|
 |
| silly_name |
| Posted: Wed Oct 27, 2021 1:18 pm Post subject: Fire Flow When All Messages in MQ are Processed (MQ Empty) |
|
|
Newbie
Joined: 26 Oct 2021
Posts: 5
|
Not 100% sure on what your requirement here is.
You said:
| Quote: |
Flow 3: HTTP Input -> Nodes to collect some statistics from other systems that is dependent on flow 2 output.
|
Do "other systems" not need to do some processing of their own before you can "collect statistics" ? Or is it that you send that data synchronously and they don't do anything to it once flow 2 has completed for each message?
Sticking to your original query, the simplest and most robust way I can think of it is persist in Flow 1 (in a DB?) the initial list of messages - some identifier at least. Then update the status of each identifier in Flow 2. Flow 3 would then run on a timer and query if everything was processed.
Of course things brings to mind a list of other considerations: how do you want to handle errors in case for instance a message is *never* processed, how do you want to archive/cleanup previous runs (sounds like you run in batches once per day?).
Later edit: This sounds like a scenario for the aggregation pattern. Have you looked at the aggregation nodes?
https://www.ibm.com/docs/en/integration-bus/10.0?topic=nodes-group-aggregation
Cheers,
B |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Oct 27, 2021 2:14 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2494
Location: Melbourne, Australia
|
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
