| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Fire Flow When All Messages in MQ are Processed (MQ Empty) |
« View previous topic :: View next topic » |
| Author |
Message
|
| ReemRashwan |
 Posted: Mon Aug 23, 2021 5:11 am Post subject: Fire Flow When All Messages in MQ are Processed (MQ Empty) |
 |
|
Novice
Joined: 09 Mar 2021
Posts: 12
|
TL;DR I need to monitor a queue when its empty after being filled with messages, how can I do that.
Understanding The Application
Hi, I have the following flows in my application
Flow 1: HTTP Input -> some nodes -> HTTP Request (get list of data from other system) -> Compute Node (to split data list into one message at a time) -> MQ Output Node (Name: APPIN)
Flow 2: MQ Input (APPIN) -> some heavy processing -> send data to other system.
Flow 3: HTTP Input -> Nodes to collect some statistics from other systems that is dependent on flow 2 output.
I have a cron job that starts the application using http node in flow 1. Flow 2 may take one or two hours dependening on incoming data from other systems even with multithreading. Flow 3 is called on demand whenever statistics are required
The Question
How can I monitor my queue when its done processing all message that came from other systems, say I have 2000 messages, I need to fire flow 3 only when all of these messages are processed (either succeeded or failed after retries). Any ideas? |
|
| Back to top |
|
 |
| abhi_thri |
| Posted: Tue Aug 24, 2021 1:18 am Post subject: Re: Fire Flow When All Messages in MQ are Processed (MQ Empt |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
|
| Back to top |
|
|
| ReemRashwan |
| Posted: Thu Aug 26, 2021 5:38 am Post subject: Re: Fire Flow When All Messages in MQ are Processed (MQ Empt |
|
|
Novice
Joined: 09 Mar 2021
Posts: 12
|
Thank you for poining out to these resources.
| abhi_thri wrote: |
| If you are just after statistics why should it matter to wait for the queues to be empty at all? How about just running the stats every hour or so, that should tell you what exactly is going on including the busy time stats. |
Will this seems logical, but the end system that displays statistics doesn't show how many messages in general that should have been processed.
Example: if I have 2000 messages and they take 2 hours, the statistics flow will return: 50 messages have failed and 950 have succeeded, but it will know nothing about the other 1000, I think that is kinda misleading in my case since the statistics flow is meant to run only at the end of operation. Am I missing something?
| abhi_thri wrote: |
| If you really want to run the stats only when the queue is empty add an additional current depth check before starting the stats job. |
How can I check that from IIB? I am thinking of a MQGet node with browse only option checked, and with the No Message Terminal is connected to the stats flow.
Maybe a loop in another flow that keeps calling the MQGet node say every 10 minutes? |
|
| Back to top |
|
|
| timber |
| Posted: Mon Aug 30, 2021 12:49 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I'm confused. You said:
| Quote: |
| Flow 3 is called on demand whenever statistics are required |
but then you also said:
| Quote: |
| I need to fire flow 3 only when all of these messages are processed |
So when can Flow 3 be called?
a) once, but only after Flow 2 has completed
b) multiple times, but only after Flow 2 has completed
c) any time, regardless of whether Flow 2 has completed |
|
| Back to top |
|
|
| ReemRashwan |
| Posted: Tue Aug 31, 2021 12:48 am Post subject: |
|
|
Novice
Joined: 09 Mar 2021
Posts: 12
|
| timber wrote: |
So when can Flow 3 be called?
a) once, but only after Flow 2 has completed
b) multiple times, but only after Flow 2 has completed
c) any time, regardless of whether Flow 2 has completed |
Sorry for misconfusion.
It's "b": ( multiple times, but only after Flow 2 has completed). This happens in a daily basis. If flow 2 runs everyday at 1 am and finishes at 2 am, I can run flow 3 anytime that day after 2 am, but I don't want flow 3 to start for example at 12:30 (before flow 2) or 1:30 am (in the middle of flow), and the faster I run flow 3 after flow 2 completion is better. |
|
| Back to top |
|
|
| timber |
| Posted: Tue Aug 31, 2021 5:53 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
It sounds as if you really need 4 flows, not 3.
- Flow 1: HTTP -> MQ
- Flow 2: MQ -> [other system(s)]
- Flow 3a: Runs immediately after Flow2. Collects statistics data and writes to a database.
- Flow 3b: API that provides read-only access to the most recent Flow2 statistics.
The key design challenge is how to detect that Flow2 has 'finished'. I think you are intending to use 'APPIN queue empty' as the trigger for this. I would kick off a monitoring flow in Flow 1, and automatically trigger Flow 3a (collect statistics) when APPIN queue has been empty for 2 minutes. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
