| Author |
Message
|
| ChristianH |
 Posted: Mon Jan 19, 2009 5:11 am Post subject: Deleting permanent dynamic queues in WMB |
 |
|
Novice
Joined: 27 Sep 2007
Posts: 19
Location: London, UK
|
Hello,
I was wondering, if there is any way to delete permanent dynamic queues within a Message Broker Flow!?
I couldn´t find anything in the forum nor in the info center.
We have some flows that pass on data over some queues. These flows can run in several instances.
The data of each instance has to be processed separatly, which we realised by reading it by correlation id in the next flow. We would like to change this for performance reasons and also to have it easier, if any flow stops because of an error.
That´s why we came up with the idea to separate the data by putting them on different permanent dynamic queues, which are read with a Get-Node in the following flow.
The question is: How can we delete the permanent dynamic queues after the data is processed from the flow.
Do we have to do it "manually" in a Java Compute Node? (Which would be out of transaction control, wouldn´t it?)
Or is there another way?
Thanks in advance for any tips/help! |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Mon Jan 19, 2009 6:35 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Give each unique flow its own regular local queue that is accessed by the MQGET node.
Multiple instances of the same flow can share the same local queue. I can't imagine the queues growing so deep tha the MQGET node takes too long to get its message. If you have performance problems because of this, something else is wrong.
Now that you have regular local queues, you can monitor them and alert on them, and you don't have to worry about cleaning up dynamic queues.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ChristianH |
| Posted: Mon Jan 19, 2009 6:51 am Post subject: |
|
|
Novice
Joined: 27 Sep 2007
Posts: 19
Location: London, UK
|
How do you choose which local queue is free to use for putting on it?
If we define as many flows as the number of instances we have, how do you tell each flow which queue to use?
To explain how we get to have so much data on the queues:
The flows are processing SEPA data which comes in as a big file. This leads to many messages on the queue. If several files are being received at the same time, we want to process them parallel. Each file is split up into several messages, which is the reason why we have so many messages and performance problems.
But the performance is not the only reason why we would like to separate the data. If something goes wrong in the processing, it´s easier for the attendance, if the data/file causing the problem is on a separate queue. Having the file name as identifier as part of the queue name would also help.
Our idea was to create different dynamic queues with a fixed prefix and the filename as identifier at the end for separating the data and passing the name of the queue over a trigger message. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 6:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ChristianH wrote: |
| How do you choose which local queue is free to use for putting on it? |
a) Why do you care? The messages will queue up in the approved manner.
b) Use any kind of distribution mechanism in the ESQL you like
| ChristianH wrote: |
| If we define as many flows as the number of instances we have, how do you tell each flow which queue to use? |
Supply it as a property of the deploy.
| ChristianH wrote: |
| And passing the name of the queue over a trigger message. |
You can't trigger flows. Not without a lot of work that's not worth it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Jan 19, 2009 6:56 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Use the MQGET node to get by CorrelID from the local queue. That way it doesn't matter if there are other messages waiting, either by other active instances, or perhaps orphaned replies that never came back for whatever reason.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 6:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| Use the MQGET node to get by CorrelID from the local queue. That way it doesn't matter if there are other messages waiting, either by other active instances, or perhaps orphaned replies that never came back for whatever reason. |
Ingenious! And allows for trigger messages (if not "trigger" in the WMQ sense).
I wish I was this clever. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ChristianH |
| Posted: Mon Jan 19, 2009 7:14 am Post subject: |
|
|
Novice
Joined: 27 Sep 2007
Posts: 19
Location: London, UK
|
| PeterPotkay wrote: |
| Use the MQGET node to get by CorrelID from the local queue. That way it doesn't matter if there are other messages waiting, either by other active instances, or perhaps orphaned replies that never came back for whatever reason. |
That is our current solution.
| Quote: |
| Supply it as a property of the deploy. |
I don´t know, if I understand correctly. I think you are refering to a flow which is deployed several times. In fact we deploy the flow only a single time, but with the additional instances set to e.g. 4.
In that case all the properties of the instances are the same aren´t they?
One disadvantage of the multiple local queue based solution is that you can not increase the number of additional instances without having to have an MQAdministrator create new local queue objects.
| Quote: |
| You can't trigger flows. Not without a lot of work that's not worth it. |
Sorry, "trigger message" was the wrong term. I forgot that this name is already assigned to the messages created when triggers are used.
What I meant is that we have the following flow listen on another queue and we want to pass the queue name for the Get-Node over a message to "trigger" that flow.
Your answers make me think that there is no easy way to delete dynamic queues from inside a flow. Maybe we just need another applications that cleans up unused dynamic queues after they aren´t used anymore... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jan 19, 2009 7:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You don't understand what Peter meant by "unique" flow.
Each additional instance is not a "unique" flow.
Even if you deploy the same flow to multiple EGs on the same broker, with as many additional instances as you want, you still only have one flow that needs one queue.
Each instance of the flow, regardless of whether it's an additional instance or not, will be able to get it's specific and unique message from the same queue that all the rest are using by using the CorrelID.
If you have a different flow (that is, a different .msgflow *file* in your toolkit) that is doing different things, then that is a "unique" message flow, and needs it's own queue.
Again, it's not clear how you can have performance issues using this approach - unless there are hundreds of messages piled up on the queue for a given flow, such that issuing MQGET with CorellId takes a while. But then you have to look at everything else the flow is doing, and not at whether you should be using permdyn queues. Like maybe you just need more additional instances. Or to recode your flow for better performance...
If you really think, somehow, you will be increasing performance by using permdyn queues, you can write a flow that starts with a TimeoutNotification node and issues a PCF message to the command server to delete some permdyn queues you think are "not used" any more. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 7:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ChristianH wrote: |
| In that case all the properties of the instances are the same aren´t they? |
Yes. How hard it is to issue a deploy command 4 times? You could script it.
One disadvantage of
| ChristianH wrote: |
| the multiple local queue based solution is that you can not increase the number of additional instances without having to have an MQAdministrator create new local queue objects. |
Or have the administrator create 200 local queue objects as a one-time task and only use the ones you need (presumably 4 in this example). Then if you need 5, or 6, or 200, you just deploy the flows.
| ChristianH wrote: |
| Maybe we just need another applications that cleans up unused dynamic queues after they aren´t used anymore... |
Maybe you need to think what's more work - reworking your flows along the lines suggested, or building an application that cleans up as you suggest. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ChristianH |
| Posted: Mon Jan 19, 2009 7:34 am Post subject: |
|
|
Novice
Joined: 27 Sep 2007
Posts: 19
Location: London, UK
|
The correlation id solution is exactly what we are using at the moment.
The reason why we have a lot of messages piled up before the 2nd flow can start getting them by correlation id is that the first flow calls a WTX map which processes the file (~500 mb Sepa files) and puts a lot of messages on the queue for the next flow. Only after the WTX map is finished the "trigger" message is written by the 1st flow to the 2nd flow which starts getting the data by correlation id.
We experienced performance issues when several files are processed at the same time. That is why we think putting the data on separate queues should speed up the 2nd flow.
Would it be possible to issue the PCF Command within the transactional control of the flow? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 7:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ChristianH wrote: |
| Would it be possible to issue the PCF Command within the transactional control of the flow? |
Only if the flow was running with mqm authority and you wouldn't want that.
IMHO this whole permanent dynamic thing is more trouble than it's worth; you might make it work but you'll tie yourself in knots doing it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jan 19, 2009 7:44 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Only if the flow was running with mqm authority and you wouldn't want that. |
That's a curious assertion. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 7:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| Only if the flow was running with mqm authority and you wouldn't want that. |
That's a curious assertion. |
Ok, I wouldn't want that.
I stand chided. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ChristianH |
| Posted: Mon Jan 19, 2009 7:52 am Post subject: |
|
|
Novice
Joined: 27 Sep 2007
Posts: 19
Location: London, UK
|
| Vitor wrote: |
| ChristianH wrote: |
| Would it be possible to issue the PCF Command within the transactional control of the flow? |
Only if the flow was running with mqm authority and you wouldn't want that.
IMHO this whole permanent dynamic thing is more trouble than it's worth; you might make it work but you'll tie yourself in knots doing it. |
Ok, thanks for the advice! I will carefully discuss all the advantages and disadvantes of the 3 possible solutions with my colleagues. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 19, 2009 7:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ChristianH wrote: |
| I will carefully discuss all the advantages and disadvantes of the 3 possible solutions with my colleagues. |
Remember that, as with many things in the integration space, there's no right design solution; just a design solution that's right for you.
Please post what you eventually decide on.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
