| Author |
Message
|
| hilla |
 Posted: Sun Nov 23, 2008 2:27 am Post subject: Writing a browser application and running it parallely. |
 |
|
Novice
Joined: 15 Oct 2006
Posts: 14
|
Hey,
I want to write a browser application (that goes over a queue, browses messages and start an application transactions according to the message data).
I'd like to write it so that it is possible to run many instances of this transaction on more than one CICS.
My concern is that the 2 (or more) browsers running parallely will go over the same messages and the application transactions will be started twice instead of once.
Is there a reason for my concern? or does MQ know to "sign" messages that has been browsed already so that the other browser won't handle messages that has been handled already?
Any idea , tips or examples for writing this code?
thanks!
hilla. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Sun Nov 23, 2008 9:54 am Post subject: Re: Writing a browser application and running it parallely. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I'm guessing this is connected to this thread
| hilla wrote: |
| My concern is that the 2 (or more) browsers running parallely will go over the same messages and the application transactions will be started twice instead of once. |
You're right, this will happen.
| hilla wrote: |
| Is there a reason for my concern? |
Yes there is.
| hilla wrote: |
| does MQ know to "sign" messages that has been browsed already so that the other browser won't handle messages that has been handled already? |
Nope, it will only hold a handle for the length of time the message is being browsed.
| hilla wrote: |
| Any idea , tips or examples for writing this code? |
Because you have two applications in 2 different CICS regions (or more) there's no easy way to communicate between the various browsers. Now if you write along the lines you suggest, you can defeat the multiple message handling by passing the triggered transaction the message id it needs to process, and coding it to abort immediately if the message is no longer on the queue. So you still get multiple, duplicate transactions but at least the redundant ones get out of the way quickly.
A more significant issue is (if I understand your design correctly) is that this browser will browse the message, start the correct transaction based on message data, and this transaction will then destructively read the message off the queue.
This read will need to be done by message id, to identify the specific message in question. Now even assuming you've exploted the additional feature of the platform and indexed the queue by message id, this is a hideously inefficient way of reading a queue. If your queue builds to any depth, your throughput will drop like a stone tied to a very heavy anvil.
What about this? Rather than browsing the message, your application reads the message off, decides what transaction is needed and puts it in a specific queue for that transaction where it's started by triggering, being long running or any other method you decide. This gets round the duplicate transaction problem, and means you can scale any of the transactions as needed.
Other solutions are possible and may be better 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Nov 23, 2008 10:52 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I'd say Vitor's dispatch model is pretty much the best you can do.
Have the dispatch input queue triggered in each CICS. Have the applications queues triggered in each CICS. Thus you will share load. Remember it still is a first come/first serve model.
Tclassing the transactions and running with trigger every will allow you to scale running in parallel up to the number of instances specified in the tclass.
Now you have not only a scalable dispatcher but also a scalable transaction execution model.(Tclassed). And you achieve some load balancing across the CICS regions (first come, first serve).
Don't think you can get it any better than that.
I guess you could also look into the "shared queue" concept specific to zOS and CICS.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Nov 23, 2008 12:52 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| My concern is that the 2 (or more) browsers running parallely will go over the same messages and the application transactions will be started twice instead of once. |
WMQ V7 allows browsed messages to be marked as browsed.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Last edited by bruce2359 on Sun Nov 23, 2008 3:05 pm; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Nov 23, 2008 3:28 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Please clear this up for me:
Do you intend to have multiple browsing apps running concurrently in the same CICS; and each spawning processing apps in the local CICS?
Or do you intend to have a single browsing apps running in multple CICSs; and spawning processing apps in the local CICS?
Do you intend to get the browsed message destructively at some point in this process? Like a Unit of Work?
I suppose your browsing app could simply EXEC CICS START the processing app, passing the message data in one of the usual CICS (non-MQ) fashions.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hilla |
| Posted: Tue Nov 25, 2008 3:05 am Post subject: |
|
|
Novice
Joined: 15 Oct 2006
Posts: 14
|
Hey,
I intend to have one application that will run on the TOR, It will read the messages sequentialy, and active by exec cics start the app transaction that will do the destructive get by the msgid it receives.
This browser application is supposed to be run parallely (or on the same CICS, or on 2+ different TOR's), for availability and scalability reasons.
I thought about reading the msg destructively and passing the data using the start data of cics but i am not sure about msgs that will be big (23K and more). I have to figure that out.
I liked the answer of WMQ version 7. This can fix me 
I don't think there is a problem reading the message sequentialy and activating transactions bu msgid, like Vitor wrote. Why is this a problem?
Did you mean msgs that come last will take time untill they are proccesed? What did you mean by -
"If your queue builds to any depth, your throughput will drop like a stone tied to a very heavy anvil. "
Hilla |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 25, 2008 3:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The other approach is not to use browse & triggering (which is relatively inefficient in MQI terms) and just run the CICS application as a long running task (a listener if you like).
Code the application to use MQGET with WAIT. The multiple CICS transactions will then "compete" for messages on the queue.
Since browse is no longer needed, the applications will only get messages that are available to them. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 25, 2008 3:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hilla wrote: |
What did you mean by -
"If your queue builds to any depth, your throughput will drop like a stone tied to a very heavy anvil. "
|
What I meant is that a queue manager is designed to do something different to a database, like a car is designed to do a different task than a truck. When you do a get by msg id, the queue manager has to locate that message in the queue. This is an operation it's not optimised for. Even if you index the queue (as you can under z/OS) you'll find the stone falling scenario if the queue builds to any depth as the queue manager will increasingly struggle to find the required message in a timely fashion.
It's something to watch for in your design.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 25, 2008 3:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hilla wrote: |
I liked the answer of WMQ version 7. This can fix me
|
All you have to do now is convince your system programmers they want to do the upgrade... 
I still have some misgivings how efficient this browse / dispatch model will be. Depends how efficient you need it to be of course.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 25, 2008 3:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Since browse is no longer needed, the applications will only get messages that are available to them. |
I had assumed that each message needed a different transaction to process it, hence my intermediate dispatcher step to filter them.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 25, 2008 5:15 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Sort of, but the MQGET layer could just invoke another CICS transaction via the commarea.
Doesn't have to be a one queue per CICS transaction correspondance.
Triggering is fine for low volume stuff, but when you need to load balance over regions, the volumes are likely to justify a dedicated listener style of interface. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 25, 2008 5:25 am Post subject: Re: Writing a browser application and running it parallely. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Vitor wrote: |
| Other solutions are possible and may be better |
I stand by this!
Seriously, you can skin this cat in a wide variety of interesting ways. As I (and indeed you) intimated above, a lot depends on the actual volumes that you're trying to balance. A solution for 1000 messages per hour looks a lot different to one for 100,000 messages per hour. Also a solution for 1000 2Kb messages per hour looks a lot different to a solution for 1000 2Mb messages per hour.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 25, 2008 7:47 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm pondering the real need for the dispatcher-type browser application you propose.
What is it that the browser application will know about the messages that need consuming that the consuming applications won't know? Wouldn't it be simpler if multiple consuming applications browse the queue looking for eligible messages and destructively MQGETting them?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 25, 2008 8:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
| Wouldn't it be simpler if multiple consuming applications browse the queue looking for eligible messages and destructively MQGETting them? |
Possibly. Certainly conceptually. I can see a couple of instances where you might want exactly that.
But you might not want an unknown number of transactions all browsing the same queue looking for messages. You might be concerned about the number of handles or browse cursors in use especially in a high volume environment where you could end up with multiple copies of each type of transaction. You might want a smaller, simpler transaction that throws the messages onto specialised queues where it's easier to monitor & control how many of a given kind of transaction are running at a given time. Depending on how the messages are identified, it might also mean you can hive off all the message identifying code into a single transaction rather than each specific transaction having it's own, especially if the specific transactions are currently written to assume they only deal with their own data.
Or you might want the solution you've described. It all depends on your situation. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| hilla |
| Posted: Wed Nov 26, 2008 12:08 am Post subject: The whole picture |
|
|
Novice
Joined: 15 Oct 2006
Posts: 14
|
Hey,
I'm about to give you the whole story so hold tight
The thing is that We wrote an alternative for the IBM DPL bridge.
We wanted the alternative code to be as similar to the native one, but to add more features: The ability to run it in a TOR+AORs system and route transaction dynamicly by CICSPLEX, add more logging possibilities, support more than 32K.
All messages arive to the same queue and each message is supposed to be read by a different transaction, that why we need the system application to browse the messages for the CIH header and start the transaction written there.
The way we implemented it, is that the transaction is connected to application program (Which was re-written to support this) which calls a number of system routines that gets the msg, builds the reply, send the msg.
The rates of the messages can get to 30 msgs per second at the peak and our cics system programer would like to have the means to run more than one semi-ckbr (browser application) for the times the TOR won't gave enought resources to deal with the msgs.
That's the whole story.... |
|
| Back to top |
|
|
|
|
