| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Segmentation |
« View previous topic :: View next topic » |
| Author |
Message
|
| bruce2359 |
 Posted: Thu Jan 06, 2011 8:56 pm Post subject: |
 |
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| However we do not want to incur the operational overhead of sending several million transactions one by one to the mid tier as then we would need to implement some kind of throttling mechanism. |
Huh? Why? When sent from a really, really fast mainframe, several million transactions will arrive one-by-one on the mid-tier platform only as fast as the WMQ channel on the mid-tier can process them.
Are you concerned that the AIX box will be overwhelmed with workload? Is this the purpose of your efforts to get IBM to implement segmentation on z/OS? If IBM were to implement segmentation, then what?
Again I ask, exactly and precisely what is it that you are trying to accomplish?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Jan 06, 2011 8:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| bulls hit wrote: |
What is changing is that the transactions will now be processed by the mid tier AIX app. However we do not want to incur the operational overhead of sending several million transactions one by one to the mid tier as then we would need to implement some kind of throttling mechanism.
|
Can you elaborate on this? Especially the need for a throttling mechanism... I would have thought that in this case, all you had to do is scale the processing side??, and maybe use more than one channel (i.e. if first xmitq fills up use second channel, then third one, then rotate again....)...
We do have a mainframe app (WII) that puts an awful lot of messages within a very short time period to the xmitq to AIX and seems to want to fill it up. Creating a second channel seemed to ease somewhat the bottle neck and increase the throughput... (we split the messages by type to use a different channel)...
Have fun 
_________________
MQ & Broker admin
Last edited by fjb_saper on Thu Jan 06, 2011 9:10 pm; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 9:05 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| the operational overhead of sending several million transactions one by one |
WMQ offers channel attributes that allow for MCAs to create and transmit batches of messages, with a range from 1 to 9999.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Fri Jan 07, 2011 2:39 am Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
| fjb_saper wrote: |
| bulls hit wrote: |
What is changing is that the transactions will now be processed by the mid tier AIX app. However we do not want to incur the operational overhead of sending several million transactions one by one to the mid tier as then we would need to implement some kind of throttling mechanism.
|
Can you elaborate on this? Especially the need for a throttling mechanism... I would have thought that in this case, all you had to do is scale the processing side??, and maybe use more than one channel (i.e. if first xmitq fills up use second channel, then third one, then rotate again....)...
We do have a mainframe app (WII) that puts an awful lot of messages within a very short time period to the xmitq to AIX and seems to want to fill it up. Creating a second channel seemed to ease somewhat the bottle neck and increase the throughput... (we split the messages by type to use a different channel)...
Have fun |
If I can process 1 million transactions with a single message in a single unit of work, what advantage am I getting from processing the same workload with a million messages and a million units of work? Not much other than a lot of logging overhead as far as I can see.
We're gradually moving from a batch to an online real time processing operation. We're trying to find the right mix of processing models and if could use MQ for everything, that would be great |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 07, 2011 2:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bulls hit wrote: |
| If I can process 1 million transactions with a single message in a single unit of work, what advantage am I getting from processing the same workload with a million messages and a million units of work? Not much other than a lot of logging overhead as far as I can see. |
And the fact that you can decide to process those million transactions in MORE THAN ONE PLACE.
So right now if you process 1 million transactions with a single message in a single unit of work, you are incurring the CPU impact of that in a single location.
If you choose to break that out into logical transactions, you can spread the CPU and memory impact over multiple systems - either distributed boxes or LPARS - and thus potentially process MORE than one million transactions at the same time. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 6:57 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| If I can process 1 million transactions with a single message in a single unit of work, what advantage am I getting from processing the same workload with a million messages and a million units of work? |
Are all of the 1million transactions all related? Does all that activity comprise a single Unit of Work? Does all that activity need to either be committed or backed out as a single UofW?
Or, are the 1million transactions really 1million individual, discrete transactions, that arrive as a package from your customer?
| Quote: |
| Not much other than a lot of logging overhead as far as I can see. |
No. You don't seem to understand WMQ logging. A 1million message UofW is a long-running UofW. Please research "long running units of work" in the relevant WMQ documentation.
Logging is one consideration. In order to understand the overall affect of a huge UofW, you will need to estimate (or guess-timate):
How long will this 1million message UofW take on your mid-range AIX box?
If the UofW fails for any reason, how long will it take to backout 1million messages?
If the qmgr fails, how long will it take the qmgr to recover to the last point of consistency?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Jan 07, 2011 7:07 am Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| bruce2359 wrote: |
| Are all of the 1million transactions all related? Does all that activity comprise a single Unit of Work? Does all that activity need to either be committed or backed out as a single UofW? |
Absolutely key point. If so, its a file. Process it in batch.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 7:10 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
If the 1million messages comprise one transaction in a long running UofW, then you will [edit] might [/edit] need to (re)configure your WMQ logs to accommodate that much log space. How to calculate the space required is documented in the WMQ System Admin manual for your platform.
Might need to = If you are using circular logs, I doubt you have enough logs defined for a 1million message UofW.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Sun Jan 09, 2011 11:38 am Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
Guys many thanks again for the considered replies and suggestions.
One of the reasons we don't want to break the file open and send individual transactions to the mid tier cluster because groups of transactions within the file are related, and need to be processed together on the same node.
My only issue is to get the files to the mid tier, one file per message, with the entire contents of the file treated as a CLOB. How the mid tier app handles syncpointing and rollback is its problem, one of the things we're paying the vendor to look after. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jan 11, 2011 10:01 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
You might want to take a look at MQ File Mover offering from one of our sponsors: http://www.capitalware.biz/mqfm_overview.html
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Tue Jan 11, 2011 2:34 pm Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
Or any one of another squillion file transfer products. e.g. PM4Data, which is file transfer over MQ, if you really need to use MQ for file transfer, from CommerceQuest now Metastorm so dunno what the product is called called these days.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
