| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Segmentation |
« View previous topic :: View next topic » |
| Author |
Message
|
| rockNroll |
 Posted: Wed Jan 05, 2011 7:41 pm Post subject: Segmentation |
 |
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
Hi guys.
We have a scenario wherby external customers send files to our host (z/OS, MQv6), and the host onforwards the files to clustered app servers (AIX, WMQv7).
Our preference is to use MQ, instead of MQ FTE or CA-XCOM, to deliver the files to the app server(s).
However some of the files are likely to exceed 100MB in size. As z/OS doesn't support Segmentation we need to split these open into bite sized chunks and use Grouping to ensure they get to the other end in the right sequence.
Is anyone aware of any plans by IBM to
1. Implement message segmentation on z/OS, and/or
2. Increase the maximum message size to something larger than 100MB? |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Jan 05, 2011 10:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
My understanding is that zOS ftp has a lot more rigorous controls around it than Unix ftp and will guarantee that the file is fully there. So I don't quite see your point 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 05, 2011 10:24 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 8:02 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| We have a scenario wherby external customers send files to our host (z/OS, MQv6), and the host onforwards the files to clustered app servers (AIX, WMQv7). |
Exactly how are your external customers creating messages from files?
IBM supplied a pair of exits (as support pacs, if memory serves) to be installed at each end of the file-message-file route. Are you using these exits?
How does your AIX mq subsequently create files from the messages?
If the z/OS qmgr is merely forwarding the inbound messages to a transmission queue, then you should have no problems.
What am I missing here?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Thu Jan 06, 2011 12:15 pm Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
Guys many thanks for taking the time to reply.
| fjb_saper wrote: |
| My understanding is that zOS ftp has a lot more rigorous controls around it than Unix ftp and will guarantee that the file is fully there. So I don't quite see your point |
We looked at sftp but it doesn't give us the monitoring/automation facilities we need in our current environment (e.g. file arrival triggering a job through CA-Solve). In addition to sending files to the AIX app, we will also be receiving files from it back to the host. The app vendor doesn't support xcom (our standard file transfer mechanism). As we can control the file sizes coming back from AIX and as we will also be implementing online services, we would like to use MQ for everything.
| PeterPotkay wrote: |
| But you may want to raise the requirement via official channels if you would like to see this in the product. |
Thanks for the link. I've searched around this site and it seems like the 100MB limit comes up quite a few times. To me it seems like an arbitrary limit - I could understand 64K or 16MB, but where does 100MB come from?
| bruce2359 wrote: |
| Exactly how are your external customers creating messages from files? |
The external customers ship their files to us via host-host connections, not via MQ - these are old legacy feeds. The files may contain millions of individual transactions, which we need to forward to the AIX app for processing. We have no control over the size of the files they create and therefore we can't simply drop them onto a batch channel MQ queue without first ensuring we haven't broken the 100MB limit |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 12:57 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| We have a scenario wherby external customers send files to our host (z/OS, MQv6), and the host onforwards the files to clustered app servers (AIX, WMQv7). |
So, WMQ is not yet involved in this process, yes?
The 100MB physical message limit is only a single physical message size limit. An application can create a message group that creates a logical message that can exceed the 100MB limit.
Message groups are described in the WMQ Application Programming Reference, and the WMQ Application Programming Guide.
So, yes, an application can read a file from the filesystem, and create a logical message (of nearly any size). That logical message can be passed to/through a z/OS qmgr to another midrange qmgr.
Message segmentation is different from message groups.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Thu Jan 06, 2011 1:48 pm Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
| bruce2359 wrote: |
So, WMQ is not yet involved in this process, yes?
|
That's correct Bruce. I mentioned (z/OS, MQv6) just to indicate what our current environment.
| bruce2359 wrote: |
| The 100MB physical message limit is only a single physical message size limit. An application can create a message group that creates a logical message that can exceed the 100MB limit. |
Yes agreed, but we have to write code to split the message on the sending side, create a group and then reassemble the segments on the receiving side. If z/OS supported segmentation we could use the MQMF_SEGMENTATION_ALLOWED and MQGMO_COMPLETE_MESSAGE options to do the job for us.
Or alternatively IBM could extend the 100MB limit to say 2GB and the issue goes away. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 06, 2011 1:50 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bulls hit wrote: |
| Or alternatively IBM could extend the 100MB limit to say 2GB and the issue goes away. |
Hit that link. Raise that requirement.
And be glad it's not the good old days, when distributed platforms dreamed of a 100Mb limit.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Thu Jan 06, 2011 2:06 pm Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
| Vitor wrote: |
And be glad it's not the good old days, when distributed platforms dreamed of a 100Mb limit. |
Haha indeed.
Seems to me however that the 100MB limit itself dates to the 'good old days' |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 2:41 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
In a prior post, you said that MQ is not involved currently. I this true?
Or, have you already written a program that creates msgs from files; and now you want z/OS to do something with these messages? I'm confused.
| Quote: |
| Yes agreed, but we have to write code to split the message on the sending side, create a group and then reassemble the segments on the receiving side. If z/OS supported segmentation we could use the MQMF_SEGMENTATION_ALLOWED and MQGMO_COMPLETE_MESSAGE options to do the job for us. |
No. Do not confuse message groups with message segmentation. Two different animals. You need to focus on message groups.
You are correct that z/OS doesn't support message segmentation.
But if z/OS WMQ is only to perform as a hub, then no application coding is necessary on z/OS. Secondly, MQMF_SEGMENTATION_ALLOWED is a directive to the qmgr that it (the qmgr) can segment a message if necessary (too big for application buffer, too big for queue, etc.). This has to do with message segmentation, NOT file segmentation.
You will need to write an application that creates a message group.
An application can read the file you wish to ship, copy the first 100MB from the file, then mqput this logical message (not segment) to the queue. The MQMD has a field for GroupId - like MsgId.
As your app mqputs physical messages that are part of the message group, the app will request that the qmgr create a unique GroupId, then the app will set a message-in-group flag. For the last message in group, the app will set a last-message-in-group flag. There is a msg-sequence number field that increments by 1...
The app that mqgets the message group will specify _LOGICAL_ORDER and _MSG_IN_GROUP flag.
Take a look at the MQPMO in the WMQ APR manual for: MQPMO_LOGICAL_ORDER (read until you see the words 'logical message'.
Take a look at the MQGMO in the WMQ APR manual for:
MQGS_MSG_IN_GROUP and MQGS_LAST_MSG_IN_GROUP
Message groups will and can do exactly what your are asking.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Thu Jan 06, 2011 5:24 pm Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
| bruce2359 wrote: |
In a prior post, you said that MQ is not involved currently. I this true?
Or, have you already written a program that creates msgs from files; and now you want z/OS to do something with these messages? I'm confused.
|
Apologies for the lack of clarity. We currently receive files of transactions from external parties and process them on the host. No MQ involvement.
What we are planning to implement is a new AIX-based app to process the transactions. What we would like to do is to simply trigger a host batch job on file arrival (from the external party) that creates a single MQ message containing the file. The batch program would achieve this by reading the file sequentially into an array and then PUT the contents of the array to MQ as a single message. The receiving app would extract the individual transactions from the message for processing.
| bruce2359 wrote: |
No. Do not confuse message groups with message segmentation. Two different animals. You need to focus on message groups.
An application can read the file you wish to ship, copy the first 100MB from the file, then mqput this logical message (not segment) to the queue. The MQMD has a field for GroupId - like MsgId.
As your app mqputs physical messages that are part of the message group, the app will request that the qmgr create a unique GroupId, then the app will set a message-in-group flag. For the last message in group, the app will set a last-message-in-group flag. There is a msg-sequence number field that increments by 1...
The app that mqgets the message group will specify _LOGICAL_ORDER and _MSG_IN_GROUP flag.
Take a look at the MQPMO in the WMQ APR manual for: MQPMO_LOGICAL_ORDER (read until you see the words 'logical message'.
Take a look at the MQGMO in the WMQ APR manual for:
MQGS_MSG_IN_GROUP and MQGS_LAST_MSG_IN_GROUP
Message groups will and can do exactly what your are asking. |
Yes agreed. But the point I'm trying to make is that I wouldn't need to concern myself with Grouping and the coding required to break the file into 100MB chunks if z/OS supported Segmentation. Given enough buffer space I could simply create a large MQ message and let the queue manager break it into as many bits as necessary for delivery and then reassemble them at the other end and deliver the entire MQ message to the receiving app.
Beats me why Segmemtation is supported on every other platform apart from z/OS.
At any rate I've sent a change request to IBM. Who knows, maybe they'll do something about it before I die |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jan 06, 2011 5:44 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You should really be thinking about breaking the file up into logical transactions rather than just into chunks.
And it seems to make a lot more sense to do this before the data gets anywhere near the mainframe in the first place. So you can do lovely things like validate it and ensure it comes from who it is supposed to come from.
But that's just me, I suppose. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 5:56 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm not optimistic that you will see segmentation supported in z/OS in any meaningful time-frame... or ever.
Some things that have killed off the real need for segmentation include: 64-bit app support on most platforms; and primarily, most of the WMQ community is at v6 and v7.
In any case, segmentation is something done either by the qmgr or an application that you write.
The qmgr can attempt to segment under a limited number of conditions, as documented in the APR and APG. Segmentation done by the qmgr will likely not satisfy your need to break apart a really big message in an application-oriented way.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jan 06, 2011 6:14 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| mqjeff wrote: |
You should really be thinking about breaking the file up into logical transactions rather than just into chunks.
But that's just me, I suppose. |
It's not you. I agree. I'm wondering why one application packages (blocks) up a bunch of transactions, only to have some other application unpackage (deblock) them.
Is this design an attempt to reduce network flows? Haven't we learned not to design and write applications around environment/infrastructure that changes frequently?
The OP seems to have proposed a solution, without stating the underlying problem (issue, design criteria). Why this design? What does this design accomplish? What will be the benefits?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rockNroll |
| Posted: Thu Jan 06, 2011 7:24 pm Post subject: |
|
|
Novice
Joined: 05 Jan 2011
Posts: 10
|
| mqjeff wrote: |
You should really be thinking about breaking the file up into logical transactions rather than just into chunks.
And it seems to make a lot more sense to do this before the data gets anywhere near the mainframe in the first place. So you can do lovely things like validate it and ensure it comes from who it is supposed to come from.
But that's just me, I suppose. |
Yes the file is currently validated (and all the transactions processed) on the host after receipt from the customer. We don't want any customer or upstream impact so we're not changing the customer interface, and file validation will continue to be performed on the host.
What is changing is that the transactions will now be processed by the mid tier AIX app. However we do not want to incur the operational overhead of sending several million transactions one by one to the mid tier as then we would need to implement some kind of throttling mechanism.
| bruce2359 wrote: |
| It's not you. I agree. I'm wondering why one application packages (blocks) up a bunch of transactions, only to have some other application unpackage (deblock) them. |
Batch processing old boy. Customers create transaction files and we process them.
| bruce2359 wrote: |
Is this design an attempt to reduce network flows? Haven't we learned not to design and write applications around environment/infrastructure that changes frequently?
|
It would be great if we could write our application to be agnostic to the environment in which it runs. Unfortunately when a vendor (e.g IBM) provides a middleware solution (e.g. MQ) with features (e.g. Segmentation) that are available on some platforms (e.g. AIX) and not on others (e.g. z/OS), then this is not always possible. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
