| Author |
Message
|
| cml |
 Posted: Wed Dec 14, 2016 7:42 am Post subject: BizTalk with MQSC and queue size limit |
 |
|
Newbie
Joined: 14 Dec 2016
Posts: 3
|
Hi all,
We have a Windows BizTalk server running with the MQSC adapter installed that GET's messages from an MQ queue hosted on a IBM WebSphere MQ server in zOS/Mainframe. Our setup is this:
Server 1: Windows, BizTalk, MQSC adapter, WebSphere MQ Client.
Server 2: z/OS Mainframe, WebSphere MQ.
Message fetching from the queue and parsing in BizTalk works fine for whole messages (one physical message on MQ) but not if the split the actual file into multiple messages and put it on the queue. The MQSC adapter on BizTalk does not seem to able to correlate/assemble the different physical messages into one and parse it along. Right now, it would just process each received message individually and then we get a bunch of parsing errors in BizTalk.
So, what do we need to do on the host side (z/OS) to split a very large file into multiple messages, PUT it on the queue and GET it with MQSC in BizTalk? For instance we want to transefer an XML file in three parts. We have tried with messageID and groupID, but still MQSC/BizTalk would process each physical MQ message individually and not as a group.
I hope I've described my problem thoroughly and that someone are able to help.
Thanks, Christian. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Dec 14, 2016 7:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
A group is a set of messages that are tagged so that they can be assembled by the receiving application.
A segment is part of a set of messages that can be assembled by either the queue manager or the application.
You might look at putting an intermediate queue manager between zOS and BizTalk, then you could maybe have that queue manager assemble segments, instead of having the zOS queue manager do it.
But you first need to get the zOS application to create segmented messages...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Dec 14, 2016 7:59 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| mqjeff wrote: |
| ...But you first need to get the zOS application to create segmented messages... |
As per the KC:
| Quote: |
Message segmentation
Note: Not supported in IBM® MQ for z/OS® or by applications using IBM MQ classes for JMS. |
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 14, 2016 8:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Dec 14, 2016 8:52 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| mqjeff wrote: |
| I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java. |
It's the 'Not supported in IBM® MQ for z/OS®' that was the eye-catcher for me 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 14, 2016 9:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| exerk wrote: |
| mqjeff wrote: |
| I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java. |
It's the 'Not supported in IBM® MQ for z/OS®' that was the eye-catcher for me |
Sure... but I did mention something about an intermediate qmgr...
Dunno if that can segment messages from channels.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 14, 2016 9:15 am Post subject: Re: BizTalk with MQSC and queue size limit |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cml wrote: |
| For instance we want to transefer an XML file in three parts. We have tried with messageID and groupID, but still MQSC/BizTalk would process each physical MQ message individually and not as a group. |
Why are you trying to send 100Mb+ as a single XML document?
(Anything smaller would fit in a single message)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| cml |
| Posted: Wed Dec 14, 2016 9:26 am Post subject: |
|
|
Newbie
Joined: 14 Dec 2016
Posts: 3
|
We would sometimes have to process files larger than 100MB. It could be a statement of account over the last couple of years or something like that.
Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 14, 2016 9:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| cml wrote: |
| Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file. |
You'd have to look at the BizTalk/client adapter documentation.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 14, 2016 11:37 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| cml wrote: |
We would sometimes have to process files larger than 100MB. It could be a statement of account over the last couple of years or something like that.
Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file. |
I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements?
Why do you need such big messages / files. Typically they contain a number of aggregations that you could probably split along the aggregation lines into a number of messages that are a lot smaller...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 14, 2016 11:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements? |
I also question why the drive to send this over MQ. If you're constrained to use MQ as a transport, then use MQ FTE to move what is in fact a file and get BizTalk to read that. Failing that, use ftp / sftp / Connect : Direct / any of the other z/OS -> distributed file transport tools.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 14, 2016 12:19 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| Vitor wrote: |
| fjb_saper wrote: |
I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements? |
I also question why the drive to send this over MQ. If you're constrained to use MQ as a transport, then use MQ FTE to move what is in fact a file and get BizTalk to read that. Failing that, use ftp / sftp / Connect : Direct / any of the other z/OS -> distributed file transport tools. |
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cml |
| Posted: Wed Dec 14, 2016 12:25 pm Post subject: |
|
|
Newbie
Joined: 14 Dec 2016
Posts: 3
|
| Thanks, that certainly clarifies it. Think we would settle for some application level segmentation on z/OS or just use another protocol to transfer really large files, likely FTP. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 14, 2016 12:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cml wrote: |
| Think we would settle for some application level segmentation on z/OS |
They tell me COBOL/370 has an XML parser now. It was all different back in my day..... 
| cml wrote: |
| just use another protocol to transfer really large files, likely FTP. |
This is probably your fastest & easiest route. @mqjeff will be along in a moment to talk about the halting problem, which is a lot easier to deal with if your file is XML (very easy to tell if bits are missing) 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 14, 2016 12:45 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| which is a lot easier to deal with if your file is XML (very easy to tell if bits are missing) |
Only if we accept that the XML was well formed in the first place...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
