ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportBizTalk with MQSC and queue size limit

Post new topicReply to topic Goto page 1, 2  Next
BizTalk with MQSC and queue size limit View previous topic :: View next topic
Author Message
cml
PostPosted: Wed Dec 14, 2016 7:42 am Post subject: BizTalk with MQSC and queue size limit Reply with quote

Newbie

Joined: 14 Dec 2016
Posts: 3

Hi all,

We have a Windows BizTalk server running with the MQSC adapter installed that GET's messages from an MQ queue hosted on a IBM WebSphere MQ server in zOS/Mainframe. Our setup is this:

Server 1: Windows, BizTalk, MQSC adapter, WebSphere MQ Client.
Server 2: z/OS Mainframe, WebSphere MQ.

Message fetching from the queue and parsing in BizTalk works fine for whole messages (one physical message on MQ) but not if the split the actual file into multiple messages and put it on the queue. The MQSC adapter on BizTalk does not seem to able to correlate/assemble the different physical messages into one and parse it along. Right now, it would just process each received message individually and then we get a bunch of parsing errors in BizTalk.

So, what do we need to do on the host side (z/OS) to split a very large file into multiple messages, PUT it on the queue and GET it with MQSC in BizTalk? For instance we want to transefer an XML file in three parts. We have tried with messageID and groupID, but still MQSC/BizTalk would process each physical MQ message individually and not as a group.

I hope I've described my problem thoroughly and that someone are able to help.

Thanks, Christian.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Dec 14, 2016 7:51 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

A group is a set of messages that are tagged so that they can be assembled by the receiving application.

A segment is part of a set of messages that can be assembled by either the queue manager or the application.

You might look at putting an intermediate queue manager between zOS and BizTalk, then you could maybe have that queue manager assemble segments, instead of having the zOS queue manager do it.

But you first need to get the zOS application to create segmented messages...
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Dec 14, 2016 7:59 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5999

mqjeff wrote:
...But you first need to get the zOS application to create segmented messages...

As per the KC:

Quote:
Message segmentation

Note: Not supported in IBM MQ for z/OS or by applications using IBM MQ classes for JMS.

_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Dec 14, 2016 8:05 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Dec 14, 2016 8:52 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5999

mqjeff wrote:
I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java.

It's the 'Not supported in IBM MQ for z/OS' that was the eye-catcher for me
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Dec 14, 2016 9:02 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

exerk wrote:
mqjeff wrote:
I'd be surprised if the BizTalk MQ adapter used JMS... given that it's almost certainly not written in Java.

It's the 'Not supported in IBM MQ for z/OS' that was the eye-catcher for me


Sure... but I did mention something about an intermediate qmgr...

Dunno if that can segment messages from channels.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Dec 14, 2016 9:15 am Post subject: Re: BizTalk with MQSC and queue size limit Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25554
Location: Ohio, USA

cml wrote:
For instance we want to transefer an XML file in three parts. We have tried with messageID and groupID, but still MQSC/BizTalk would process each physical MQ message individually and not as a group.


Why are you trying to send 100Mb+ as a single XML document?

(Anything smaller would fit in a single message)
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
cml
PostPosted: Wed Dec 14, 2016 9:26 am Post subject: Reply with quote

Newbie

Joined: 14 Dec 2016
Posts: 3

We would sometimes have to process files larger than 100MB. It could be a statement of account over the last couple of years or something like that.

Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Dec 14, 2016 9:31 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

cml wrote:
Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file.


You'd have to look at the BizTalk/client adapter documentation.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Dec 14, 2016 11:37 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19923
Location: LI,NY

cml wrote:
We would sometimes have to process files larger than 100MB. It could be a statement of account over the last couple of years or something like that.

Question is, does the MQSC client adapter support grouping? I would like the adapter to concatenate the received messages according to their logical sequence number and grouping. Say we have a XML file split into two, they would be concatenated and processed as one single XML file.

I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements?

Why do you need such big messages / files. Typically they contain a number of aggregations that you could probably split along the aggregation lines into a number of messages that are a lot smaller...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Wed Dec 14, 2016 11:49 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25554
Location: Ohio, USA

fjb_saper wrote:
I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements?




I also question why the drive to send this over MQ. If you're constrained to use MQ as a transport, then use MQ FTE to move what is in fact a file and get BizTalk to read that. Failing that, use ftp / sftp / Connect : Direct / any of the other z/OS -> distributed file transport tools.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Dec 14, 2016 12:19 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8343
Location: US: west coast, almost. Otherwise, enroute.

Vitor wrote:
fjb_saper wrote:
I am certain if you get something that bit that zOS has the better performance for going through it and creating multiple messages out of it.
Say a multi-year statement gets split into yearly / monthly statements?




I also question why the drive to send this over MQ. If you're constrained to use MQ as a transport, then use MQ FTE to move what is in fact a file and get BizTalk to read that. Failing that, use ftp / sftp / Connect : Direct / any of the other z/OS -> distributed file transport tools.


_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
cml
PostPosted: Wed Dec 14, 2016 12:25 pm Post subject: Reply with quote

Newbie

Joined: 14 Dec 2016
Posts: 3

Thanks, that certainly clarifies it. Think we would settle for some application level segmentation on z/OS or just use another protocol to transfer really large files, likely FTP.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Dec 14, 2016 12:29 pm Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25554
Location: Ohio, USA

cml wrote:
Think we would settle for some application level segmentation on z/OS


They tell me COBOL/370 has an XML parser now. It was all different back in my day.....

cml wrote:
just use another protocol to transfer really large files, likely FTP.


This is probably your fastest & easiest route. @mqjeff will be along in a moment to talk about the halting problem, which is a lot easier to deal with if your file is XML (very easy to tell if bits are missing)
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Dec 14, 2016 12:45 pm Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Vitor wrote:
which is a lot easier to deal with if your file is XML (very easy to tell if bits are missing)


Only if we accept that the XML was well formed in the first place...
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexGeneral IBM MQ SupportBizTalk with MQSC and queue size limit
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.