ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » piping data to MQ / segemented PUT call

Post new topic  Reply to topic Goto page Previous  1, 2
 piping data to MQ / segemented PUT call « View previous topic :: View next topic » 
Author Message
fjb_saper
PostPosted: Wed Feb 18, 2015 10:05 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

It all depends on how much data, what distance, what urgency (SLA) you need to achieve. TCP/IP is a poor protocol over huge distances (continent to continent) because of the way it handles the transmission after a lost packet...
So the smaller the unit of transmission, the better. However this generates more overhead.
If you are looking for specific long distance, high volume file transfer, IBM has a solution for that...
We do not know enough about your environment to be of any help.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
RogerLacroix
PostPosted: Wed Feb 18, 2015 10:54 am    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

thorstenhirsch wrote:
When calling PUT the file is being split by MQ into chunks of MAXMSGSIZE automatically.

I don't know where you got that idea from but it is not true. If your application puts a message larger than MAXMSGSIZE (i.e. 4MB default) then MQ will return a reason code of 2030 (MQRC_MSG_TOO_BIG_FOR_Q) or 2218 (MQRC_MSG_TOO_BIG_FOR_CHANNEL) if it is connecting in client mode. There is nothing Auto-Magic in MQ and maybe this is where your confusion is coming from.
thorstenhirsch wrote:
Is you point that the MQ Server is not good at handling huge amounts of data, no matter what size the segments have?

No. Paul is talking about tuning your MQ network (aka messages & channels) to optimize the speed at which messages will travel through the MQ network at your company.

You need to do a bunch of volume tests at your location to determine this. I've done volume testing at some companies and 256KB is the prefer message size whereas other companies it is 800KB. What Paul was politely saying was don't be a dumb ass (not trying to be rude, just trying to use the non polite term & be funny) and think that 50MB or 250MB is an ok message size because it is not.

thorstenhirsch wrote:
@Roger: UFM is next on my list. I downloaded it last week, but wanted to finish MQ first.

Great. Note: When UFM breaks large files into chunks, it uses a message size of 4190000. If you want a smaller size then update the code and change MAX_GROUP_MSG_SIZE to be whatever you want but do NOT exceed 4190000.

thorstenhirsch wrote:
I think you're both right, I will need 2 different technologies. MQ will stay there for authentication, metadata and transaction handling, but we will need a 2nd technology for the "heavy load".

Why? UFM can do it all.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
mqjeff
PostPosted: Wed Feb 18, 2015 11:03 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

RogerLacroix wrote:
thorstenhirsch wrote:
When calling PUT the file is being split by MQ into chunks of MAXMSGSIZE automatically.

I don't know where you got that idea from but it is not true. If your application puts a message larger than MAXMSGSIZE (i.e. 4MB default) then MQ will return a reason code of 2030 (MQRC_MSG_TOO_BIG_FOR_Q) or 2218 (MQRC_MSG_TOO_BIG_FOR_CHANNEL) if it is connecting in client mode. There is nothing Auto-Magic in MQ and maybe this is where your confusion is coming from.

I don't know off the top of my head whether or not segmentation takes channel maxmsgsize into account or not.

RogerLacroix wrote:
thorstenhirsch wrote:
I think you're both right, I will need 2 different technologies. MQ will stay there for authentication, metadata and transaction handling, but we will need a 2nd technology for the "heavy load".

Why? UFM can do it all.

To be fair, MFT can as well.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Wed Feb 18, 2015 12:26 pm    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

mqjeff wrote:
I don't know off the top of my head whether or not segmentation takes channel maxmsgsize into account or not.

Are you referring to the MCA breaking the message into chunks of 32758 bytes that are sent across the channel? If Thorsten was thinking of this then I could see the confusion but what the MCA does at the network level has no bearing on what happens when a regular MQ application does an MQPUT.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
mqjeff
PostPosted: Wed Feb 18, 2015 12:46 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

RogerLacroix wrote:
mqjeff wrote:
I don't know off the top of my head whether or not segmentation takes channel maxmsgsize into account or not.

Are you referring to the MCA breaking the message into chunks of 32758 bytes that are sent across the channel? If Thorsten was thinking of this then I could see the confusion but what the MCA does at the network level has no bearing on what happens when a regular MQ application does an MQPUT.


No, I was thinking of MQMF_SEGMENTATION_ALLOWED.
Back to top
View user's profile Send private message
thorstenhirsch
PostPosted: Wed Feb 18, 2015 2:54 pm    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

Roger, I was referring to MQ Message segmentation handled by the queue manager. Am I confused...? Well, I wonder about the low message sizes of 256kb and 800kb you mentioned. Without letting the QM do the segmentation that would mean that you implemented application-side splitting in all sending applications (or at least in the mq adapter application they use) ...or to just not handle messages > 256kb (> 800kb) at all. Right? And you optimized for message speed. Well, that's not the primary goal of the MQ infrastructure at my workplace. We use MQ whenever possible. There are about 200 servers connected via MQ. Some of them sending up to 50 messages per second with <100kb, and some of them sending only a file once a year, but that one can be larger than 1gb. So we try to optimize for the broadest range of use cases, including large files. Therefore we use the highest possible message size.

Actually it's been working pretty well for 15 or 20 years. MQ is probably the most stable part of our infrastructure ...I would even say that we never had a downtime because of MQ. So far we had a message limit of 500mb, a technical limit due to an application in our main message flow that couldn't handle higher volumes. Now that this application is gone we can handle up to 2gb. The limiting factors are now:

- a 32bit C API for one of our own libraries
- MQLONG

Most applications use a generic adapter application, but some come with their own MQ adapter. We have to deal with about 15 different MQ adapter technologies. I don't want or need to push MQ beyond its limits. I just needed to _prove_ that MQ cannot be used for files >2GB, because moving away from an MQ-only infrastructure implies high costs. Providing a 2nd technology for huge file sizes means that we need to provide a new solution for at least 50 servers and 10 different technologies.

Concerning high performance messaging: in my opinion the requirements are always very specific to applications, and they don't fit well into a centralized integration infrastructure. So we keep them separate and let them use low-latency technologies. That way nobody asks us to provide end-to-end message delivery with MQ in < 1sec.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Wed Feb 18, 2015 3:37 pm    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

thorstenhirsch wrote:
or to just not handle messages > 256kb (> 800kb) at all.

I never said, never implied that.

thorstenhirsch wrote:
And you optimized for message speed.

Yes. Sometimes you need an MQ environment to push 200 messages per second for a particular application, so tuning is very important.

thorstenhirsch wrote:
We use MQ whenever possible.

I never said that either. Anytime 2 applications need to communicate, MQ is usually the best solution.

thorstenhirsch wrote:
that one can be larger than 1gb. So we try to optimize for the broadest range of use cases, including large files.

MQ does NOT send files. It sends messages.

thorstenhirsch wrote:
I don't want or need to push MQ beyond its limits. I just needed to _prove_ that MQ cannot be used for files >2GB, because moving away from an MQ-only infrastructure implies high costs.

Again, MQ does NOT send files. It sends messages. You need to purchase MQ MFT or use UFM, both of which use MQ as the transport mechanism to move files from point A to point B.

thorstenhirsch wrote:
Providing a 2nd technology for huge file sizes means that we need to provide a new solution for at least 50 servers and 10 different technologies.

MQ MFT or UFM is a single product that you would deploy across your MQ environment.

thorstenhirsch wrote:
That way nobody asks us to provide end-to-end message delivery with MQ in < 1sec.

I don't see why you shy away from using MQ. I been involved in several trading applications (financial services) that used MQ which delivered millisecond response time.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
smdavies99
PostPosted: Wed Feb 18, 2015 10:43 pm    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

But don't expect messages that are greater than 5Mb to travel from London to Tokyo[1] in less than 1 second even with direct leased fibre connections.

[1] Other widely spaced destinations are available.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Feb 19, 2015 6:05 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

smdavies99 wrote:
But don't expect messages that are greater than 5Mb to travel from London to Tokyo[1] in less than 1 second even with direct leased fibre connections.

[1] Other widely spaced destinations are available.

Isn't that why you have ASPERA and faspex as a MFT solution???
Using the full bandwidth, over such distances, with ftp / ftps /sftp is a challenge...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2 Page 2 of 2

MQSeries.net Forum Index » IBM MQ API Support » piping data to MQ / segemented PUT call
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.