ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » piping data to MQ / segemented PUT call

Post new topic  Reply to topic Goto page 1, 2  Next
 piping data to MQ / segemented PUT call « View previous topic :: View next topic » 
Author Message
thorstenhirsch
PostPosted: Fri Feb 13, 2015 8:09 am    Post subject: piping data to MQ / segemented PUT call Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

There's this fine q utility from Paul Clarke, which can handle STDIN:
Code:
echo "foo" | q -m $QMgr -o $Queue

Unfortunately this is really slow. I couldn't find a way to speed it up, so currently I'm writing a temp file:
Code:
echo "foo" > file.tmp && q -m $QMgr -o $Queue -F file.tmp && rm file.tmp

This works for most file sizes, but when the file is getting really huge q fails:
Code:
Failed to find end of file

So in order to put this huge file into a Queue, the first code snippet with the pipes might be the solution:
Code:
cat hugefile | q -m $QMgr -o $Queue -as

(-as should enable segmentation)
But after waiting 1 hour q is still working. It is really, really slow.

1.) Is there a parameter in q I've missed so far that lets me set the buffer size? Any other way to speed up STDIN processing in q?

2.) How does q work with STDIN anyway? I'm working with the MQSeries Perl module and couldn't find a way to do a MQPUT in chunks. All the ways to put a message I know so far require me to provide the data at once.


I've placed this thread in the MQ API section because of the 2nd question. Though the first one might provide an easier solution, I'm more interested in the 2nd one ...or in general I'm searching for effective ways to incorporate MQ into my shell/perl scripts.

P.S.: Please ignore the fact, that it's not a good idea to put huge files into a queue. Oh, and just in case you want to suggest to use compression - let's assume the huge file is already compressed.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Feb 13, 2015 8:36 am    Post subject: Re: piping data to MQ / segemented PUT call Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

thorstenhirsch wrote:
P.S.: Please ignore the fact, that it's not a good idea to put huge files into a queue. Oh, and just in case you want to suggest to use compression - let's assume the huge file is already compressed.

Why do you believe that it is not a good idea to have big messages, whatever the content? MQ allows for 100MB max msglength for a single physical message. MQ message-groups allows for logical messages that contain multiple physical messages.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Fri Feb 13, 2015 8:38 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Are you trying to put a file as a single message ? Or do you want each line to be a separate message ?
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
thorstenhirsch
PostPosted: Fri Feb 13, 2015 1:43 pm    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

@Bruce: That's what my MQ administrators tell me. But here in this case they can't forbid me anything. I'm doing a proof-of-concept in a sandbox environment.

@Paul: The whole file should be a single message (logically). MQ may use segmentation, of course, because the file's size is well over 100MB, but that's okay, because my application's MQ adapter that receives the message uses the "complete message" flag in the MQGET call. Actually files larger than 100MB are working fine already. I was using a file with 6GB when q aborted with "Failed to find end of file".
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Fri Feb 13, 2015 5:35 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

By default, the C runtime file functions only support files up to INT_MAX (2GB). You need to compile/link the program with LARGE FILE support to use bigger files. Before I left IBM I did build Q on various platforms with large file support but to be honest I can't remember which or whether I made it generally available.

For a file this large have you considered using File Transfer Edition which is specifically designed for moving files? Q is not really designed for this kind of thing.

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
tczielke
PostPosted: Fri Feb 13, 2015 6:16 pm    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

Also, for this command:

Code:
cat hugefile | q -m $QMgr -o $Queue -as


I wonder if it would be helpful to turn off the standard buffering. I have seen situations where I will run a command like this:

Code:
echo "test1" | amqsput TCZ.TEST1


but sometimes amqsput misses the echo and does not get the input (i.e. no message is PUT). I think you can use stdbuf to turn off that buffering on you shell command.
Back to top
View user's profile Send private message
thorstenhirsch
PostPosted: Sat Feb 14, 2015 4:11 am    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

Thanks for your input. I'll run some tests with stdbuf on monday ...and I guess I should also turn on autoflush in my perl script.

@Paul: Unfortunately the version of q in the support pac I downloaded last week (V6.0, 2012) seems to be compiled without large file support. Any chance of publishing q as open source, under a free license? Should I pose this question to my IBM representative?

One of my colleagues is checking out MQ FTE. He wanted to analyze the requirements in a decentralized scenario... well, he also needs to take a look on huge files, so yes, that might be a solution for my problem as well.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Sat Feb 14, 2015 4:27 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Q has been published as open source take a look at https://github.com/ibm-messaging/mq-q-qload. So you can do pretty much whatever you want with it. However, as I said before, Q was not designed as a file mover and certainly not ones in the GB range.

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
thorstenhirsch
PostPosted: Sat Feb 14, 2015 9:16 am    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

Yay!
Back to top
View user's profile Send private message
zpat
PostPosted: Sat Feb 14, 2015 9:49 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

Great - I hope that this will mean bug fixes (perish the thought) and small enhancements will be made to these excellent programs (by the original author!)?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
zpat
PostPosted: Sat Feb 14, 2015 9:54 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

thorstenhirsch wrote:
One of my colleagues is checking out MQ FTE. He wanted to analyze the requirements in a decentralized scenario... well, he also needs to take a look on huge files, so yes, that might be a solution for my problem as well.


Check out Capitalware's UFM as well.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
thorstenhirsch
PostPosted: Tue Feb 17, 2015 3:12 pm    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

Alright, here's an update: huge files cannot be PUT into a Queue in any application that just takes the file and calls the MQ API. Unfortunately I guess most "mq file-to-queue" applications work that way: q does, a colleague of mine wrote a similar C application that does, and even the fully-featured C++ adapter application that is in use in the integration platform at my workplace works that way.

The reason for the limit is the BufferLength in the MQ API:
Code:
MQLONG  BufferLength

MQLONG is a 4 Bytes signed integer on 32bit and 64bit platforms, so the limit is 2^31, which is 2GB. But that's not the end of the story. You can circumvent that limit when you segment the message yourself, you just have to...

1.) split the file yourself into segments of 100MB
2.) set the corresponding fields in the MQMD header:
- segmented
- offset and length (so that the GET knows how to join the segments)
- flag the last segment with "last"
- and some id so that MQ knows the segments belong together
3.) call PUT for each segment

You can do that with rfhutil (manually), it lets you set the MQMD fields.

Now you are not bound anymore to 20x 100MB segments on the wire. You can PUT as many 100MB segments as you want. Sounds great? Unfortunately it isn't, because you won't be able to GET the (complete) message back from MQ anymore. Have a look at the GET call:

Code:
void      MQGET(MQHCONN Hconn,
                MQHOBJ Hobj,
                PMQVOID pMsgDesc,
                PMQVOID pGetMsgOpts,
                MQLONG BufferLength,
                PMQVOID pBuffer,
                PMQLONG pDataLength,
                PMQLONG pCompCode,
                PMQLONG pReason);

There's that damn MQLONG BufferLength again. Yes, MQ wants to know the length of the Buffer you provide and truncate your message to that length. I guess all applications I want to reach via MQ use this GET call, so they're all bound to the 2GB limit. Our neat trick to PUT huge files to MQ is of no use.

Now that's where the story ends ...at least I think so. Please prove me wrong.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Tue Feb 17, 2015 4:37 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

As has been mentioned many times. Putting messages which are 100MB is a bad idea....so putting messages which are greater than 2GB would be an extremely bad idea. If you need to send a 5GB file then break it up into 'small' pieces. Exactly what small is may depend on a number of factors but personally I'd be looking at <= 10MB.

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
RogerLacroix
PostPosted: Tue Feb 17, 2015 4:52 pm    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

thorstenhirsch wrote:
Now that's where the story ends ...at least I think so. Please prove me wrong.

Yes and no. Sometimes you are too close to the problem and you can't see the forest through the trees.

Which is better, have your application use TCP or MQ to send a message? Hands down MQ.

Which is better to send a file? MQ or a MFT (Managed File Transfer). Hands down MFT.

Now under the covers, if the MFT uses MQ great but if the file moves from point A to point B, securely, arrives intact and is managed by the MFT who really cares if it is by MQ or carrier pigeons.

As zpat mentioned, you should have a look at Universal File Mover (UFM) which is a free open source project. You use the UFM's MQSend action on 1 server to send a file (any size) and on another server use UFM's MQReceive action to retrieve the file.

My point is that you want to deal with files rather than messages.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
thorstenhirsch
PostPosted: Wed Feb 18, 2015 9:52 am    Post subject: Reply with quote

Newbie

Joined: 13 Feb 2015
Posts: 8

@Paul: I don't get your point. When calling PUT the file is being split by MQ into chunks of MAXMSGSIZE automatically. So why should my app care about the constraints of MQ if MQ does so itself? If 10MB is a best practice, I will set MAXMSGSIZE to 10MB, so that my 2 GB file will show up as 200x 10MB messages in MQ, that's just as fine for me. But in that case 500x 10MB would be as fine, too, but you say that 5GB is not okay. Why?

Is you point that the MQ Server is not good at handling huge amounts of data, no matter what size the segments have? Sorry, I don't want to sound rude, it's just that I prefer technical explanations in favor of a gut feeling. So "MQLONG is a 4 byte signed integer" is a much better argument than "xxx is a bad idea".

@Roger: UFM is next on my list. I downloaded it last week, but wanted to finish MQ first.

I think you're both right, I will need 2 different technologies. MQ will stay there for authentication, metadata and transaction handling, but we will need a 2nd technology for the "heavy load".
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » IBM MQ API Support » piping data to MQ / segemented PUT call
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.