| Author |
Message
|
| Dolapevich |
 Posted: Tue Jun 04, 2013 2:58 am Post subject: qload limits |
 |
|
Novice
Joined: 29 Oct 2012
Posts: 11
Location: Buenos Aires, Argentina
|
I hit upon a qload on limit while trying to import messages from 2 Gigabytes files (Linux X86_64 RedHat 5.
./qload -m QMGR1 -o QUEUE1 -f /path/to/big/message/file
QLOAD Program by Paul Clarke [ V1.4 Build:Mar 29 2007 ]
Can not open input file /path/to/big/message/file'
I straced it and saw:
open("/path/to/big/message/file", O_RDONLY) = -1 EFBIG (File too large)
write(2, "Can not open input file '/path/to/b"..., 59) = 59
My solution was to
mkfifo ifile
./qload -m QMGR1 -o QUEUE1 -f ifile
cat /path/to/big/message/file > ifile
My questions:
Is this a known limit I was not aware?
-I couldn't find qload source. Is it available somewhere to fix it?
-I see my binary is 32bits and that explains. However 64 bits is compiled against glibc 2.7 . Is there any place where I can get it compiled for 2.5 or better, source code? Nevermid, I got this thread ( www.mqseries.net/phpBB2/viewtopic.php?t=64221 )
-Is there any known simple method to split dump files in smaller ones?
Thanks in advance!
Sebastian |
|
| Back to top |
|
 |
| hughson |
| Posted: Tue Jun 04, 2013 3:23 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
QLOAD files are user-editable, you could easily split the file into two, just make sure to do it between messages and not in the middle of a message. Then load the two files up one after the other.
Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| Dolapevich |
| Posted: Tue Jun 04, 2013 10:43 am Post subject: |
|
|
Novice
Joined: 29 Oct 2012
Posts: 11
Location: Buenos Aires, Argentina
|
I noticed that. However, I was wondering if there was a tool to say, for instance, automatically split in 1000 messages or something like that.
It seems I will need to shape up my perl and regex skills 
Thanks!
Sebastian |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jun 04, 2013 1:30 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| QLOAD can remove the messages from the queue as it goes, so you could just code a simple shell script to execute QLOAD several times, incrementing a file name suffix number, and unloading 1000 messages each time into each of these files until the queue is empty. |
|
| Back to top |
|
|
| Dolapevich |
| Posted: Tue Jun 04, 2013 5:13 pm Post subject: |
|
|
Novice
Joined: 29 Oct 2012
Posts: 11
Location: Buenos Aires, Argentina
|
That is to think using the head!
Nice! Hadn't thought of that option.
Will do that for the time being.
Thanks! |
|
| Back to top |
|
|
| MaximusDecimus |
| Posted: Fri Jul 11, 2014 5:32 am Post subject: |
|
|
Newbie
Joined: 11 Jul 2014
Posts: 2
|
| hughson wrote: |
QLOAD files are user-editable, you could easily split the file into two, just make sure to do it between messages and not in the middle of a message. Then load the two files up one after the other.
Cheers
Morag |
Can you tell me how do you split the file exactly without damaging any message ? |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jul 11, 2014 7:04 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
It's pretty obvious how to separate the messages in the unloaded file - have you looked at one?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Jul 11, 2014 8:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| MaximusDecimus wrote: |
| hughson wrote: |
QLOAD files are user-editable, you could easily split the file into two, just make sure to do it between messages and not in the middle of a message. Then load the two files up one after the other.
Cheers
Morag |
Can you tell me how do you split the file exactly without damaging any message ? |
Go on copy the file and use any UTF-8 capable editor and see what happens.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jul 11, 2014 10:54 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Why UTF8? Have you looked at a qload file? 
It's plain text and very obvious which is the MQMD and which is the data.
Clue - the next message starts when the MQMD fields start!! 
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Fri Jul 11, 2014 9:46 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Have you considered using a later version of QLOAD or is there some reason you have to stick with 1.4 ? As you can see 1.4 is quite old now.
I added large file support (ie. greater than 2GB) into QLOAD a few releases ago. Although I'm afraid I can't remember which one. If you move to the latest version 1.9 there's a fair chance that the problem will just go away.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| MaximusDecimus |
| Posted: Wed Jul 16, 2014 4:21 am Post subject: |
|
|
Newbie
Joined: 11 Jul 2014
Posts: 2
|
| PaulClarke wrote: |
Have you considered using a later version of QLOAD or is there some reason you have to stick with 1.4 ? As you can see 1.4 is quite old now.
I added large file support (ie. greater than 2GB) into QLOAD a few releases ago. Although I'm afraid I can't remember which one. If you move to the latest version 1.9 there's a fair chance that the problem will just go away.
Cheers,
Paul. |
Thanks for the info Paul. The issue now I am facing is I have offloaded 10k messages from a queue and need to load them back to another queue. The file that has those 10k messages is about 2.3GB and now I am facing issue to load them back. So now I will have to split the file into two files and load them back.
How do I do it now ? |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Wed Jul 16, 2014 4:44 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
If you are using 1.9 then it should just work. Does it not ?
As you all know, when I left IBM I asked IBM if I could take QLOAD with me. However, for some reason that is not at all clear to me they refused. I would love to be able to continue to enhance and support QLOAD but at the moment IBM has not given me permission. The odd thing is that IBM seems to show no interest in continuing to support QLOAD although they have incorporated some of its features into MQ V8.
If enough customers lobbied IBM then perhaps you could get either IBM to support QLOAD fully or them to release it back to me and allow MQGem to support it. As things stand though I'm afraid my hands are tied.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jul 21, 2014 5:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
IBM has continued to update saveqmgr, even though it's now dmpmqcfg. They've recently added some of the more useful but not obvious features (the --twoLine feature).
So I suspect that as people keep using dmpmqmsg (qload), RFEs and defects and etc can cause additional qload features to be added. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jul 21, 2014 6:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mqjeff wrote: |
So I suspect that as people keep using dmpmqmsg (qload), RFEs and defects and etc can cause additional qload features to be added. |
I just wish IBM were more specific in the V8 documentation. It only specifies qload having been integrated in the product and does not link to dmpmqmsg... but the links available on that page go all to qload!... Confusing! 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Fri Jul 25, 2014 8:55 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
| I seem to be having difficulty today... Please provide a link to dmpmqmsg. I didn't even get a hit on qload when searching... |
|
| Back to top |
|
|
|
|
