| Author |
Message
|
| pal |
 Posted: Wed Aug 15, 2001 11:49 am Post subject: |
 |
|
Apprentice
Joined: 14 Aug 2001
Posts: 35
|
Hi
This is my first post ... must say this seems like a nice place for mq nuts 
OK, here is my problem. My mq java program on HP-UX 11 is writing 50000 messages in transactions of 500 each in a for loop. Each message is approximately 2-3K. After about 20000 or so the MQ Queue stops showing any increase in the current queue depth ... I increased the circular log file size etc and it still stops at around the same number.
The current log file config is:
LogPrimaryFiles=12
LogSecondaryFiles=6
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=30
which seems more than enough for the task.
Any ideas? Thanks a lot. |
|
| Back to top |
|
 |
| bduncan |
| Posted: Wed Aug 15, 2001 12:31 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Pal-
My first question is what did you have it set to originally?
My second question - and possible answer - is did you rebuild or restart the queue manager after increasing the logs? The number of primaries and secondaries (and the size of each one) is set at the time the queue manager is created. The only thing that you can change once the queue manager has been built is the number of secondaries. The number of primaries and the size (in Kb) of both types of logs are fixed when the queue manager is created. Furthermore, when you increase the number of secondaries, you must stop and restart the queue manager before it will take effect. If you want to increase the primaries or the size of the logs you need to alter the mqs.ini in /var/mqm/ (or pass the appropriate parameters through crtmqm) and build a new queue manager.
Another thing you want to keep in mind is that if you decide to simply increase the number of secondaries (because maybe rebuilding the queue manager is too much trouble) that the number of secondaries you can specify has an upper limit of 10 if I'm not mistaken (correct me if I'm wrong). But I do know there is an upper limit. So if you really want to make sure that you don't have problems in the future, I would recommend rebuilding the queue manager...
Now, if you actually did rebuild the queue manager and you are still having these problems, then forget everything I just said 
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| pal |
| Posted: Wed Aug 15, 2001 1:00 pm Post subject: |
|
|
Apprentice
Joined: 14 Aug 2001
Posts: 35
|
Brandon
I did rebuild the queue manager with the new file sizes. The ones earlier were the defaults (primaries-3, size-1024, secondaries-2) and the queue was still registering about the same number of messages.
Also, the file system allows file sizes to go upto 2G's so that should not be a problem; nevertheless I have requested the admin to enable large file sizes just in case.
Any help to resolve this would be great. Thanks. |
|
| Back to top |
|
|
| bduncan |
| Posted: Thu Aug 16, 2001 8:03 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Pal,
Are you using syncpoint for this operation? And more to the point, if so, are you trying to put all 50,000 messages under a single unit of work? If so, that could definitely be a possible cause of the problem.
Another thing. In your originial post you say that the queue depth just stops going up. You never mentioned any error codes or alerts. Are there no errors in the /var/mqm/qmgrs/qmgrname/errors/ log files? If MQSeries is indeed running out of disk space it would definitely be logging the fact... If there was output or logs of any sort to indicate an error it would be helpful if you could provide those, because it will probably explain what's going on...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| pal |
| Posted: Fri Aug 17, 2001 7:11 am Post subject: |
|
|
Apprentice
Joined: 14 Aug 2001
Posts: 35
|
Brandon
I am using syncpoint but only 50 messages per syncpoint (1000 such syncpoints in a loop) and also tried 500 messages/syncpoint (100 such in a loop) for a total attempt to send 50000 messages.
I increased the space for the /var/mqm partition to 1.5G and changed the QM config to 8 primaries, 4 secondaries with the same as before 16+ for file size and this time it stopped at 37 000 for persistent and less for non persistent.
No meaningful log messages ... |
|
| Back to top |
|
|
| royr |
| Posted: Sun Aug 19, 2001 1:49 am Post subject: |
|
|
Acolyte
Joined: 30 Jun 2001
Posts: 65
Location: Israel
|
Changing the number of primary or secondary log files only requires a queue manager restart. Changing the size of the files requires a full rebuild. Also, the number of secondary files isn't limited by itself, but the total number of primary+secondary log files should not exceed 63. See this section of MQSeries System Administration. If you increase the number of primary files, the queue manager will create them at the next strmqm.
The log secondary files are created on demand at runtime. This involves a long I/O peak (the files are created and formatted at their full size) and could cause the queue manager to suspend further processing of messages until the files have been fully formatted. This could take up to several minutes, depending on the file size and disk performance. Did you give your application (and the queue manager) enough time to complete?
BTW, Pat, have you tried running your program without syncpoint - in other words, are you sure the problem is related to the log mechanism?
|
|
| Back to top |
|
|
| bduncan |
| Posted: Sun Aug 19, 2001 10:04 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
royr,
I read your response and looked at the link you provided. Apparently the number of primaries can be extended with a restart without having to resort to rebuilding the queue manager like I suggested, but I was wondering, has this changed from previous verions of MQSeries? Because I sure that you had to rebuild the queue manager to extend the number of primaries... Perhaps this was changed in 5.1 or 5.2?
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| woolies |
| Posted: Sun Aug 19, 2001 4:32 pm Post subject: |
|
|
Newbie
Joined: 18 Aug 2001
Posts: 1
|
Hi all,
We are experiencing a similar problem here,
i have a need to increase the number of logs to cater for larger file transfers.
Currently we have 5 Primary and 5 secondary, with 4 x 256KB pages, giving us a maximum file transfer size of 10mb.
We are running MQ 5.2 on an NT system.
I have tried changing the number of Primary and secondary log files to 25 and 25 (totalling 50, as the System Admin manual suggests that the logs be less than 63 in total) in the MMC but no additional log files are created. We have tried stopping the QMGR and then making the req'd changes and then starting the QMGR but no effect.
Is there a bug i am not aware of, and has anyone ever got this to work? How does everyone change the number of log files on their NT MQ 5.2 systems?
It is not feasible for me to re-create the QMGR as this change has to be pushed out to over 700 sites.
Your help will be highly appreciated. |
|
| Back to top |
|
|
| emileke |
| Posted: Mon Aug 20, 2001 1:50 am Post subject: |
|
|
Centurion
Joined: 19 Aug 2001
Posts: 110
Location: South Africa
|
You might be hitting the DefaultQFileSize Limit on MQseries.
Undocumented feature of MQ.
Look at the QM.INI and the limit varies per platform.
TuningParameters:
DefaultQFileSize =1000000000
|
|
| Back to top |
|
|
| pal |
| Posted: Mon Aug 20, 2001 8:12 am Post subject: |
|
|
Apprentice
Joined: 14 Aug 2001
Posts: 35
|
I don't see this default queue size param in either the global mqs.ini file nor the qm.ini file for each of the queue managers. This may very well explain why increasing the size is not affecting number of messages.
Firstly, what does it mean if this param is not present anywhere?
And secondly, which file does this go into (the global mqs.ini or the specific to a QMgr qm.ini)? Can I put it in there if it was not there originally?
Thanks. |
|
| Back to top |
|
|
| royr |
| Posted: Tue Aug 21, 2001 6:42 am Post subject: |
|
|
Acolyte
Joined: 30 Jun 2001
Posts: 65
Location: Israel
|
There isn't any mqs.ini file on NT.
The settings are in the registery, under "MyComputerHKEY_LOCAL_MACHINESOFTWAREIBM MQSeriesCurrentVersionConfiguration".
Setting that don't appear get their default values.
Where exactly have you changed the number of primary and secondary files? This should be done in the MQSeries Services window. Right-click the queue manager, select properties and then the Log tab in the window that appears.
Make sure you perform a successful queue manager restart after changing the values.
You can find out if the change took effect after restarting by checking the number of log files in the MQSerieslogQMGRNAMEactive directory.
[ This Message was edited by: royr on 2001-08-21 07:43 ] |
|
| Back to top |
|
|
| emileke |
| Posted: Tue Aug 21, 2001 7:05 am Post subject: |
|
|
Centurion
Joined: 19 Aug 2001
Posts: 110
Location: South Africa
|
Add it to QMGR qm.ini file.
Recycle QMGR and redefine the Q and try again. |
|
| Back to top |
|
|
| pal |
| Posted: Wed Aug 22, 2001 9:20 am Post subject: |
|
|
Apprentice
Joined: 14 Aug 2001
Posts: 35
|
This is what I did:
I created a new queue manager with the following params:
Log:
LogPrimaryFiles=8
LogSecondaryFiles=4
LogFilePages=8192
LogType=CIRCULAR
LogBufferPages=17
And then I added the follwoing lines to the qm.ini file for that queue manager:
TuningParameters:
DefaultQFileSize =1000000000
Still stops at around the same number ...
I do thank you all for the help so far ... and I am willing to try anything else that even has a remote chance of solving this.
[ This Message was edited by: pal on 2001-08-22 10:20 ] |
|
| Back to top |
|
|
| emileke |
| Posted: Thu Aug 23, 2001 3:23 am Post subject: |
|
|
Centurion
Joined: 19 Aug 2001
Posts: 110
Location: South Africa
|
The queue definition is stored on DASD and is used to allocate resources at MQOpen. Queue definitions may have been created with various values used for DefaultQFileSize and DefaultQBufferSize. Customers who want two
queues to have large values and the rest to take defaults, for example will start the queue manager with the specified tuning parameters, create the two queues (definitions stored on disk), and terminate the queue manager
|
|
| Back to top |
|
|
| emileke |
| Posted: Thu Aug 23, 2001 3:30 am Post subject: |
|
|
Centurion
Joined: 19 Aug 2001
Posts: 110
Location: South Africa
|
Maximum amount of bytes in a queue
The maximum size of each queue is 320MB on DASD. This can be set to values up to 1GB for an individual queue by using DefaultQFileSize (see following example). Between 95KB and 195KB of preallocated shared storage is used to hold tables, which is obtained when the queue is opened. (OS/2 Warp needs 95KB per queue, Windows NT 195KB; UNIX requirements are between these two values). When the DefaultQFileSize is set to 1GB, the preallocated shared storage grows to between 125KB and 350KB.
Maximum amount of memory (RAM) reserved for non-persistent messages
The amount of shared memory (that is, RAM) used to hold non-persistent messages is 64KB per queue. This can be raised to 1MB by using the DefaultQBufferSize (see following example). This amount of shared storage is allocated, (together with other control blocks) when the queue is opened and so has a direct impact on the amount of real resources needed by the queue manager. It has a significant effect on both virtual shared memory and real memory. A more realistic buffer size would be less than 1MB and that would depend on the usage of the queue. An iterative method would be to double the 64KB default size of the buffer to 128KB and observe the effect on response time. If messages are no longer spilt to disk, they will be removed from the queue more quickly and hence have an additional effect on the queue depth. Persistent messages are held in the file system(disk) only, while non-persistent are initially held in the buffer but subsequent messages spilt out to disk when the buffer fills. The effect is enhanced performance at the expense of storage, but it may degrade the performance of the previously balanced system. If this increase in storage usage pushes the system into paging, the performance will get worse because page faults are more expensive than having MQSeries intelligently manage the disk I/O for non-persistent messages.
Opening Queues
The queue definition is stored on DASD and is used to allocate resources at MQOpen. Queue definitions may have been created with various values used for DefaultQFileSize and DefaultQBufferSize. Customers who want two queues to have large values and the rest to take defaults, for example will start the queue manager with the specified tuning parameters, create the two queues (definitions stored on disk), and terminate the queue manager. The queue manager is then started without the TuningParameter stanza, and the two queues previously defined with large values keep their large values. Any subsequent queue definitions use the normal defaults. As each queue is opened, the necessary resources are allocated depending on the DASD definition of the queue
|
|
| Back to top |
|
|
|
|
