| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| setting shmmax manually |
« View previous topic :: View next topic » |
| Author |
Message
|
| jcv |
 Posted: Mon Dec 14, 2015 9:47 am Post subject: setting shmmax manually |
 |
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
According to this: http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSFKSJ_8.0.0/com.ibm.mq.ins.doc/q008550_.htm
| Quote: |
| If you intend to run more than two queue managers, or the queue managers are to process a significant workload, you might need to increase the values displayed as Yes in the Increase column. |
I read this that values displayed as No, you might not need to increase, even under described circumstances. That is, those are recommended "minimum tunable values that need not be increased". As oxymoronic as it seems, those are optimum values then. However, in older versions, there is no such distinction between Increase No and Yes:
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/hr/SSFKSJ_7.5.0/com.ibm.mq.ins.doc/q008550_.htm
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/hr/SSFKSJ_7.1.0/com.ibm.mq.doc/zi00760_.htm
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/hr/SSFKSJ_7.0.1/com.ibm.mq.amq1ac.doc/lq10410_.htm
The boldest recommendation given here by IBM with respect to increasing these values, describes their vision of the meaning of the phrase "minimum value":
| Quote: |
| If any of the values is less than the minimum value, you need to increase it to at least the minimum value. |
I have never ever read more useful documentation than this in my life. I'm writing this because a colleague of mine, WAS administrator, found we should better drop any manually set shmmax limit, and leave it be automatically tuned on SLES10, based on this info: https://www.novell.com/support/kb/doc.php?id=3664772
| Quote: |
Note:
Starting with SLES10 the values for shmall and shmmax are already set to what look like very high values. These values are now set to the maximum for the system by the kernel automatically, so there is no need to change them manually anymore. |
The investigation was motivated by the inability of 32 bit java application to connect via shared memory to a qmgr, accompanied by FDCs very similar to that described here: http://www-01.ibm.com/support/docview.wss?uid=swg21369950 with major (only) difference in Comment1 and the beginning of MQM Function Stack.
It was in our case:
| Comment1 :- Address space cannot accommodate this request: |
| shmat(ShmId 0x00048008) [rc=-1 errno=12] Cannot allocate memory |
and
MQM Function Stack
Java_com_ibm_mq_jmqi_local_internal_base_Native_MQCTL
MQCTL
zstMQCTL
zstMQCTLCommon
ziiQMgrInstruction
xcsInitialize
xcsConnectSharedSubpool
xcsAttachSharedSubpool
xcsConnectSharedMemSet
xstAttachSharedMemSet
xstConnectExtent
xcsFFST
I don't know how important that difference is, but none of the suggestions given in that APAR mentions increasing shmmax. Note also that the comment given in FDC does not suggest the possible cause of shmat error, for example did we unneccessary hit some software limit, such as perhaps shmmax too low (we kind of ruled out shmmni and shmall as possible cause), or were we (highly unlikely) physically out of memory at that moment? The reason for this is described here, https://www.cl.cam.ac.uk/cgi-bin/manpage?2+shmat , shmat simply does not return anything more specific than ENOMEM, so nothing more specific can be reported in FDC. Any thoughts on this? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Dec 14, 2015 10:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
One thing to double check is that the JVM didn't simply run out of memory itself - a 32-bit JVM has only so much it can use.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Dec 14, 2015 1:03 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
Your process was probably virtual memory constrained, as mqjeff is mentioning. A 32 bit process would normally have a max limit of 2 gig of virtual memory. You could use sar to check if the system shows any evidence of physical memory constraint.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| jcv |
| Posted: Mon Dec 14, 2015 8:07 pm Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
| OK, one point clear, and that is with respect to reducing the heap size of 32 bit JVM, that would leave more space for its shared memory. I said highly unlikely there, because we have 10GB RAM on the system, forgetting that 32 bit process cannot address more than 2GB, so extra GB does not help. The other point is still a bit unclear to me, if we allow bigger shared memory segment, would it actually increase or decrease probability (or none of this) of such shmat error at the same load? Thanks to both of you, Tim and Jeff. |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Dec 15, 2015 11:20 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
This support link that you mentioned -> http://www-01.ibm.com/support/docview.wss?uid=swg21369950
does have this comment, which I would agree with.
"Examine the Java process to determine why it has requested 32-bit address space requirements. The Java heap could be configured in such a way that it does not leave very much 32-bit storage for other data."
Basically, what it is saying is that if your Java heap is needlessly taking up too much virtual storage, you could look at tuning it so there is more virtual storage to be used for things like shared memory.
Virtual memory and physical memory are very different. Application processes use virtual memory and get a virtual memory address space to work within (i.e. 2GB for a 32 bit process), but the virtual memory addresses are not real addresses. They are just indexes into page tables that then point to real addresses. If your application is using a virtual memory address like 0x41C92030, that is really just an arbitrary number that indexes into the page tables for that process address space, and could point to a physical address like 0x0000000218711A00. So your application is really using physical address 0x0000000218711A00 (which is higher than 2GB).
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| jcv |
| Posted: Thu Dec 17, 2015 1:54 am Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
I also think it's all about memory consumption by java processes, not about shm limits. The concrete figures are like this:
| Code: |
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18 0 2702m 2.1g 25m S 8 22.2 746:01.76 java
15 0 2722m 2.1g 25m S 6 22.6 626:18.82 java
18 0 2641m 2.1g 24m S 4 22.5 641:42.43 java
System V Shared Memory
shmmax 268435456 bytes IBM>=268435456 PASS
shmmni 32 of 4096 sets (0%) IBM>=4096 PASS
shmall 20705 of 268435200 pages (0%) IBM>=2097152 PASS |
I added Code tags here because I don't know how else to format it in monospace manner. Anyway, having 4kb PAGESIZE, our overall shared memory consumption is only cca 80MB, (20705*4kb), and I don't see how would such consumption be possibly constrained in any way by shmmax which is 256MB. But, I also must admit that I don't have any experience in system programming with shared memory, and therefore not much intuition for administrating shared memory, ie I kind of depend on precise instructions what parameters to set, ie to tell Linux administrators to set. On the other hand, I see java application server processes cca roughly (also not my territory) on the maximum of what they can allocate, that's why I draw the same conclusion. If I said something funny, feel free to jump in at any time. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 17, 2015 5:55 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
There is one thing I am quite confused about here. If your OS is 64 bit, why are you using a 32 bit JVM instead of a 64 bit one? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jcv |
| Posted: Thu Dec 17, 2015 6:29 am Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
| Hi, our OS is 32 bit. How exactly did you come to conclusion it's 64 bit one? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 17, 2015 6:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| jcv wrote: |
| Hi, our OS is 32 bit. How exactly did you come to conclusion it's 64 bit one? |
Because any modern and supported OS should be 64-bit these days.
I shudder to think what OS you are running that is still 64-bit, and worse on what hardware.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| jcv |
| Posted: Thu Dec 17, 2015 7:16 am Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
| mqjeff wrote: |
| jcv wrote: |
| Hi, our OS is 32 bit. How exactly did you come to conclusion it's 64 bit one? |
Because any modern and supported OS should be 64-bit these days.
I shudder to think what OS you are running that is still 64-bit, and worse on what hardware. |
First of all, it's still 32 bit, and second of all, thank you once again for your kind post. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 17, 2015 7:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Sure, I get that it's still 32-bit.
I was just making a point about how FJ could have decided it was 64-bit.
As a guess, not necessarily based in fact, I'd think that increasing the shmax will only change the tradeoffs you can make between JVM heap and shm segments.
With a 2gb limit for a single 32-bit process, you should look as much at tuning the GC times to ensure that your JVM heap is used efficiently, so you can find a lower limit that also gives you enough room for the shm segments you need.
And strongly consider running multiple separate processes on the same machine...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| jcv |
| Posted: Thu Dec 17, 2015 10:05 am Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
OK, I was just not sure if I specifically led FJ to that conclusion by something I wrote. Not just as a response to that incident, we have prepared new servers with 64 bit OS, qmgr and JVM. That should at least alleviate the problem. Plus, we prepared configuration with TCP instead of shm binding. That will totaly bypass the problem. But, being significant infrastructural changes, this cannot be rolled out immediately to production. As for the current situation, I got all points with respect to heap sizing, garbage collecting and having multiple server processes. As for the shmmax, it's not like I'm incapable of finding such explanations:
http://www.pythian.com/blog/the-mysterious-world-of-shmmax-and-shmall/
or different recommendations for its setting for DB2, Oracle, you name it. But after all that, I'm still not quite sure how relevant that parameter was for that specific incident, nor can I say that the recommendation for its setting for MQ server is especially clear. |
|
| Back to top |
|
|
| jcv |
| Posted: Thu Dec 17, 2015 10:10 am Post subject: |
|
|
Chevalier
Joined: 07 May 2007
Posts: 411
Location: Zagreb
|
| ... I mean, the only way to find its relevance is to experiment a bit with that setting. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 17, 2015 11:00 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I really wasn't trying to state, imply, nor had I inferred that you hadn't or couldn't do your own research or otherwise didn't know what you were doing...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Dec 17, 2015 11:48 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
| jcv wrote: |
I also think it's all about memory consumption by java processes, not about shm limits. The concrete figures are like this:
| Code: |
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18 0 2702m 2.1g 25m S 8 22.2 746:01.76 java
15 0 2722m 2.1g 25m S 6 22.6 626:18.82 java
18 0 2641m 2.1g 24m S 4 22.5 641:42.43 java
System V Shared Memory
shmmax 268435456 bytes IBM>=268435456 PASS
shmmni 32 of 4096 sets (0%) IBM>=4096 PASS
shmall 20705 of 268435200 pages (0%) IBM>=2097152 PASS |
I added Code tags here because I don't know how else to format it in monospace manner. Anyway, having 4kb PAGESIZE, our overall shared memory consumption is only cca 80MB, (20705*4kb), and I don't see how would such consumption be possibly constrained in any way by shmmax which is 256MB. But, I also must admit that I don't have any experience in system programming with shared memory, and therefore not much intuition for administrating shared memory, ie I kind of depend on precise instructions what parameters to set, ie to tell Linux administrators to set. On the other hand, I see java application server processes cca roughly (also not my territory) on the maximum of what they can allocate, that's why I draw the same conclusion. If I said something funny, feel free to jump in at any time. |
Your java processes have more than 2 Gb for the virtual memory. I did some tests on a Linux (v3 kernel and 64 bit) and Solaris (5.10 and 64 bit) server, where I had a 32 bit C program malloc 4,096 bytes until it ran out of virtual memory. On the Linux example, I got up to almost 4 Gb. On the Solaris example, it got up to almost 2 Gb. So you might want to validate what is the virtual memory limit for a 32 bit process on your 32 bit server. Possibly, it is 3 Gb.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
