| Author |
Message
|
| Gideon |
 Posted: Wed Feb 13, 2013 2:07 pm Post subject: Broker apparently causing unkown disk activity |
 |
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
II am using WMB 8.0.0.1 on WMQ 7.5 on RHEL
I have situation where I have no actvity, no messages coming into queues (on WMQ), or leaving WMB on WMQ, but I am getting almost 100 percent disk activity for about 5 minutes maybe longer.
| Code: |
Tasks: 848 total, 1 running, 769 sleeping, 0 stopped, 78 zombie
Cpu(s): 0.3%us, 1.4%sy, 0.0%ni, 93.2%id, 5.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 99036800k total, 98740804k used, 295996k free, 1548k buffers
Swap: 68005880k total, 67861704k used, 144176k free, 172020k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
338 root 20 0 0 0 0 D 20.2 0.0 4:37.10 kswapd0
339 root 20 0 0 0 0 D 18.2 0.0 6:28.89 kswapd1
3428 mqm 20 0 12.3g 3.8g 80m S 1.0 4.0 29:12.82 DataFlowEngine
3434 mqm 20 0 12.3g 3.8g 80m S 1.0 4.0 28:46.84 DataFlowEngine
3442 mqm 20 0 12.3g 3.8g 80m S 1.0 4.0 28:56.65 DataFlowEngine
3443 mqm 20 0 12.3g 3.8g 81m S 1.0 4.0 28:34.57 DataFlowEngine |
So the swap is engaging because of the disk writes. I have no apps on the machine other than WMB and WMQ. I usually do not see this problem, but it comes up every now and then and I can not prove it but it appears to be WMB related (often happens after activity hits the WMB, but after all the incoming and outgoing queues have become enplty)
Any idea what is happening. Seems like some sort of residual activity from the flows themselves (as impossible as that may seem)
Notice I am getting about 7 percent CPU. On a system that does not have any incoming messages.
Any ideas on how I should resolve? |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Wed Feb 13, 2013 3:13 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
3443 mqm 20 0 12.3g 3.8g 81m S 1.0 4.0 28:34.57 DataFlowEngine
You are running broker as mqm service Id which is causing this problem.
http://www.mqseries.net/phpBB2/viewtopic.php?t=61512
How many times must you be told not to use the mqm id? Also, you should sudo to the broker Id before issuing any mqsi commands. Don't issue commands from your local Id.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 14, 2013 4:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You assert that it has no incoming messages, and you assert that you have verified that there are no messages in process on your queue manager.
Which unfortunately means that you've eliminated exactly one type of Input node from Broker as being the source of the problem.
There could also very easily be a message flow deployed to broker that has a TimerNotification node that runs every 24 hours or whatever, and does a lot of stuff.
This would not require any "input".
You could also simply have a message flow that's stuck. Next time you see the problem, try enabling a user-level trace of the relevant EG.
And, for once, listen to lancelotlinc and don't use mqm to launch your broker. Unless you're using MI and having the queue manager control the Broker as a service. |
|
| Back to top |
|
|
| adubya |
| Posted: Thu Feb 14, 2013 6:27 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| lancelotlinc wrote: |
3443 mqm 20 0 12.3g 3.8g 81m S 1.0 4.0 28:34.57 DataFlowEngine
You are running broker as mqm service Id which is causing this problem.
http://www.mqseries.net/phpBB2/viewtopic.php?t=61512
How many times must you be told not to use the mqm id? Also, you should sudo to the broker Id before issuing any mqsi commands. Don't issue commands from your local Id. |
If people aren't supposed to use mqm then I'm surprised IBM don't explicitly spell that out in the Infocenter docs (if they do then I've missed it).
FWIW we run our production Bkr as mqm. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 14, 2013 6:47 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| adubya wrote: |
| lancelotlinc wrote: |
3443 mqm 20 0 12.3g 3.8g 81m S 1.0 4.0 28:34.57 DataFlowEngine
You are running broker as mqm service Id which is causing this problem.
http://www.mqseries.net/phpBB2/viewtopic.php?t=61512
How many times must you be told not to use the mqm id? Also, you should sudo to the broker Id before issuing any mqsi commands. Don't issue commands from your local Id. |
If people aren't supposed to use mqm then I'm surprised IBM don't explicitly spell that out in the Infocenter docs (if they do then I've missed it).
FWIW we run our production Bkr as mqm. |
IBM does document this 'feature':
| Code: |
| A semaphore is created when the first broker starts after a reboot (or after an initial installation), and only members of the primary group of the semaphore creator can access this semaphore. This problem is a consequence of the UNIX System V IPC primitives that are used by WebSphere Message Broker. Avoid this problem by ensuring that all user IDs used to start WebSphere Message Broker have the same primary group. |
Since the primary group of mqm Id does not equal the primary group of mqbrkrs Id, the semaphore ownership gets confused. Also, this is why you must sudo into the Broker service Id before issuing mqsi commands since your local Id's primary group is likely not mqbrkrs.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 14, 2013 7:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
IBM does document this 'feature':
| Code: |
| A semaphore is created when the first broker starts after a reboot (or after an initial installation), and only members of the primary group of the semaphore creator can access this semaphore. This problem is a consequence of the UNIX System V IPC primitives that are used by WebSphere Message Broker. Avoid this problem by ensuring that all user IDs used to start WebSphere Message Broker have the same primary group. |
Since the primary group of mqm Id does not equal the primary group of mqbrkrs Id, the semaphore ownership gets confused. Also, this is why you must sudo into the Broker service Id before issuing mqsi commands since your local Id's primary group is likely not mqbrkrs. |
For those who want to look it up themselves, and view the context in which this comment is made, you'll find it here.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 14, 2013 7:11 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Also, it is possible to use mqm Id, if it is used consistently throughout (as the semaphore owner). The OS doesn't care that the owner of the semaphore is not mqbrkrs, however, the owner and all accessors of the semaphore (ie those that run commands against the broker runtime) must all be consistent with what PRIMARY GROUP ID they are a member. In other words, if mqm owns the semaphore and Joe User with pgid of Admin group issues an mqsi command, then the semaphore gets jumbled and you end up with processes that fight over who can access what broker resources, chewing up memory, and having alot of page faults into the virtual memory pool.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Gideon |
| Posted: Thu Feb 14, 2013 10:52 am Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
I have created a userID named brk and am testing the id right now.
I have 2 questions
--- Using the mqm ID ---
In the previous config, I used mqm (I did not have a mqbrkrs ID), as follows
| Code: |
[brk@SandyBridge2 ~]$ groups mqm
mqm : mqm mqbrkrs |
I do not use a sudo, but I always login to mqm as follows to perform all mq and mqsi tasks
Since I did all mq and mqsi commands from this ID, as I read this note, this should have worked since the primary gid (mqm) would have always been the same. Is taht correct ?
--- Using the brk ID ---
Now that I am using the following config,
| Code: |
[root@SandyBridge2 mqm]# su - brk
[brk@SandyBridge2 ~]$
[brk@SandyBridge2 ~]$ groups brk
brk : mqbrkrs mqm
[brk@SandyBridge2 ~]$ |
I tried starting up a 'trusted' qmgr, but it would not start since mqm was not the primary gid. How can I get around that if I create a user ID named brk separate from mqm ?
Thanks |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 14, 2013 10:56 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Please post the error message from syslog that shows the reason why the Broker would not start. /var/adm/messages unless you changed the default for syslog.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Last edited by lancelotlinc on Thu Feb 14, 2013 11:05 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 14, 2013 11:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| Please post the error message from syslog that shows the reason why the Broker would not start. /vad/adm/messages unless you changed the default for syslog. |
It's not starting because it's not running as the mqm user, because he or she has built it to use a TRUSTED connection to the queue manager, i.e. IPC rather than normal bindings.
Brokers using TRUSTED connections to the queue manager MUST run as mqm. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 14, 2013 11:07 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Correct mqjeff. Nevermind about the user id then. You'll just have to live with DataFlowEngines that generate needless page file use until you can get an APAR from IBM that fixes the semaphore issue. You might also consider a nightly reboot. Unless you want to go back to classic MQ access (ie. not trusted).
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Gideon |
| Posted: Thu Feb 14, 2013 11:14 am Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
Yes, I totally understand. The broker was created as trusted, and since I am now running with mqbrkrs as the primary group, it can not run as trusted (because mqm must be primary for a trusted broker).
My question is that lancelotlinc suggested I use the mqbrkrs as the primary, is there a way to use a trusted broker with mqbrkrs as the primary
| Code: |
groups brk
brk : mqbrkrs mqm |
|
|
| Back to top |
|
|
| Gideon |
| Posted: Thu Feb 14, 2013 11:16 am Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
lancelotlinc, I am sorry, just read your post right after I posted mine
k, I will try this with brk and see how it works |
|
| Back to top |
|
|
| Gideon |
| Posted: Thu Feb 14, 2013 11:19 am Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
However, since I previously used mqm (which has mqm as the primary group, and mqbrkrs as the secondary group, why would this have caused a problem
Since I always di work in both mq and mqsi with the following
su - mqm
Why would this have caused a problem since I always used the same primary group ?
Thanks |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 14, 2013 11:22 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
The UNIX System V IPC engine is very old and needs rework. Unfortunately, this is the service implementation that is available on Linux platforms. Special custom code is needed from IBM to work around the IPC issue on Linux platforms. You could switch to AIX for a more advanced IPC implementation.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
