Author |
Message
|
rk1891 |
Posted: Wed Oct 12, 2011 1:05 pm Post subject: Deployment request on WMB 6.1.0.2 with MQ 7.0.1 times out |
|
|
Apprentice
Joined: 12 Oct 2011 Posts: 42
|
Hi,
I'm a newbie to not only this forum but also to MQ and WMB.
We are trying to set up our new env on linux.
We've installed MQ('mqm'-user and 'mqm'- Group) and WMB('mqbrkr'- user and 'mqbrkrs' - Group).
I've checked all the authorities for 'mqbrkr'.It is also part of 'mqm' group, also has all Auths same as that of 'mqm' user for all broker related channels and queues.
We wanted to use DB2 as broker database so configured everything as per IBM info center.
Sucessfully created Test Qmanager, Test Broker and Test ConfigManager.
When we tried to deploy one simple bar file - The reqest is being timed out even though we've given 200sec wait time.
Command used-
mqsideploy -i localhost -p 1416 -q TEST -b TESTBRK -e default -a /home/mqbrkr/Deployments/Sample.bar -w 200 -v ~/deploy_verbose.log
The MQ active logs has the following entry. Couldn't find anything in IBM info center to resolve those errors
****
<Broker uuid="2d0bc6f8-3201-0000-0080-b72a7a132126" label="TESTBRK" version="1"><OverallCompletionCode result="failure"><LogEntry catalog="BIPv610" number="2087"><Insert type="string" text="TESTBRK"/><Insert type="string" text="2d0bc6f8-3201-0000-0080-b72a7a132126"/></LogEntry></OverallCompletionCode><ControlGroupCompletionCode uuid="" result="unknown"><LogEntry catalog="BIPv610" number="2066"><Insert type="string" text="TESTBRK"/><Insert type="string" text="2d0bc6f8-3201-0000-0080-b72a7a132126"/><Insert type="string" text="$SYS_mqsi"/><Insert type="string" text="360"/></LogEntry></ControlGroupCompletionCode><ControlGroup/></Broker>
****
In deployement verbose log - I saw the below..
*********
2011-10-12 11:55:40.0712 CMPNotification d[1]: Awaiting notification message...
2011-10-12 11:55:50.0417 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:00.0419 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:10.0423 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:20.0427 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:30.0442 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:34.0690 CMPObjectReaper d[3]: Polling reference queue...
2011-10-12 11:56:40.0443 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:56:50.0447 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:00.0452 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:10.0456 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:20.0460 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:30.0483 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:34.0695 CMPObjectReaper d[3]: Polling reference queue...
2011-10-12 11:57:40.0488 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:57:50.0492 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:00.0497 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:10.0501 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:20.0505 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:30.0509 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:34.0700 CMPObjectReaper d[3]: Polling reference queue...
2011-10-12 11:58:40.0513 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:58:50.0517 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-12 11:59:00.0312 main........... d[2]: Time limit exceeded.
********
I think the Confing Manager is waiting for a message on SYSTEM.BROKER.ADMIN.REPLY..but not quite sure where the process is being stuck
Any help is greatly appreciated.Thanks in Advance |
|
Back to top |
|
 |
lancelotlinc |
Posted: Wed Oct 12, 2011 1:12 pm Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Often the cause of the service Id from Broker runtime being primary group membership of mqm rather than mqbrkrs. The service Id that the Broker runtime runs under must have primary group membership in mqbrkrs and secondary group membership of mqm, not the other way around. Also, any user who executes mqsistart must do so from their sudo mqbrkrs identity, not their login id. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Oct 13, 2011 1:56 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
You shouldn't be starting with Broker 6.1.0.2.
You should be starting with Broker 6.1.0.9 at least.
You appear to have looked at many logs, but do not appear to have shown the entries from the log that will actually have useful information... At 6.1.0.9, at least, deployment requests will be logged to the Broker system log - and this will help you determine where in the process things are stuck.
My guess is that you have not completed the initial setup fully or correctly. But nothing you've posted shows any information relevant to that. |
|
Back to top |
|
 |
rk1891 |
Posted: Thu Oct 13, 2011 6:54 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011 Posts: 42
|
I believe we are at 6.1.0.9 not at 6.1.0.2 because when I run mqsiservice -v, I got 6.1.0.9 only.
I posted the value of 6.1.0.2 by looking at 'Fixpack capability level' value of 'mqsireportbroker TESTBRK'.
Not quite sure why they were different.
So now I've issued 'mqsichangebroker TESTBRK -f 6.1.0.9' to change the effective level.
Things are the same as before.I've checked the logs at /var/mqsi/components/TESTBRK
In that I've two folders with different uuids.I believe one is for $SYS_mqsi and another for my 'default' execution group.
Not sure though which one is what but in one of those 'stderr', I found the below.. (In all others, all I've is Execution group start messages)
Quote: |
2011-10-13 13:49:49.470540 Execution group started. UUID is: '82a6e590-12b3-11d3-81a2-00203506a68e'. Broker is: TESTBRK
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00002AAAB1859200 Handler2=00002AAAB1ABF0B0 InaccessibleAddress=00007FFF31C8F000
RDI=00002AAAC950AAD0 RSI=00007FFF31C8EFC4 RAX=53454D414E454C49 RBX=524F4C4F4300313D
RCX=0000000000000755 RDX=00000000FFFF8038 R8=0000000000020000 R9=2F00313D4D524554
R10=6973716D2F74706F R11=7461442F6E69622F R12=676E45776F6C4661 R13=0000000000656E69
R14=465F4E454B4F5242 R15=0000000000000380
RIP=00002AE07ADF333E GS=0000 FS=0000 RSP=00007FFF31C842E8
EFlags=0000000000010206 CS=0033 RBP=00002AAAC950004C ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000004A00 CR2=00007FFF31C8F000
xmm0 006c006f00720074 (f: 7471220.000000, d: 1.246117e-306)
xmm1 0020006c006f0072 (f: 7274610.000000, d: 4.450606e-308)
xmm2 0000000000000074 (f: 116.000000, d: 5.731161e-322)
xmm3 0073007300650063 (f: 6619235.000000, d: 1.691212e-306)
xmm4 5353595320202020 (f: 538976256.000000, d: 2.522525e+93)
xmm5 2020202030303248 (f: 808464960.000000, d: 6.013470e-154)
xmm6 2020202020202020 (f: 538976256.000000, d: 6.013470e-154)
xmm7 5353595320202020 (f: 538976256.000000, d: 2.522525e+93)
xmm8 2020202030303248 (f: 808464960.000000, d: 6.013470e-154)
xmm9 5953202020202020 (f: 538976256.000000, d: 1.975472e+122)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 00002aaaab920000 (f: 2878472192.000000, d: 2.317786e-310)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libc.so.6
Module_base_address=00002AE07AD7E000 Symbol=memcpy
Symbol_address=00002AE07ADF3060
Target=2_30_20100509_57823_LHdSMr (Linux 2.6.16.60-0.21-smp)
CPU=amd64 (16 logical CPUs) (0x1f2cdd000 RAM)
JVMDUMP006I Processing dump event "gpf", detail "" - please wait.
JVMDUMP032I JVM requested System dump using '/home/mqbrkr/Deployments/core.20111013.094951.3893.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/mqbrkr/Deployments/core.20111013.094951.3893.0001.dmp
JVMDUMP032I JVM requested Snap dump using '/home/mqbrkr/Deployments/Snap.20111013.094951.3893.0002.trc' in response to an event
JVMDUMP010I Snap dump written to /home/mqbrkr/Deployments/Snap.20111013.094951.3893.0002.trc
JVMDUMP032I JVM requested Java dump using '/home/mqbrkr/Deployments/javacore.20111013.094951.3893.0003.txt' in response to an event
JVMDUMP010I Java dump written to /home/mqbrkr/Deployments/javacore.20111013.094951.3893.0003.txt
JVMDUMP013I Processed dump event "gpf", detail "".
*** glibc detected *** DataFlowEngine: double free or corruption (!prev): 0x0000000000d3dfa0 ***
JVMDUMP006I Processing dump event "abort", detail "" - please wait.
JVMDUMP032I JVM requested System dump using '/home/mqbrkr/Deployments/core.20111013.094951.3893.0004.dmp' in response to an event |
Hope this helps..to find the cause. |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Oct 13, 2011 7:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
If you've still got anything for $SYS_MQSI that exists, then your Broker has never properly started up.
This, and the fact that the EG is core dumping, explains why you can't deploy to it...
Go back to square 0, and make sure you can start the broker successfully. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Thu Oct 13, 2011 7:14 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Chapter 6 of the Installation Guide has some wonderful recommendations in the form of steo-by-step instructions. Installing a Broker runtime is not as simple as copying files. There are specific setup steps needed, and specific groups/users created before the first file is laid down on the filesystem. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
rk1891 |
Posted: Thu Oct 13, 2011 7:25 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011 Posts: 42
|
When I run 'ps -eaf | grep bip', as per the below - I believe they are running
Quote: |
mqbrkr 3112 1 0 09:42 ? 00:00:00 bipservice testconfigMgr
mqbrkr 3117 3112 0 09:42 ? 00:00:09 bipconfigmgr testconfigMgr
mqbrkr 3852 1 0 09:49 ? 00:00:00 bipservice TESTBRK
mqbrkr 3858 3852 0 09:49 ? 00:00:00 bipbroker TESTBRK
mqbrkr 3889 3858 0 09:49 ? 00:00:01 biphttplistener TESTBRK
mqbrkr 11335 11297 0 11:12 pts/3 00:00:00 grep bip
|
and when I run 'mqsicvp TESTBRK', I got the below -
Quote: |
BIP8873I: Starting the component verification for component 'TESTBRK'.
BIP8876I: Starting the environment verification for component 'TESTBRK'.
BIP8894I: Verification passed for 'Registry'.
BIP8894I: Verification passed for 'MQSI_REGISTRY'.
BIP8894I: Verification passed for 'Java Version - 1.5.0 IBM Linux build pxa64devifx-20110209 (SR11 FP2 +IZ94331)'.
BIP8894I: Verification passed for 'MQSI_FILEPATH'.
BIP8878I: The environment verification for component 'TESTBRK' has finished successfully.
BIP8882I: Starting the WebSphere MQ verification for component 'TESTBRK'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.ADMIN.QUEUE' on queue manager 'TESTQM'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.CONTROL.QUEUE' on queue manager 'TESTQM'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.EXECUTIONGROUP.QUEUE' on queue manager 'TESTQM'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.EXECUTIONGROUP.REPLY' on queue manager 'TESTQM'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.INTERBROKER.QUEUE' on queue manager 'TESTQM'.
BIP8886I: Verification passed for queue 'SYSTEM.BROKER.MODEL.QUEUE' on queue manager 'TESTQM'.
BIP8884I: The WebSphere MQ verification for component 'TESTBRK' has finished successfully.
BIP8879I: Starting the database verification for component 'TESTBRK'.
BIP8890I: Verification passed for table 'DB2INST1.BACLENTRIES' in datasource 'TESTBKDB'.
BIP8890I: Verification passed for table 'DB2INST1.BCLIENTUSER' in datasource 'TESTBKDB'.
BIP8890I: Verification passed for table 'DB2INST1.BGROUPNAME' in datasource 'TESTBKDB'.
BIP8890I: Verification passed for table 'DB2INST1.BMQPSTOPOLOGY' in datasource 'TESTBKDB'. |
some how it stuck after the last print above in the verification..What comes next after this step?[ |
|
Back to top |
|
 |
lancelotlinc |
Posted: Thu Oct 13, 2011 7:28 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
|
Back to top |
|
 |
rk1891 |
Posted: Thu Oct 13, 2011 7:31 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011 Posts: 42
|
BTW..forgot to mention 'Thank you' to both of you for your active posts..because busy banging my head against wall..
@lancelotlinc - I've this document already but we did not install it.It was taken care by someone else..I'll ask them to verify the install against this document.. And as per your previous post - 'mqbrkrs' is not the primary group of 'mqbrkr' and same for 'mqm' also. W've requested them to change those..waiting on it..Hopefully that would resolve these issues. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Thu Oct 13, 2011 8:37 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Your welcome. I'm happy to help when able, and it is good to see an OP that takes to heart the information. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Oct 13, 2011 8:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
lancelotlinc wrote: |
Your welcome. I'm happy to help when able, and it is good to see an OP that takes to heart the information. |
It's good general advice.
I don't believe it will fix the issue. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Thu Oct 13, 2011 9:04 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
I agree mqjeff, and without actually being on that site looking and poking around in the installation, its hard to know what else was missed during the installation.
Just about every customer I have ever been to has had to reinstall their binaries because the 'consultant' who did the first install failed to follow the guidance in the Installation Guide. A very common failure is to install the binaries, then create the groups and service ids. Another common failure is to avoid the mandatory use of sudo.
Might be a good thought for V9 of WMB to put some idiot-proofing in the install process. I'm not an expert, just a pert. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Thu Oct 13, 2011 9:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
lancelotlinc wrote: |
I agree mqjeff, and without actually being on that site looking and poking around in the installation, its hard to know what else was missed during the installation. |
I agree.
It's also hard to say that anything was missed during the installation, or that the installation was in any way done incorrectly... |
|
Back to top |
|
 |
rk1891 |
Posted: Thu Oct 13, 2011 12:59 pm Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011 Posts: 42
|
We changed the primary groups of 'mqbrkr' to 'mqbrkrs' and 'mqm' to 'mqm' but still the deployments are timing out
Broker logs
/var/mqsi/common/log - All of the files has '0' size except that of 'mqsicvp.trace.bin' files.
/var/mqsi/components/TESTBRK/execution_group_uuid - In stderr,It has the same verbose which was posted earlier
/var/mqsi/common/errors - Doesn't have any files.
MQ logs
/var/mqm/qmgrs/TESTQM/errors/AMQERR01.LOG - Says everything is started
/var/mqm/log/TESTQM/active/S0000000.LOG - Has same 2087 and 2066 errors
/var/mqm/errors/AMQERR01.LOG - Nothing new written in this file
DB2 logs
/home/db2inst1/sqllib/db2dump/db2diag.log - This also doesn't have
anything that says error
The deployment verbose log has
Quote: |
2011-10-13 16:32:33.0184 CMPNotification d[1]: Sending basic actionresponse notification: listener = com.ibm.broker.config.proxy.DeployResultManager@231c231c, opType = actionresponse, parameters[0] = success, parameters[1] = [{message=BIP1520I, detail=BIP1520I: The Configuration Manager has initiated a deployment operation.
The Configuration Manager received a request to deploy configuration data and has consequently asked the following brokers to change their configuration: TESTBRK
The receipt of this message does not necessarily mean that deployment was successful; view the Event Log Editor in the Message Brokers Toolkit to check the outcome of the deployment. If you are deploying programmatically using the Config Manager Proxy, check the returned DeployResult object or LogProxy. There will be a separate set of log messages for each broker., source=ConfigMgr, timestamp=Thu Oct 13 16:32:32 EDT 2011}], parameters[2] = {uuid=300bc6f8-3201-0000-0080-b72a7a132126, commsmessage.lastinbatch=true, parent.uuid=2d0bc6f8-3201-0000-0080-b72a7a132126, configmanagerproxy.osname=Linux, deployment.barfilename=/home/mqbrkr/Deployments/Sample.bar, deployment.identifier=ws-h361-l336\mqbrkr, deployment.type=complete, userid=mqbrkr, configmanagerproxy.hostname=ws-h361-l336, commsmessage.configobjecttype=ExecutionGroup, commsmessage.operationtype=deploy, configmanagerproxy.noeventlog=false}, parameters[3] = 0, parameters[4] = true, object = default...
2011-10-13 16:32:33.0184 CMPNotification { com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager()
2011-10-13 16:32:33.0184 CMPNotification } com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager() returning false
2011-10-13 16:32:33.0184 CMPNotification { com.ibm.broker.config.proxy.DeployResultManager.processActionResponse(affectedObject=default, ccType=success)
2011-10-13 16:32:33.0184 CMPNotification d[3]: operationType = deploy
2011-10-13 16:32:33.0184 CMPNotification { com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager()
2011-10-13 16:32:33.0185 CMPNotification } com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager() returning false
2011-10-13 16:32:33.0185 CMPNotification d[3]: objectThatStartedThisDeploy = default (correct DeployResult found!)
2011-10-13 16:32:33.0185 CMPNotification { com.ibm.broker.config.proxy.DeployResult.setCompletionCodeForBroker(newCC=initiated)
2011-10-13 16:32:33.0185 CMPNotification { com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager()
2011-10-13 16:32:33.0185 CMPNotification } com.ibm.broker.config.proxy.AdministeredObject.hasBeenRestrictedByConfigManager() returning false
2011-10-13 16:32:33.0185 CMPNotification d[2]: Set completion code initiated for broker TESTBRK
2011-10-13 16:32:33.0185 CMPNotification d[2]: updateOverallCompletionCode() to initiated
2011-10-13 16:32:33.0185 CMPNotification } com.ibm.broker.config.proxy.DeployResult.setCompletionCodeForBroker()
2011-10-13 16:32:33.0185 CMPNotification { com.ibm.broker.config.proxy.LogEntry.getTimestamp()
2011-10-13 16:32:33.0185 CMPNotification } com.ibm.broker.config.proxy.LogEntry.getTimestamp() retVal=Thu Oct 13 16:32:32 EDT 2011
2011-10-13 16:32:33.0185 CMPNotification d[3]: Looking at message: {message=BIP1520I, detail=BIP1520I: The Configuration Manager has initiated a deployment operation.
The Configuration Manager received a request to deploy configuration data and has consequently asked the following brokers to change their configuration: TESTBRK
The receipt of this message does not necessarily mean that deployment was successful; view the Event Log Editor in the Message Brokers Toolkit to check the outcome of the deployment. If you are deploying programmatically using the Config Manager Proxy, check the returned DeployResult object or LogProxy. There will be a separate set of log messages for each broker., source=ConfigMgr, timestamp=Thu Oct 13 16:32:32 EDT 2011}
2011-10-13 16:32:33.0185 CMPNotification } com.ibm.broker.config.proxy.DeployResultManager.processActionResponse()
2011-10-13 16:32:33.0185 CMPNotification d[1]: ...Actionresponse notification completed.
2011-10-13 16:32:33.0185 CMPNotification d[1]: Awaiting notification message...
2011-10-13 16:32:42.0442 CMPMQReceiver.. d[3]: Polling response queue...
2011-10-13 16:32:52.0444 CMPMQReceiver.. d[3]: Polling response queue... |
..and then times out
Not sure what to do next except cross-checking all the installation instructions.
@lancelotlinc - How could say that my group name is wrong by lokking at my previous logs..It has just username I guess.
Are there any other logs/tables that I should be checking..
Thanks, |
|
Back to top |
|
 |
lancelotlinc |
Posted: Thu Oct 13, 2011 1:13 pm Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Its not as simple as changing the name. You have to create the security tokens before you run the installer. Then run the installer.
mqbrkrs group must be created before the installer runs. Read chapter 6. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
|