| Author |
Message
|
| Mqdevops |
 Posted: Mon Oct 23, 2023 8:07 am Post subject: Unable to create RDQM queue manager on RHEL9 |
 |
|
Newbie
Joined: 21 Oct 2023
Posts: 7
|
Folks i'm running into an issue building RDQM queue manager.
I'm building RDQM HA/DR. There are 6 servers, 3 PROD and 3 DR. I will share the content of the rdqm.ini file as well as the error i'm seeing when attempting to create the queue manager. Just background these 6 RHEL9 servers are running on KVM (libvirt) on RHEL9 server.
| Code: |
Name: IBM MQ
Version: 9.3.0.10
Level: p930-010-230816
BuildType: IKAP - (Production)
Platform: IBM MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 5.14.0-284.30.1.el9_2.x86_64
O/S Details: Red Hat Enterprise Linux 9.2 (Plow)
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 930
LicenseType: Production
rdqm.ini file
[root@rdqmprd01 ~]# cat /var/mqm/rdqm.ini
# The configuration in this file is not dynamic.
# The HA configuration is read when an HA group is created.
# The DR configuration is read when when a DR/HA queue manager is created.
Node:
Name=rdqmprd01
HA_Replication=10.10.50.11
DR_Replication=10.10.60.11
Node:
Name=rdqmprd02
HA_Replication=10.10.50.12
DR_Replication=10.10.60.12
Node:
Name=rdqmprd03
HA_Replication=10.10.50.13
DR_Replication=10.10.60.13
DRGroup:
Name=DRREPLGRP
DR_Replication=10.10.60.21
DR_Replication=10.10.60.22
DR_Replication=10.10.60.23
|
| Code: |
[root@rdqmprd01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 4.8G 33M 4.7G 1% /dev/shm
tmpfs 1.9G 9.4M 1.9G 1% /run
/dev/mapper/rhel-root 44G 5.6G 39G 13% /
/dev/mapper/vg_mq-varmqm 20G 176M 20G 1% /var/mqm
/dev/mapper/vg_mq-optmqm 20G 1.8G 19G 9% /opt/mqm
/dev/mapper/vg_mq-mqmtrace 10G 104M 9.9G 2% /var/mqm/trace
/dev/mapper/vg_mq-mqmlog 20G 175M 20G 1% /var/mqm/log
/dev/mapper/vg_mq-mqmerror 10G 104M 9.9G 2% /var/mqm/errors
/dev/vdb1 1014M 406M 609M 40% /boot
contact admin:/mnt/contact admin/ 812G 208G 605G 26% /software
tmpfs 764M 52K 764M 1% /run/user/42
tmpfs 764M 36K 764M 1% /run/user/0
[root@rdqmprd01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/vda drbdpool lvm2 a-- <100.00g <100.00g
/dev/vdb2 rhel lvm2 a-- <49.00g 0
/dev/vdc vg_mq lvm2 a-- <100.00g <20.00g
[root@rdqmprd01 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rhel -wi-ao---- <44.00g
swap rhel -wi-ao---- 5.00g
mqmerror vg_mq -wi-ao---- 10.00g
mqmlog vg_mq -wi-ao---- 20.00g
mqmtrace vg_mq -wi-ao---- 10.00g
optmqm vg_mq -wi-ao---- 20.00g
varmqm vg_mq -wi-ao---- 20.00g
|
Command used to create first qmgr and error
| Code: |
[root@rdqmprd01 ~]# sudo crtmqm -sx -rr p -rn DRREPLGRP -rp 7017 -fs 10G -lp 20 -ls 20 -lc -lf 16384 -h 1000 -u "TEST.DLQ" -p 1417 TEST
Creating replicated data queue manager configuration.
Secondary queue manager created on 'rdqmprd02'.
Secondary queue manager created on 'rdqmprd03'.
AMQ3817E: Replicated data subsystem call '/usr/sbin/drbdadm -- --force
--stacked create-md test.dr' failed with return code '10'.
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/drbd100 at byte offset 10736721920
Do you really want to overwrite the existing meta-data?
*** confirmation forced via --force option ***
initializing bitmap (320 KB) to all zero
ioctl(/dev/drbd100, BLKZEROOUT, [10736361472, 327680]) failed: Input/output error
initializing bitmap (320 KB) to all zero using pwrite
pwrite(5,...,327680,10736361472) in md_initialize_common:BM failed: Input/output error
Command 'drbdmeta 7017 v09 /dev/drbd100 internal create-md 1 --force' terminated with exit code 10
AMQ3812E: Failed to create replicated data queue manager configuration.
Secondary queue manager deleted on rdqmprd02.
Secondary queue manager deleted on rdqmprd03.
|
| Code: |
[root@rdqmprd01 ~]# rdqmstatus
Node: rdqmprd01
OS kernel version: 5.14.0-284.30.1
DRBD OS kernel version: 5.14.0-284.30.1
DRBD version: 9.1.15+ptf.1.g2ec62f6cb988
DRBD kernel module status: Loaded
[root@rdqmprd01 ~]# rdqmstatus -n
Node rdqmprd01 is online
Node rdqmprd02 is online
Node rdqmprd03 is online
[root@rdqmprd01 ~]#
|
| Code: |
[root@rdqmprd01 ~]# crm status
Cluster Summary:
* Stack: corosync
* Current DC: rdqmprd02 (version 2.1.2.linbit-4.el9-ada5c3b36e2) - partition with quorum
* Last updated: Mon Oct 23 08:43:47 2023
* Last change: Mon Oct 23 07:43:15 2023 by root via crm_attribute on rdqmprd03
* 3 nodes configured
* 0 resource instances configured
Node List:
* Online: [ rdqmprd01 rdqmprd02 rdqmprd03 ]
Full List of Resources:
* No resources
|
|
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sat Oct 28, 2023 12:35 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
run vgs as root and show us the output...
After you created the rdqm.ini did you run rdqmadm -c (on each node)?
After creating the queue manager you should get in the output the command to run on the DR side.
The output of crtmqm clearly states that the qmgr was not created due to some error. You can try and run each of the commands on each of the nodes manually and see if it makes a difference ...
Your rdqmstatus clearly shows that the qmgr was not created...
Check that the qmgr is there running rdqmstatus -m QMNAME -a
-a will show any failed resource action
also run
Did you check the network and firewall?
The documentation has commands for you to run to open the ports for communication...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mqdevops |
| Posted: Sat Oct 28, 2023 9:32 am Post subject: fjb_saaper thank you for the reply |
|
|
Newbie
Joined: 21 Oct 2023
Posts: 7
|
[root@rdqmprd01 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
drbdpool 1 0 0 wz--n- <100.00g <100.00g
rhel 1 2 0 wz--n- <49.00g 0
vg_mq 1 5 0 wz--n- <100.00g <20.00g
I ran rdqmadm -c on the primary node and that created the cluster, when I attempted to run it on the other two nodes it said it was already done and it is part of the cluster
the crtmqm commands starts creating the qmgr, but then it deletes it after it runs into a problem.
I tried to run the command on the other two nodes and I get the same exact behavior.
I did check firewall, I stopped the firewall service for now. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 31, 2023 5:34 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Do not run the crtmqm -sx command on the other nodes, run the crtmqm -sxs command.
The output might tell you why your -sx command fails...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mqdevops |
| Posted: Tue Nov 07, 2023 9:58 am Post subject: still at square one |
|
|
Newbie
Joined: 21 Oct 2023
Posts: 7
|
| It is still unclear what the root cause of the problem is. |
|
| Back to top |
|
|
| RobertK |
| Posted: Mon Nov 13, 2023 9:25 am Post subject: |
|
|
Newbie
Joined: 17 Oct 2023
Posts: 3
Location: NC, USA
|
Please try to troubleshoot the problem by performing these steps:
1. Open two ssh sessions to node rdqmprd01
2. In one session, type: tail -f /var/log/messages
3. In the other, enter your crtmqm command, but use a different replication port (-rp) and a different queue manager name (QM123 maybe?).
View the failure in the /var/log/messages file happen before your eyes. DRBD logs to the /var/log/messages file by default.
Post the errors or success story here --V |
|
| Back to top |
|
|
| ralok |
| Posted: Sun Jan 07, 2024 3:43 pm Post subject: RDQM QManager will not start |
|
|
Newbie
Joined: 07 Jan 2024
Posts: 3
|
Hi mqdevops
did you get the problem resolved ? I am working on development environment and using 9.3.4.0-IBM-MQ-Advanced-for-Developers-LinuxX64.tar binary for the MQ .
RHEL : 5.14.0-284.30.1.el9_2.x86_64 |
|
| Back to top |
|
|
| Mqdevops |
| Posted: Mon Jan 08, 2024 5:02 am Post subject: Hey Ralok |
|
|
Newbie
Joined: 21 Oct 2023
Posts: 7
|
| unfortunately the issue still stands, IBM pointed to a network issues. However, I have done several network diagnoses and there hasn't been any issues found. I have built several other clusters on a different environment with no issues. Unsure what is causing this problem on this one still. |
|
| Back to top |
|
|
| ralok |
| Posted: Mon Jan 08, 2024 4:14 pm Post subject: RDQM QManager will not start |
|
|
Newbie
Joined: 07 Jan 2024
Posts: 3
|
Thanks mqdevops for your response .
Hi all
. I have same issue while building this on AWS .
I am using EC2 t3 across AZs and EBS ( gp2) as storage .
i am currently using a single persistent IP . Though the each box has got two IPs . One persistent and another Non persistent .
Had security groups allowed traffic to ports used by DRBD and pacemaker .
i do see below but this seems more of a warning to me .
2024-01-08 22:05:24.587: 21241 >>STDERR:
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/drbdpool/qma_00 at byte offset 3221221376
dmesg has following which looks a bit suspicious
bd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
[ 1519.582156] drbd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
[ 1519.589911] drbd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
[ 1519.597186] drbd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
[ 1519.606964] drbd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
[ 1519.617831] drbd qma/0 drbd100: FIXME drbd_a_qma[21329] op clear, bitmap locked for 'demote' by drbd_w_qma[21301]
Any help will be appreciated . |
|
| Back to top |
|
|
| ralok |
| Posted: Mon Jan 08, 2024 4:39 pm Post subject: |
|
|
Newbie
Joined: 07 Jan 2024
Posts: 3
|
Further info ..
I also discovered the followings
4582.846973] drbd qma/0 drbd100: disk( Attaching -> Inconsistent )
[ 4582.850922] drbd qma/0 drbd100: attached to current UUID: 0000000000000004
[ 4582.857945] drbd qma ip-xxxxxxxxl: conn( StandAlone -> Unconnected )
[ 4582.865804] drbd qma ip-xxxxxxxxxxxx: conn( StandAlone -> Unconnected )
[ 4582.872076] drbd qma ip-xxxxxxxxxxxx: Starting receiver thread (from drbd_w_qma [22122])
[ 4582.879001] drbd qma ip-xxxxxxxx: Starting receiver thread (from drbd_w_qma [22122])
[ 4582.879008] drbd qma ip-xxxxxxxxxx: conn( Unconnected -> Connecting )
[ 4582.893933] drbd qma xxxxxxxx: conn( Unconnected -> Connecting )
[ 4583.409275] drbd qma ip-xxxxxxxxx: Handshake to peer 1 successful: Agreed network protocol version 121
[ 4583.411257] drbd qma ip-xxxxxxxxxx: Handshake to peer 2 successful: Agreed network protocol version 121
[ 4583.416984] drbd qma ip-xxxxxxxxxx.ec2.internal: Feature flags enabled on protocol level: 0x6f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
[ 4583.424261] drbd qma ip-xxxxxxxxxxxx: Feature flags enabled on protocol level: 0x6f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
[ 4583.431259] drbd qma ip-xxxxxxxx: Starting ack_recv thread (from drbd_r_qma [22144])
[ 4583.439222] drbd qma ip-xxxxxxxxx: Starting ack_recv thread (from drbd_r_qma [22145]) |
|
| Back to top |
|
|
| marc.CH |
| Posted: Wed Apr 24, 2024 5:54 am Post subject: |
|
|
Newbie
Joined: 23 Apr 2024
Posts: 5
Location: Geneva,CH
|
|
| Back to top |
|
|
|
|
