| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Managing MQ running on Openshift |
« View previous topic :: View next topic » |
| Author |
Message
|
| andres |
 Posted: Wed Feb 15, 2023 4:23 am Post subject: Managing MQ running on Openshift |
 |
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
HI,
recently i managed to deploy an QM in OPenshift, correctly preconfigured (mq.ini, mqsc files, authority etc) that runs on development. Im not an MQ admin but a OpenShift admin and all is deployed with ARgoCD.
IN theory all configuratin changes should be done via git repository that ArgoCD watch and apply them.i like that. Not via web console or MQ Explorer.
The problem here is somehow:
1. Should we Close admin access on MQ EXplorer or Web Brownser to do changes? There is a risk that some changes would be re-applied by GitOps or have configuration drift.
2. What i can see if that anytime i need to do a change (like new queue, change auth, etc ) it should be done via Gi, But this will regenerate again the pods (Which i guess is ok if the cluster is HA). I guess it should be more simple to reapply the config files and refresh the cluster?
Some opinions of admin managing mq on containers, would be appreciate it.
Thanks! |
|
| Back to top |
|
 |
| avs01 |
| Posted: Thu Feb 16, 2023 1:06 am Post subject: |
|
|
Newbie
Joined: 16 Feb 2023
Posts: 6
|
Hi andres,
We recently started with MQ on Openshift, also using GIT, ArgoCD.
All MQ config is stored in GIT.
For administration of MQ objects (queue, topic, sub, etc) we use the scripting in github repos 'cloud-native-toolkit/mq-qm01', so we dont have to restart a pod for each configuration change.
For remote administration we use the MQ Explorer. I think, for planned changes you can use GIT and ArgoCD. But if you have an urgent incident (queue full for example), then I think you can quickly edit the queue using the MQ explorer because it is faster than using GIT, ArgoCD. When the incident is resolved, you should of course apply the change in GIT as well.
Regards,
Arthur |
|
| Back to top |
|
|
| andres |
| Posted: Thu Feb 16, 2023 7:38 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Hi Arthur,
Actually im using https://github.com/ibm-messaging/mq-helm and adding the mqsc commands as configMaps and authentication records as secrets.
i dont understand " github repos 'cloud-native-toolkit/mq-qm01', so we dont have to restart a pod for each configuration change."
Do you connect remotly to the pods and apply the new scripts or have a separated argocd application to sync the mqsc scripts? |
|
| Back to top |
|
|
| avs01 |
| Posted: Mon Feb 20, 2023 3:14 am Post subject: |
|
|
Newbie
Joined: 16 Feb 2023
Posts: 6
|
Hi Andres,
I meant the git repository https://github.com/cloud-native-toolkit/mq-qm01
In the repository is a scripts directory, which contains a script start-mqsc.sh to load MQSC in the qmgr. The script is deployed in a ConfigMap and started as a qmgr service.
The scripts looks every x seconds for changes to a file '/dynamic-mq-config-mqsc/dynamic-definitions.mqsc'. If the file is changed (e.g. new MQSC definitions), then the contents of the file are loaded in the qmgr.
The file /dynamic-mq-config-mqsc/dynamic-definitions.mqsc is mounted by the qmgr pod, and is maintained by a ConfigMap 'dynamic-mqsc-configmap'.
So, any changes to the ConfigMap (through GIT, ArgoCD) are deployed to the ConfigMap, resulting in an updated MQSC file. The scripts notices a change to the MQSC file and loads it.
Regards,
Arthur |
|
| Back to top |
|
|
| andres |
| Posted: Thu Feb 23, 2023 4:08 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Thanks, it sounds like a good idea.
Are you using MQ Explorer?
I wonder if you are connecting to MQ using a NodePort or a route.
Im not having problems connecting from NodePort, but im trying to use a route to the "1414" service and connect to a SSL channel., ind it doesnt work.
IM not even sure if MQ Explorer can use anm openshift route |
|
| Back to top |
|
|
| avs01 |
| Posted: Mon Feb 27, 2023 7:08 am Post subject: |
|
|
Newbie
Joined: 16 Feb 2023
Posts: 6
|
Yes, I got MQ Explorer working with mutual TLS to qmgrs on Openshift.
It took me some time and frustration to configure, so I will share my lessons learned with you:
First, you need a keystore/truststore containing a client certificate and CA certificates for the MQ Explorer and choose a cipherspec that meets your requirements.
The Openshift qmgr should be configured with TLS, so create a secret containing the private key, server certificate and CA certificate(s) and reference to it in the qmgr.yaml:
pki:
keys:
- name: ibmwebspheremq<myqmgr>
secret:
secretName: <myqmgr>
items:
- tls.key
- tls.crt
- signer.crt
On the qmgr pod, you can then check the generated keystore (based on the secret) with:
export LD_LIBRARY_PATH=/opt/mqm/gskit8/lib64
/opt/mqm/gskit8/bin/gsk8capicmd_64 -cert -list -db /run/runmqserver/tls/key.kdb -stashed -v
And validate the keystore with:
/opt/mqm/gskit8/bin/gsk8capicmd_64 -cert -validate -db /run/runmqserver/tls/key.kdb -stashed
Next, create a svrconn channel for the MQ Explorer:
DEFINE CHANNEL(<MYQMGR>.TST) CHLTYPE(SVRCONN) TRPTYPE(TCP) SSLCAUTH(REQUIRED) SSLPEER('CN=<your client CN>') SSLCIPH('ANY_TLS12_OR_HIGHER') REPLACE
MQ authorization should be setup by CHLAUTH records, but for this example we disable MQ authorization in the qmgr.yaml:
spec:
template:
pod:
containers:
- env:
- name: MQSNOAUT
value: 'yes'
If you have your qmgr configured correctly, then setup connectivity through Openshift routes. By default, our route port is exposed on port 443.
Example is here:
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=dcqmumo-configuring-route-connect-queue-manager-from-outside-red-hat-openshift-cluster
Routing on Openshift is done by SNI.
Depending on the client implementation (OutboundSNI=channel or OutboundSNI=hostname), you need a Openshift route for every (svrconn, rcvr) channel on Openshift qmgr.
So, if a client is using OutboundSNI=channel, you need a Openshift route for the MQ channel the client is using.
I found out that the MQ Explorer is using OutboundSNI=hostname, so you could just use the hostname from the default generated route for the qmgr ‘<MYQMGR>-ibm-mq-qm’.
I assume you have the IBM MQ operator installed, which will create a default route for each qmgr.
Because we have a wild variety of clients, it is unknown which SNI implementation the clients use.
As a best practice, we always create a Openshift route for each svrconn channel.
Using SNI, traffic is then routed to your qmgr:1414.
Beware, the Openshift routes for MQ channels have specific SNI naming conventions:
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=requirements-how-mq-provides-multiple-certificates-capability
Especially pay attention when using dots or other characters in your channel names!
So, if you have a MQ channel defined as "<MYQMGR>.TST", you need a route configuration for that:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: <MYQMGR>-tst
spec:
host: <MYQMGR>2e-tst.chl.mq.ibm.com
to:
kind: Service
name: <MYQMGR>-ibm-mq
weight: 100
port:
targetPort: 1414
tls:
termination: passthrough
wildcardPolicy: None
Hopefully this information is usefull for your situation!
Kind regards,
Arthur |
|
| Back to top |
|
|
| andres |
| Posted: Thu Mar 02, 2023 7:23 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Thanks avs01,
I did perform most of the steps, but i think something is happening with my router.
i created a channel "SSL":
DEFINE CHANNEL(SSL) CHLTYPE(SVRCONN) TRPTYPE(TCP) SSLCAUTH(REQUIRED) SSLPEER('CN=MY DOmain') SSLCIPH('ANY_TLS12_OR_HIGHER') REPLACE
disable CHLAUT:
spec:
template:
pod:
containers:
- env:
- name: MQSNOAUT
value: 'yes'
and create the route:
spec:
host: ssl.chl.qm.mqdev.apps.ocptest.com
to:
kind: Service
name: mqdev-ibm-mq
weight: 100
port:
targetPort: 1414
tls:
termination: passthrough
wildcardPolicy: None
From a browser: ssl.chl.qm.mqdev.apps.ocptest.com -> ity will show logs on the pods like:
The data received from host '10.131.0.2' on channel '????' is not valid. [CommentInsert1(10.131.0.2), CommentInsert2(TCP/IP), CommentInsert3(????)]
2023-03-02T15:13:33.956Z AMQ9999E: Channel '????' to host '10.131.0.2' ended abnormally. [CommentInsert1(????), CommentInsert2(620), CommentInsert3(10.131.0.2)]
2023-03-02T15:13:50.244Z mqhtpass: MQStart options=Secondary qmgr=mqdev
And then from MQ Explorer,
Host: ssl.chl.qm.mqdev.apps.ocptest.com
port:443
channel: SSL
I got error: Could not establish a connection to the queue manager - reason 2009. (AMQ4059)
Could not establish a connection to the queue manager - reason 2009. (AMQ4059)
Severity: 10 (Warning)
Explanation: The attempt to connect to the queue manager failed. This could be because the queue manager is incorrectly configured to allow a connection from this system, or the connection has been broken.
Response: Try the operation again. If the error persists, examine the problem determination information to see if any information has been recorded.
And i dont see any logs in the pods, like the queue manager is rejecting access or the route is not routing correctly |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Mar 03, 2023 1:32 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Looks like you're not using the channel name format for the route, so you need to set OutboundSNI=HOSTNAME for your client connection. I think that OutboundSNI=CHANNEL is the default. Have you tried doing that?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| andres |
| Posted: Fri Mar 03, 2023 3:02 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
I actually tried both, but if i understand MQ Explorer uses
OutboundSNI=HOSTNAME
So i simple route like:
qm.mqdev.apps.ocptest.com
domain apps.ocptest.com
mqdev: name of the queue manager
qm: (just suffix to differenciate this from web console route)
Im tracing mq explorer and i get this:
03/03/23 10:35:54.235.16] 00000003 @97d23939 c.i.m.j.remote.impl.RemoteTCPConnection(RemoteConnection) ----+----+----+----+- f connect(RemoteTls,MQCSP)
[03/03/23 10:35:54.235.17] 00000003 @b7144c10 c.i.m.j.remote.impl.RemoteConnectionSpecification ----+----+----+----+ X getNewConnection(RemoteTls, MQCNO, MQCSP, RemoteExitChainPair,ChannelSharingMode)<catchIndex 4>
[03/03/23 10:35:54.235.17] 00000003 CC=2;RC=2009;AMQ9208: Error on receive from host 'qm.mqdev.apps.ocptest.com/10.21.64.102:443 (qm.mqdev.apps.ocptest.com)'. [1=-1,2=ffffffff,3=qm.mqdev.apps.ocptest.com/10.21.64.102:443 (qm.mqdev.apps.ocptest.com),4=TCP] [com.ibm.mq.jmqi.JmqiException] at:
[03/03/23 10:35:54.235.17] 00000003 com.ibm.mq.jmqi.remote.impl.RemoteConnection.receiveTSH(RemoteConnection.java:3341)
...
CurrentThread ClassLoader = org.eclipse.osgi.internal.framework.ContextFinder@8618d221
[03/03/23 10:35:54.251.0U] 00000003 static c.ibm.mq.ReasonCodeInfo ----+---- d static SCCS id [@(#) MQMBID sn=p932-L230207 su=_mMBuZqcAEe2pWoFAaNK_Tg pn=com.ibm.mq/src/com/ibm/mq/ReasonCodeInfo.java]
[03/03/23 10:35:54.251.0V] 00000003 static c.ibm.mq.ReasonCodeInfo ----+---- { getResourceException(MQException) [com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2009'.]
[03/03/23 10:35:54.251.0W] 00000003 static c.ibm.mq.MQResourceException ----+----+ d static SCCS id [@(#) MQMBID sn=p932-L230207 su=_mMBuZqcAEe2pWoFAaNK_Tg pn=com.ibm.mq/src/com/ibm/mq/MQResourceException.java]
[03/03/23 10:35:54.251.0X] 00000003 @840fc60 c.ibm.mq.MQResourceException ----+----+ { <init>(int,int,MQException) [2(0x2)] [2009(0x7d9)] [com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2009'.]
[03/03/23 10:35:54.251.0Y] 00000003 @840fc60 c.ibm.mq.MQResourceException ----+----+ } <init>(int,int,MQException)
[03/03/23 10:35:54.251.0Z] 00000003 static c.ibm.mq.ReasonCodeInfo ----+---- } getResourceException(MQException) returns [com.ibm.mq.MQResourceException]
[03/03/23 10:35:54.251.10] 00000003 @387b3edd c.ibm.mq.MQClientManagedConnectionFactoryJ11 ----+---- ! _createManagedConnection(MQConnectionRequestInfo,boolean)<throwIndex 2>, <null> [com.ibm.mq.MQResourceException]
[03/03/23 10:35:54.251.11] 00000003 @a495059c c.ibm.mq.MQSimpleConnectionManager ----+- X allocateConnection(MQManagedConnectionFactory,MQConnectionRequestInfo)<catchIndex 2> |
|
| Back to top |
|
|
| avs01 |
| Posted: Tue Mar 07, 2023 5:41 am Post subject: |
|
|
Newbie
Joined: 16 Feb 2023
Posts: 6
|
Hi Andres,
It is hard to analyse what the issue is from this post.
The issue could be in MQ Explorer configuration or in the router configuration.
To test your qmgr security configuration through a route, you could use the sample program amqssslc from a AIX/Linux/Windows system.
If that works, the qmgr security (keystore) and openshift routing are not the cause of your issue.
On AIX, the amqssslc program is OutboundSNI=CHANNEL so in that case you need an additional route for the channel SNI mapping:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: channelsslroute
spec:
host: ssl.chl.mq.ibm.com
to:
kind: Service
name: mqdev-ibm-mq
weight: 100
port:
targetPort: 1414
tls:
termination: passthrough
wildcardPolicy: None
Create a truststore (.kdb) using ikeyman or gskit and add the CA certificates to that store.
Run the program (truststore without .kdb extension):
/usr/mqm/samp/bin/amqssslc -m MQDEV -c SSL -x "ssl.chl.qm.mqdev.apps.ocptest.com(443)" -k /home/mqm/mytruststore -s ANY_TLS12_OR_HIGHER -p ANY
If it doesn't give a error, then you tested that your qmgr security through a route is in place:
Sample AMQSSSLC start
Connecting to queue manager <qmgr>
Using the server connection channel <channel>
on connection name <host>:<port>.
Using SSL CipherSpec ANY_TLS12_OR_HIGHER
Using SSL key repository stem <path truststore>
Certificate Validation Policy: 0
No OCSP configuration specified.
Connection established to queue manager <qmgr>
Sample AMQSSSLC end |
|
| Back to top |
|
|
| andres |
| Posted: Thu Mar 09, 2023 5:11 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Thanks for the help.
The host route ssl.chl.mq.ibm.com seems a bit strange!
Now im getting this error from AMQSSSLC:
/amqssslc -m mqdev-c "SSL" -x "ssl.chl.qm.mqdev.apps.ocptest.com(443)" -k /mnt/c/Temp/mq-helm-main/samples/openshift/test/new/key -s ANY_TLS12_OR_HIGHER -p ANY
Sample AMQSSSLC start
Connecting to queue manager flagstaff
Using the server connection channel SSL
on connection name ssl.chl.qm.mqdev.apps.ocptest.com(443).
Using SSL CipherSpec ANY_TLS12_OR_HIGHER
Using SSL key repository stem /mnt/c/Temp/mq-helm-main/samples/openshift/test/new/key |
|
| Back to top |
|
|
| andres |
| Posted: Thu Mar 09, 2023 5:28 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Other note,
i create the certs like:
openssl req -newkey rsa:2048 -nodes -keyout <keyname>.key -x509 -days 365 -out <certname>.crt -subj "CN=mqdev,OU=mydomain"
openssl pkcs12 -export -in <cert filename> -inkey <key filename> -out <p12 keystore name>
keytool -importkeystore -srckeystore <p12 keystore filename> \
-srcstoretype PKCS12 \
-destkeystore <jks keystore filename> \
-deststoretype JKS
Then i created the .kdb with IKeyman importing "jks" file.
https://www.ibm.com/support/pages/amq9637-channel-lacking-certificate-error-wmq-log-when-sslcauth-set-required
If i set the channel to SSL = Optional amd run amqssslc program i get:
dis chs(SSL)
5 : dis chs(SSL)
AMQ8417I: Display Channel Status details.
CHANNEL(SSL) CHLTYPE(SVRCONN)
CONNAME(10.131.0.2) CURRENT
STATUS(RUNNING)
(no SSL settings?)
server logs:
mqhtpass: User authentication failed user=andres effuser=andres applname=amqssslc cspuser=andres cc=1 reason=0
why is it taking this user? ("andres" is my linux shell user).
should i add "SSL Peer"?
AMQ8878I: Display channel authentication record details.
CHLAUTH(SSL) TYPE(ADDRESSMAP)
DESCR( ) CUSTOM( )
ADDRESS(*) USERSRC(CHANNEL)
CHCKCLNT(ASQMGR) ALTDATE(2023-03-09)
ALTTIME(09.40.12)
3 : dis channel (SSL)
AMQ8414I: Display Channel details.
CHANNEL(SSL) CHLTYPE(SVRCONN)
ALTDATE(2023-03-09) ALTTIME(13.16.12)
CERTLABL( ) COMPHDR(NONE)
COMPMSG(NONE) DESCR( )
DISCINT(0) HBINT(300)
KAINT(AUTO) MAXINST(999999999)
MAXINSTC(999999999) MAXMSGL(4194304)
MCAUSER( ) MONCHL(QMGR)
RCVDATA( ) RCVEXIT( )
SCYDATA( ) SCYEXIT( )
SENDDATA( ) SENDEXIT( )
SHARECNV(10) SSLCAUTH(OPTIONAL)
SSLCIPH(ANY_TLS12_OR_HIGHER) SSLPEER( )
TRPTYPE(TCP) |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Mar 10, 2023 1:32 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| andres wrote: |
If i set the channel to SSL = Optional amd run amqssslc program i get:
dis chs(SSL)
5 : dis chs(SSL)
AMQ8417I: Display Channel Status details.
CHANNEL(SSL) CHLTYPE(SVRCONN)
CONNAME(10.131.0.2) CURRENT
STATUS(RUNNING)
(no SSL settings?) |
If you want to see SSL settings on the DISPLAY CHSTATUS output, you'll need to ask to see those fields. Or you can just ask for ALL the fields.
| Code: |
| DISPLAY CHSTATUS(SSL) ALL |
If the channel is running though, that means you were successfully connected and thus must be using TLS or it would not be running.
| andres wrote: |
server logs:
mqhtpass: User authentication failed user=andres effuser=andres applname=amqssslc cspuser=andres cc=1 reason=0
why is it taking this user? ("andres" is my linux shell user). |
The user id from the client machine - your linux shell user - is flowed up to the queue manager. If you have nothing in place to use instead of it, then that is what will be used. Read All the ways to set MCAUSER for some ideas.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| andres |
| Posted: Tue Mar 14, 2023 1:03 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Thanks hughson, it takes some time to understand all auth possibilities.
IN the end i sort it out to connect to the queue manager, test connection with "amqssslc" and send messages with ampsphac and get with amqsgetc.
For some reason, amqsputc get stuck and it is not sending any message.
I created my own certificate "/CN=andres/OU=mydomain/O=myorg". But before i created a simple one "/CN=andres/OU=mydomain" and i couldnt stablish a SSL connection.
My CHLAUTH looks like this now, is there a prevalence between SSLPEERMAP and Address?. I think the AUTH is only working if a enable Address and it is not checking SSLPEER
| Code: |
Q8878I: Display channel authentication record details.
CHLAUTH(SSL) TYPE(SSLPEERMAP)
DESCR( ) CUSTOM( )
SSLPEER(CN=andres,OU=mydomain,O=myorg)
SSLCERTI( ) ADDRESS( )
MCAUSER(mqm) USERSRC(MAP)
CHCKCLNT(ASQMGR) ALTDATE(2023-03-14)
ALTTIME(08.34.01)
AMQ8878I: Display channel authentication record details.
CHLAUTH(SSL) TYPE(ADDRESSMAP)
DESCR( ) CUSTOM( )
ADDRESS(*) MCAUSER(mqm)
USERSRC(MAP) CHCKCLNT(ASQMGR)
ALTDATE(2023-03-14) ALTTIME(08.34.44)
6 MQSC commands read. |
|
|
| Back to top |
|
|
| andres |
| Posted: Tue Mar 14, 2023 1:08 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Forgot to mention that if i remove CHLAUTH (Address) and i keep only SSLPEER:
| Code: |
CHLAUTH(SSL) TYPE(SSLPEERMAP)
DESCR( ) CUSTOM( )
SSLPEER(CN=andres,OU=mydomain,O=myorg)
SSLCERTI( ) ADDRESS( )
MCAUSER(mqm) USERSRC(MAP)
CHCKCLNT(ASQMGR) ALTDATE(2023-03-14)
ALTTIME(08.34.01) |
2023-03-14T09:05:46.011Z mqhtpass: User authentication failed user=andres effuser=andres applname=amqsphac cspuser=andres cc=1 reason=0 |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
