| Author |
Message
|
| 135711 |
 Posted: Thu Dec 31, 2015 5:36 pm Post subject: Temporary Deadlock between SAP PI & WebSphere MQ |
 |
|
Newbie
Joined: 13 Jul 2015
Posts: 9
|
Hello Everyone,
Happy New Year, 2016.
I have a environment where SAP PI is integrated with WebSphere MQ(v 7.1.0.2) and messages are transported from SAP PI to MQ queues. We have only one Server Connection channel defined in MQ.
Lately we have started facing(it was working fine till last month) where we experience "slowness" in the flow of messages from SAP to MQ during one specific interval of time everyday, it happens aroundd 9 to 11 every morning and sometime the issue also gets reported at night or at some odd hours. This slowness cause message delay from SAP to other integrated systems across MQ. The issue also gets resolved automatically without any intervention at MQ end or SAP end. It just happens and gets over after a period of time.
We tried increasing log level in SAP and found the below exception in logs.
asyncDelivery0" daemon cpu=54400.00 [reset 54400.00] ms
elapsed=673440.05 [reset 673440.05] s allocated=28042231832 B (26.12
GB) [reset 28042231832 B (26.12 GB)] defined_classes=12
io= file i/o: 1350858/8504747 B, net i/o: 299606808/326010667 B, files
opened:72, socks opened:34 [reset file i/o: 1350858/8504747 B, net
i/o: 299606808/326010667 B, files opened:72, socks opened:34 ]
user="" isapplicationthread="true" prio=10 tid=0x00002ac9c26d1000
nid=0x3e28 / 15912 pthread-id=47046118115648 runnable
[_thread_in_native (_at_safepoint),
stack(0x00002ac9c7188000,0x00002ac9c7289000)] [0x00002ac9c7287000]
java.lang.Thread.State: RUNNABLE
at
java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at
java.net.SocketInputStream.read([BII)I(SocketInputStream.java:129)
- additional info (remote:
nzakleafn027.zeus.ghsewn.com/10.155.13.11:1414, local:
localhost/127.0.0.1:39419)
at
java.io.BufferedInputStream.fill()V(BufferedInputStream.java:218)
at
java.io.BufferedInputStream.read()I(BufferedInputStream.java:237)
- locked <0x000000005bcb3cb8> (a
java.io.BufferedInputStream)
at
java.io.DataInputStream.readInt()I(DataInputStream.java:370)
at
com.ibm.mq.MQInternalCommunications.timedReadInt(I)I(MQInternalCommunications.java:2748)
at
com.ibm.mq.MQInternalCommunications.receiveBytesFaster(Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;)[B(MQInternalCommunications.java:2846)
- locked <0x000000005bcc8580> (a
com.ibm.mq.MQv6InternalCommunications)
at
com.ibm.mq.MQInternalCommunications.receive(Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;)Ljava/io/DataInputStream;(MQInternalCommunications.java:1179)
- locked <0x000000005bcc8580> (a
com.ibm.mq.MQv6InternalCommunications)
at
com.ibm.mq.MQSESSIONClient.lowLevelComms(Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;[B[BI[B)Ljava/io/DataInputStream;(MQSESSIONClient.java:2841)
- locked <0x000000005b20db60> (a
java.lang.Integer)
at
com.ibm.mq.MQSESSIONClient.MQGET(IILcom/ibm/mq/MQMsg2;Lcom/ibm/mq/MQGetMessageOptions;I[BLcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;)V(MQSESSIONClient.java:1852)
at
com.ibm.mq.MQQueue.getMsg2Int(Lcom/ibm/mq/MQMsg2;Lcom/ibm/mq/MQGetMessageOptions;Lcom/ibm/mq/Pint;Lcom/ibm/mq/Pint;I)V(MQQueue.java:1217)
- locked <0x000000005bc89d98> (a
com.ibm.mq.MQSPIQueue)
at
com.ibm.mq.MQQueue.getMsg2(Lcom/ibm/mq/MQMsg2;Lcom/ibm/mq/MQGetMessageOptions;)V(MQQueue.java:1074)
- locked <0x000000005bc89d98> (a
com.ibm.mq.MQSPIQueue)
at
com.ibm.mq.jms.MQMessageConsumer.getMessage(J)Lcom/ibm/jms/JMSMessage;(MQMessageConsumer.java:3402)
at
com.ibm.mq.jms.MQMessageConsumer.receiveAsync(J)Z(MQMessageConsumer.java:2938)
- locked <0x000000005bca3f98> (a
com.ibm.mq.jms.MQQueueReceiver)
at
com.ibm.mq.jms.contact admin.run()V(contact admin.java:406)
at
java.lang.Thread.run()V(Thread.java:761)
The logs shows hanging situation while connecting to MQ and we have also
found several threads spending a long time in obtaining a connection
to MQ in the socketRead call.
There is no significant error reported in MQ logs.
Can someone please let me know what can done to rectify this issue.
Any help is very much appreciated.
Thanks. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Dec 31, 2015 11:21 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I guess that this is on a Linux/Unix system.
Have you looked at the resoures available to MQ during this period?
The 'ipcs' command may well help see what is happening (or the equivalent where it is not avaiable)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| 135711 |
| Posted: Sun Jan 03, 2016 2:28 am Post subject: |
|
|
Newbie
Joined: 13 Jul 2015
Posts: 9
|
Ours MQ is on windows. The resources should not be a problem I guess, the problem has only started from last twenty days and it also gets rectified itself.
BTW, can I execute ipcs command on command console(I am not sure)?
Thanks. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Jan 03, 2016 2:51 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Looking at the SAP logs a bit closer, I see that there are issues with an MQGET.
This seems to be problems with SAP getting data from MQ not the other way round as you describe in your post.
Perhaps you are running out of Available channels for SAP to use?
There have been a good number of discussions in recent minths about the Clicnet connection count on this site. Take a look for them and show to monitor and if needed change them.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rammer |
| Posted: Sun Jan 03, 2016 6:52 am Post subject: |
|
|
Partisan
Joined: 02 May 2002
Posts: 359
Location: England
|
| A side from the SAP Log file, ensure nothing has changed on the Windows Servers by those Window Admin people such as scheduling virus scans, backups etc. Check process / memory usage on the windows box at time of slowness? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jan 03, 2016 7:06 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If using JMS to retrieve messages from MQ in SAP you'll need to look at a minimum at 2 things: max connections allowed in pool and max connections allowed on channel. Do these match your needs? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| 135711 |
| Posted: Mon Jan 04, 2016 1:40 pm Post subject: |
|
|
Newbie
Joined: 13 Jul 2015
Posts: 9
|
Thanks All for your reply.
Message flow is bi-directional. SAP to MQ & MQ to SAP.
We have one Server Connection Channel defined for SAP. Initially the MAXCHANNEL Limit on Queue Manager was 8000. I also suspected it to be MAXCHANNEL issue and now I increased it to 20000. But the issue is not solved.
@fjb_saper, how can I check the maximum connections allowed in pool? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jan 04, 2016 2:26 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| 135711 wrote: |
Thanks All for your reply.
Message flow is bi-directional. SAP to MQ & MQ to SAP.
We have one Server Connection Channel defined for SAP. Initially the MAXCHANNEL Limit on Queue Manager was 8000. I also suspected it to be MAXCHANNEL issue and now I increased it to 20000. But the issue is not solved.
@fjb_saper, how can I check the maximum connections allowed in pool? |
This is a setting most probably defined on the JMS connection pooling as implemented by SAP PI...
If you limit the connections in the pool to something less than what you need there is no point in increasing the max connections on MQ.
Your PI server will allocate up until the max connections in the pool and then wait for a connection to be returned to the pool and become available before allocating it... If your max pool is too low, that could well be a reason for a slow down...
You say you defined 8000 as max connections on MQ. What was the max used connections at any time? If you are always hitting the MAXCHANNEL limit I would suspect that you have an application that is not releasing correctly its connections. Especially after the bump to 20,000....
By the way what is the MQ channel's SHARECNV setting for the SAP PI channel? I guess that could be bumped down to 0 without affecting too much the environment. (Be aware that you will have that many more sockets open) YMMV 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| 135711 |
| Posted: Mon Jan 04, 2016 6:02 pm Post subject: |
|
|
Newbie
Joined: 13 Jul 2015
Posts: 9
|
Thanks fjb_saper.
Around six months back when we were integrating SAP PI with MQ we encountered MQJMS2005 error at PI end, and also we encountered our Server Connection channel going down in MQ error logs. We analyzed it back then found the issue was with MAXCHANNEL limit which was by default 100 then. We increased this value to 8000 and the problem was solved and we were able to roll it out in production.
The system remained stable for around six months. Somewhere in October, 2015 again we encountered this error and we increased MAXCHANNEL limit to 12000 from 8000. Please note that we have not asked SAP PI team to do any change at their end. Today I had a talk on JMS Connection pooling and they said they will get back to me.
Next, the issue started again in December(it was stable for two months after we increased it to 12000), and we increased it to 20000, but the issue is not resolved. It happens almost daily and it gets over without any intervention either at MQ end or SAP end.
For SHARECNV, I guess this is "Sharing conversations" under Extended tab of server connection channel(please correct me if I am wrong). This value is currently 10 in my production env. Do you wan't me to bring it down to 0.
Please advise.
Thanks. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jan 05, 2016 12:35 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
go talk to SAP about the issue.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jan 05, 2016 5:54 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
For you to need such high numbers with a share conversation at 10, this looks to me like something is not correctly releasing the connection.
If you have multiple clients connecting, you may want to use the channel attributes limiting the number of connections to find out which app is the culprit. Using JMS, you should never rely on garbage collection to release a resource. Always release it explicitly.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| 135711 |
| Posted: Tue Jan 05, 2016 5:48 pm Post subject: |
|
|
Newbie
Joined: 13 Jul 2015
Posts: 9
|
Thanks All.
Our MQ is of version 7.0.1.2 where us JMS client used by SAP is v 6. I think v 6 of JMS will not be able to leverage the shared conversation feature of MQ.
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jan 05, 2016 9:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| 135711 wrote: |
Thanks All.
Our MQ is of version 7.0.1.2 where us JMS client used by SAP is v 6. I think v 6 of JMS will not be able to leverage the shared conversation feature of MQ.
Thanks. |
Of course you do realize that V7.0.x of MQ is out of support and that you do need to upgrade... (and I believe a good stable version for JMS was 7.0.1.4... so you'd be behind there as well....) and you definitely need to upgrade the SAP JMS MQ client if it still is at V6 !!!
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 06, 2016 4:53 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Unless you saw entries in the Queue Manager Error log on the MQ server specifically saying max channels was reached, the maximum number of channels allowed on the queue manager is not the root cause for this issue.
Indeed, you said:
| Quote: |
| There is no significant error reported in MQ logs. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
