| Author |
Message
|
| ucbus1 |
 Posted: Wed Aug 08, 2018 9:10 pm Post subject: |
 |
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
| PeterPotkay wrote: |
Are you 1000% sure DataPower is setting the Correlation ID to the value the mainframe app expects for the reply message?
How do you know? |
Yes. Because 90% of the messages get processed and picked up. We are finding the messages to appear late and but the mainframe is able to find the messages after the timeout on its second execution. So the app logic works, its the delay that is causing the issue. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Aug 09, 2018 5:11 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ucbus1 wrote: |
So BATCHINT=0 dictates the COMMITs to happen instantaneously ...( |
No. It is up to the application to MQCMIT the messages MQPUT to a queue - this includes a transmission queue.
| ucbus1 wrote: |
and BATCHSZ parameter only helps in reducing how frequently the COMMITS are happening.
It appears like I am back to square one.  |
The values of batch size and batchinterval only affect messages sent down an mq channel. They may/will affect throughput of messages, but do not "tell" the upstream message producing app to "hurry up and commit."
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 09, 2018 7:56 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
MQ implements “store and forward”. A message must be successfully stored (committed) before it can be forwarded. Once committed, your app can no longer affect the message or its delivery.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 09, 2018 9:34 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ucbus1 wrote: |
| ...but the mainframe is able to find the messages after the timeout on its second execution. So the app logic works, its the delay that is causing the issue. |
What timeout? Where do you see evidence of a timeout?
If it's a TCP timeout, this event will have been written to the errors file on the midrange platform, and/or on the CHIN address space syslog on the z/OS mainframe.
2033 ReasonCode is not a timeout. Rather, it means that no message was available to satisfy the MQGET request.
About the mainframe consuming application:
Does the app do only one MQGET with WAITINTERVAL? If, at the end of the waitinterval, does the app try again to MQGET a message? Or, if no message is available, does the app give up trying and end itself?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ucbus1 |
| Posted: Thu Aug 09, 2018 11:06 am Post subject: |
|
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
| bruce2359 wrote: |
What timeout? Where do you see evidence of a timeout?
If it's a TCP timeout, this event will have been written to the errors file on the midrange platform, and/or on the CHIN address space syslog on the z/OS mainframe.
2033 ReasonCode is not a timeout. Rather, it means that no message was available to satisfy the MQGET request.
About the mainframe consuming application:
Does the app do only one MQGET with WAITINTERVAL? If, at the end of the waitinterval, does the app try again to MQGET a message? Or, if no message is available, does the app give up trying and end itself? |
The timeout is from the app perspective. When the app receives 2033 error code it is what the app says as timeout. The app does a GET with wait interval which we increased it to 4 sec. When it times out it does not try for the message. However there is a secondary clean up process which runs at regular intervals which GETs these response messages. Ideally we want all responses to be processed on the first pass and the clean up process not to find any response messages for processing. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 09, 2018 11:38 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ucbus1 wrote: |
The timeout is from the app perspective. When the app receives 2033 error code it is what the app says as timeout. |
2033 is a ReasonCode indicating no message available. It could be queue is empty OR there was no message that matched the msgid, correlid, groupid specified in the MQMD. 2033 is not an error.
| ucbus1 wrote: |
| The app does a GET with wait interval which we increased it to 4 sec. When it times out it does not try for the message. |
So, the app gives up AND ends itself? Does an MQCLOSE and MQDISC?
Waitinterval tells MQ to end the MQGET with Waitinterval after, in your case, 4 seconds. What if the message arrives a millisecond later?
Have you tried to send a message with IBM's supplied utility amqsput from the client platform? Does the message traverse the channel sub-second? If so, it's likely not an MQ imposed delay, is it?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 09, 2018 3:58 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
For clarity: MQGET with WAITINTERVAL does not delay the consuming app in MQGETting the message.
If a message already exists in the queue, the MQGET will not wait, and MQ will deliver the message immediately to the app with no wait at all. If a message arrives in one second, the MQGET will end immediately, and MQ will deliver the message to the app immediately.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ucbus1 |
| Posted: Thu Aug 09, 2018 6:13 pm Post subject: |
|
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
| Will test and post my findingings |
|
| Back to top |
|
|
| ucbus1 |
| Posted: Fri Sep 28, 2018 12:53 pm Post subject: |
|
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
While we are still waiting on approval to set up a prod like test environment, I would like to add some additional information.
1. The issue is happening more in the afternoon when the transaction load is high.
2. Four different datapower flows are putting the messages- using the same cluster, cluster queue manager and different cluster queues.
3. And the cluster is defined to include queue managers on Mainframe and MQ appliance. I mention this because MQ appliance is a blackbox for me if there are any known issues with it.
So I am thinking if this could be related to network latency because of high load on MQ. If so, is there way to actually see if it is related to network performance.
Last edited by ucbus1 on Fri Sep 28, 2018 1:06 pm; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Sep 28, 2018 1:04 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Your network support team will have hardware/software network monitors to measure network utilization.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ucbus1 |
| Posted: Fri Sep 28, 2018 1:07 pm Post subject: |
|
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
| bruce2359 wrote: |
| Your network support team will have hardware/software network monitors to measure network utilization. |
Thanks. I am checking with them, but wanted to understand if this could be one of the reasons of slow down we are finding |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Sep 28, 2018 7:56 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Insufficient provisioning of disk, RAM, processors, network will adversely affect performance.
Missing SLAs for new or peak workload are indicators that performance tuning is in order.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Sep 29, 2018 2:44 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Sounds awfully like the request is forgetting to commit or to post the request message outside of the current transaction.
The wait time expires and the transaction (default or global) finishes and posts the request message. The request message is now being processed and the reply message is ready to be read. Second attempt to read the reply message is successful...
This would be a flaw in the requester app that is not committing properly its request messages or not putting them outside of syncpoint... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| ucbus1 |
| Posted: Tue Oct 09, 2018 10:19 am Post subject: |
|
|
Knight
Joined: 30 Jan 2002
Posts: 560
|
One quick question on Message persistence.
I find the messages are put with persistence on. If I understand correctly persistence increases the processing time. Now can this be possible? The PUT application is putting the message correctly. But MQ is taking more time to write the message to logs due to persistence ( which could be killing us due to high I/O, hence the delay) and is not available for GET application till MQ is done writing the logs. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 09, 2018 10:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ucbus1 wrote: |
I find the messages are put with persistence on.
...
The PUT application is putting the message correctly
|
If you're implying by "the application put the message correctly", it put the message as not persistent and it's ending up on the queue as persistent then this is false. Persistence is totally under the control of the application (unless you have an exit installed someplace which is highly unlikely) and the application has 3 options:
- persistent
- not persistent
- whatever, I don't care that much one way or the other
There will now follow a flame war where the 2 sides of the debate (application sets persistence v application takes default) fight it out but if the message is ending up on the queue as persistent then the application either specified it needed to be persistent or the application allowed the persistence to be whatever the default setting is on the queue and that default is persistent.
| ucbus1 wrote: |
| If I understand correctly persistence increases the processing time. |
Yes. Much more I/O activity. Don't forget persistent messages are always written to the queue file as well as the log file; non-persistent messages are only written to the queue file if the available buffer space fills.
| ucbus1 wrote: |
| But MQ is taking more time to write the message to logs due to persistence ( which could be killing us due to high I/O, hence the delay) and is not available for GET application till MQ is done writing the logs. |
The message won't be available to be read off until it's successfully arrived. Until it's been written safely to both locations.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
