| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Mainframe Batch Application (BATCHSZ) |
« View previous topic :: View next topic » |
| Author |
Message
|
| RHeunes |
 Posted: Thu Nov 23, 2023 1:19 am Post subject: Mainframe Batch Application (BATCHSZ) |
 |
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
Good day,
I have a question regarding how the BATCHSZ is fulfilled when it comes to MQ and Z/OS batch applications.
Environment Details:
MQ/ACE Cluster on AIX - Called ESB from here on out
MQ on z/OS (same cluster - partial repo)
Front end (FE) queries (request) via ESB to the z/OS application for person data stored on the mainframe (MF)
z/OS batch application then responds to ESB and ESB passes response to front end.
Allowed time for round trip is 40 seconds (REQ/RES)
Problem:
During the response from the MF to the ESB we see uncommitted messages and the delays start occurring, now I do not manage the MF app or z/OS, I only look after the cluster aspect of the PR queue manager, and my suspicion is that the MF application is slow but I need to be able to prove this is the case, the MF team is saying there is no issues on their end. (Not excluding networks here but I don't think that is the case its only happening on MF interfaces)
So to give you an understanding of the request/response steps it is as follows FEA > ESB > MF (REQ) and MF > ESB > FEA (RES) the delay occurs at the MF > ESB > ACP part but if we were to go look at a transaction log it would appear the ESB is holding messages for no reason and then passing them on to ACP, which would look like an ESB problem to the naked eye, but I believe the MF application code is either explicitly fulfilling the BATCHSZ(50) requirement hence there is no sync point between the two queue manager and the delay of sending it to FEA.
From and ESB perspective, there are 22 EGs that communicate to various different systems and only 2 EGs that speak to MF, there is no slowness on any other EG and the cpu, mem, iostats, OS, logs (MQ/ACE) indicate no symptom that it may be the ESB in this case.
My thinking is, that it is the BATCHSZ and uncommitted messages causing the delays due to slowness on the MF.
I might also be wrong in my analysis
Regards,
Rayn |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Nov 23, 2023 5:50 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9400
Location: US: west coast, almost. Otherwise, enroute.
|
Display channel status. What are the BATCHSZ and batch interval values?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Nov 23, 2023 2:33 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
Evidence of uncommitted messages usually indicates long-running transactions that could be caused by delays in other processing. eg. web service calls. Have a close look at the QSTATUS (type QUEUE and HANDLE) of the queue.
_________________
Glenn |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Nov 23, 2023 7:15 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1916
Location: Bay of Plenty, New Zealand
|
Unless you are on a fairly old queue manager, a BATCHSZ target of 50 will be tempered by the BATCHLIM data quantity size.
You have already been told to check QSTATUS for UNCOM(YES), and I would reiterate that request.
However, that said, an MQ Sender Channel will not wait around until BATCHSZ reaches 50 before committing the batch of messages. It's a maximum not a target! As soon as the transmission queue reaches the empty state the MQ Sender channel will start end-of-batch processing which should cause the batch of messages, even if only a single message, to be committed at both ends of the channel.
Another thing to look into... Is there any possibility that one of the applications in this pipeline is not aware that it is using SYNCPOINT? I ask this because it is not unheard of for an application either developed on distributed platforms, or using a client and connecting from a distributed platform to have left the SYNCPOINT option to be the default instead of explicitly coding either MQPMO_SYNPOINT or MQPMO_NO_SYNCPOINT.
When such an application connected to a distributed queue manager, it would be using NO_SYNCPOINT.
When such an application connects to a z/OS queue manager, it would be using SYNCPOINT and would not have an explicit MQCMIT call. This would mean that the messages it MQPUT would only be committed when it ended, perhaps after having given up waiting for 30 seconds or so. Is there any likelihood that you have such an application in your workflow here?
It would be very interesting, for you and us to help you, to know which queue or queues show UNCOM(YES) or UNCOM(number) (on distributed only) in the workflow. Or indeed, where the build-up of messages is. I assume each of your '>' symbols equates to a queue? However, perhaps also you have channels and transmission queues in the picture too since you mention BATCHSZ. Can you map out a slightly more extended workflow showing the queues, channels, and where the depths/uncommitted messages are for us?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| RHeunes |
| Posted: Fri Nov 24, 2023 4:42 am Post subject: |
|
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
Thanks for the replies, my thinking was correct, The MF had additional batches running, and they do accept 3rd party requests via web service calls, the MF admin said the batches spiked the cpu usage to 100% (why it takes 4 days to figure that out is something else)
So the ESB was waiting for messages to be processed on the MF before passing it along to FEA. (this tells me its explicitly waiting for the completion of the batch, I just don't know if that is the case)
@hughson, the > indicates steps within the flow, which in turn points to queues, that is how our transaction log will log it. Example: Persondata Request/Response: FEA > ESB (REQ_REC) ESB > MF (REQ_SND) MF > ESB (RES_REC) and ESB > FEA (RES_SND)
The env has 2 main qmgr/broker (FRs) and multiple PR qmgrs that speak to the different entities, with that we have MQ Client to MQ Server and MQ to MQ.
I am not involved on the MF side (except that their qmgr is part of our cluster but it lives on the MF not AIX, I have access to it via explorer) so I cannot comment on the coding of that application.
As for the build up it was evident on the request queues on the MF qmgr (which makes sense that it was a performance issue, but it apparently wasn't proof enough  ) the other build up came from the ESB RES queue and there were uncommitted messages (output count was MF sender channel, input count is broker flow sending to FEA qmgr) and message age ranged between 5 seconds to 60 as an example, should be 0)
I agree BATCHSZ is a max and not a target but can the MF application code override that somehow?
See I want to know exactly what is causing the problem and send a recommendation to them, these offlines affect the national offices and is not good, unfortunately we have actors that are not forthcoming with info as to what the issue is and being on the ESB side we interface with a lot of different systems, but the MF is crucial to these systems as it provides the person data to further applications from the offices, so no MF no working offices, albeit intermittent offlines, it is not good.
So the main question is, can the z/os application override how the bacthing is committed between the two queue managers
and, is there anything I can do on ESBs side to "performance tune" that?
I can confirm ESB and Networks are not to blame here.
@bruce2539, the batchsz is 50 and batch interval is 0.
@gbaddeley, I agree this is causing the issue on MF side. They have complained about 3rd parties abusing the person data service, these 3rd parties come through web services I believe, there is also some digitization project that can affect it, and the MF is used by a few government departments, not just us, so to me it looks like a resource issue.
Regards,
Rayn |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Nov 24, 2023 2:38 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1916
Location: Bay of Plenty, New Zealand
|
Can I just clarify - when you talk about Batches and BATCHSZ - are you talking about MQ sender/receiver channels, or are you talking about the way your applications use transactions?
Reading your last response leads me to think that the batches you are referring to have nothing to do with channels. Looking for confirmation before I reply to something that is not what you are even talking about! 
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Nov 25, 2023 12:22 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
It's quite annoying that the number of uncommitted messages is not viewable on z/OS queues (it is on all other platforms).
I opened a RFE for this but it's been rejected.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Nov 26, 2023 2:34 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
| zpat wrote: |
It's quite annoying that the number of uncommitted messages is not viewable on z/OS queues (it is on all other platforms).
I opened a RFE for this but it's been rejected. |
The difference between CURDEPTH and the number of messages that can be browsed is usually the number of uncommitted messages.
_________________
Glenn |
|
| Back to top |
|
|
| RHeunes |
| Posted: Mon Nov 27, 2023 1:50 am Post subject: |
|
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
@hughson
I am referring to the batch application and the batchsz (channels), so when MF replies to ESB and MF has a batch application, can that application explicitly enforce or wait for the batch to complete (some form of put message options?), rendering the batchsz attribute (not a target but max) useless?
See if the I dropped the batchsz to 25 as an example for the channel between MF and ESB it doesn't make a difference, my thinking was if MF is experiencing constraints then a lower batchsz of 25 could help for the commits. (?)
Unfortunately the issue occurred again this morning, and it was the MF cpu utilization that was 100% again, I am looking for a solution to recommend to the MF team. I have no understanding of their code of the batch app so hence my questions on how it would work with batchsz
Hope that makes sense.
Regards,
Rayn |
|
| Back to top |
|
|
| hughson |
| Posted: Mon Nov 27, 2023 2:01 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1916
Location: Bay of Plenty, New Zealand
|
BATCHSZ is a parameter on a sender channel which controls how many messages there are in the syncpoint before the channel commits them all.
This is a different transaction to the one used by the batch application.
The batch application cannot affect the size of the channel's number of messages, except in the case where it does not commit its own transaction and thus the channel sees an empty transmission queue and thus commits its own transaction.
In other words, until the batch application commits its transaction, the MQ Sender channel won't even SEE the messages to be able to get them and send them.
That said, please note that the channel only puts persistent messages in a transaction. Non-persistent messages are put outside of the transaction and are available immediately at the receiver end - no need to wait for a commit. You haven't mentioned persistence, but I assume all the messages we are talking about are persistent.
You say dropping the channel's BATCHSZ to 25 doesn't make any difference, however, you haven't indicated what your channel is getting as the achieved batch size. Do you know what that number is? If you're unsure how to find it, read IBM MQ Little Gem #1: XBATCHSZ
If the achieved batch size was only, say 10, then yes indeed, setting the maximum to 25 would have no effect at all.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Nov 27, 2023 2:42 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
| RHeunes wrote: |
...
Unfortunately the issue occurred again this morning, and it was the MF cpu utilization that was 100% again, I am looking for a solution to recommend to the MF team. |
Hi Rayn, the MF team should be concerned about CPU reaching 100%, as this will impact the performance of all subsystems (including MQ), processes and jobs. Work out what is causing 100% and resolve the issue.
_________________
Glenn |
|
| Back to top |
|
|
| RHeunes |
| Posted: Mon Nov 27, 2023 11:43 pm Post subject: |
|
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
@hughson,
I had a look at XBATCHSZ values they fluctuated a bit, but generally in and around 1,13 and 13,14 sometimes around 31,6 and 21,4. I am thinking of changing the BATCHSZ to 15 to see how it performs, Is it ok to make it 15? will it create more overhead for the MF?
Question, would setting it to 1 be a bad idea? I read that if you have network issues between two qmgrs, that it can be beneficial in such an instance.
Regards,
Rayn
Last edited by RHeunes on Mon Nov 27, 2023 11:52 pm; edited 1 time in total |
|
| Back to top |
|
|
| RHeunes |
| Posted: Mon Nov 27, 2023 11:50 pm Post subject: |
|
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
| gbaddeley wrote: |
| RHeunes wrote: |
...
Unfortunately the issue occurred again this morning, and it was the MF cpu utilization that was 100% again, I am looking for a solution to recommend to the MF team. |
Hi Rayn, the MF team should be concerned about CPU reaching 100%, as this will impact the performance of all subsystems (including MQ), processes and jobs. Work out what is causing 100% and resolve the issue. |
@gbaddeley
It would seem the MF is overburdened, but yes I do agree there is where the solution lies.
Regards,
Rayn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 28, 2023 9:17 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9400
Location: US: west coast, almost. Otherwise, enroute.
|
| RHeunes wrote: |
| gbaddeley wrote: |
| RHeunes wrote: |
...
Unfortunately the issue occurred again this morning, and it was the MF cpu utilization that was 100% again, I am looking for a solution to recommend to the MF team. |
Hi Rayn, the MF team should be concerned about CPU reaching 100%, as this will impact the performance of all subsystems (including MQ), processes and jobs. Work out what is causing 100% and resolve the issue. |
@gbaddeley
It would seem the MF is overburdened, but yes I do agree there is where the solution lies.
Regards,
Rayn |
Overburdened mainframe. Don't often hear that phrase ... perhaps under-provisioned LPAR.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| RHeunes |
| Posted: Wed Nov 29, 2023 12:23 am Post subject: |
|
|
Novice
Joined: 13 Sep 2021
Posts: 14
|
@bruce2359
tomeito tomahto?  |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
