| Author |
Message
|
| cra1gl1 |
 Posted: Tue Jun 05, 2018 11:42 am Post subject: Messages missing intermittently in cluster |
 |
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
I have a cluster of 8 qmgrs with 2 full repos. Manually defined xmit queues with DEFINE QL(CLUSTER1.XMITQ) USAGE(XMITQ) CLCHNAME(*.CLUSTER1)
QM5 and QM6 has QL(TESTQ) CLUSTER(CLUSTER1) DEFBIND(NOTFIXED)
Trying to put msgs from QM3(also tried from QM2), "amqsput TESTQ QM3" and messages are being load balanced as expected, BUT intermittently, I'm missing messages. if I put 6 msgs, 3 go to 1 qmgr and 2 go to the other and 1 is missing. Noticed this multiple times. I did check the xmitq on QM3 and deadq on the target qmgrs and the msgs aren't there.
dspmqrte -m QM3 -q TESTQ also shows no issues(puts msg on QM5 and QM6 alternaively).
Where else should i be checking?? |
|
| Back to top |
|
 |
| mvic |
| Posted: Tue Jun 05, 2018 12:23 pm Post subject: Re: Messages missing intermittently in cluster |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| This is a common thing to see if your getter app is getting outside syncpoint, then you end it with Ctrl+C, and then a message is put. The amqsget sample program gets outside syncpoint, so is susceptible to this. It trips a few people up when they are trying out MQ. How are you consuming the messages that arrive on their destination queues? |
|
| Back to top |
|
|
| cra1gl1 |
| Posted: Tue Jun 05, 2018 12:37 pm Post subject: Re: Messages missing intermittently in cluster |
|
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
[quote="mvic"]This is a common thing to see if your getter app is getting outside syncpoint, then you end it with Ctrl+C, and then a message is put. The amqsget sample program gets outside syncpoint, so is susceptible to this. It trips a few people up when they are trying out MQ. How are you consuming the messages that arrive on their destination queues?[/quote]
Damn.. that is what I’m using to get the msgs. I’ll test it again..I’ll make sure the ipprocs is 0 before I put msgs again. Thanks for your time. |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Jun 05, 2018 12:37 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
Is your TESTQ defined with DEFPSIST(YES)? From a testing stand point, that could help rule out the message was dropped in flight for some reason because it was non-persistent. amqsput does not set the persistence, and relies on the default setting of the queue.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jun 05, 2018 12:49 pm Post subject: Re: Messages missing intermittently in cluster |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| cra1gl1 wrote: |
| Trying to put msgs from QM3(also tried from QM2), "amqsput TESTQ QM3" and messages are being load balanced as expected, BUT intermittently, I'm missing messages. if I put 6 msgs, 3 go to 1 qmgr and 2 go to the other and 1 is missing. |
How did you end the amqsput program?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cra1gl1 |
| Posted: Tue Jun 05, 2018 1:12 pm Post subject: Re: Messages missing intermittently in cluster |
|
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
| bruce2359 wrote: |
| How did you end the amqsput program? |
By pressing enter twice,
And ending the amqsget with ctrl+c |
|
| Back to top |
|
|
| cra1gl1 |
| Posted: Tue Jun 05, 2018 1:14 pm Post subject: |
|
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
I can't reproduce the issue with "ALTER QL(TESTQ) DEFPSIST(YES)"
not sure if amqsget is causing the issue or something else |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jun 05, 2018 1:38 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
Maybe use something other than amqsput/amqsget.
For example, amqsblst.
To put 5 messages using syncpoint: amqsblst QM Q -W -u 1 -c 5
and
To get 5 messages using syncpoint: amqsblst QM Q -R -u 1 -c 5 |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Jun 05, 2018 4:23 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
This might have to do with NPMSPEED(FAST) on your clusrcvr channels. Try going back to non-persistent messages and a NPMSPEED setting of normal, and see if the issue goes away.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| cra1gl1 |
| Posted: Wed Jun 06, 2018 7:18 am Post subject: |
|
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
with multiple tests with amgsget and amqsblst, I noticed that
the issue doesn't occur with NPMSPEED(NORMAL) or with DEFPSIST(YES)
it happens with NPMSPEED(FAST) or with DEFPSIST(NO), with amqsget and with amqsblst- but onnly when i have to close amqsblst -R (read) with ctrl+C
Does this mean the tools are getting the msgs even after exiting (with ctrl+c) and cluster is working fine?
or is there something else i need to worry about? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 06, 2018 7:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cra1gl1 wrote: |
| or is there something else i need to worry about? |
DEFPSIST(NO) means messages added to the queue will not be persistent unless the putting application deliberately sets it (which the amqsput sample does not)
NPMSPEED(FAST) means increase the message transmission rate at the cost of message safety, and the description of this option states that non-persistent messages are at risk in the event of problems.
Using cntl+C is a forced (problematic) close of the application. Hence the software will examine it's options for cleaning up the problem, will discover that the configuration indicates the messages are not important, and take the easy route of flushing messages and starting clean.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jun 06, 2018 7:56 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
Hearing that the NPMSPEED and/or DEFPSIST setting is important, then the implication would be that the channel is losing a non-persistent message, but I find this surprising.
I would only expect a channel to lose a non-persistent message in channel abend scenarios, no matter the setting of NPMSPEED (maybe someone can correct me, though).
If you used -R -u 1 on amqsblst then a reading of the source code of that sample app suggests that MQGMO_SYNCPOINT would be used, hence no opportunity for the application message loss that I first referred to.
So, were you using -u 1 ? If you were, then that's the 2nd time I am surprised today. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jun 06, 2018 8:02 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| Vitor wrote: |
| Using cntl+C is a forced (problematic) close of the application. Hence the software will examine it's options for cleaning up the problem, will discover that the configuration indicates the messages are not important, and take the easy route of flushing messages and starting clean. |
If the MQGET is done within syncpoint then you can "safely" abend the app without risk of losing a message inflight between qmgr and app.
The same thing affects NonPersistent and Persistent messages, though.
Conclusion: use syncpoint if your app ever abends (which most apps do) and you care about your messages. Or, another alternative is syncpoint-if-persistent if the app is handling both types, and wants to handle NonPersistent msgs faster with lower QOS. |
|
| Back to top |
|
|
| tczielke |
| Posted: Wed Jun 06, 2018 8:45 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
When I saw this post, I had a case of deja vu. I remember seeing the same issue that cra1gl1 is reporting, when also doing just simple cluster PUT tests a few years ago. NPMSPEED(FAST) was dropping non-persistent messages for even these simple test cases with amqsput. There probably is a "bug" that NPMSPEED(FAST) would do this for such a trivial test case, but it is working as designed, according to the specifications of NPMSPEED(FAST).
Personally, I plan on changing all our cluster receiver channels to run with the default of NPMSPEED(NORMAL). I prefer that as the default, as quality of service should trump speed for a default, in my opinion. We also don't have speed requirements that would warrant this trade off.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| cra1gl1 |
| Posted: Wed Jun 06, 2018 9:06 am Post subject: |
|
|
Apprentice
Joined: 25 Apr 2018
Posts: 30
|
Thanks for your time guys.
@mvic, you're right. The issue doesn't occur with "-R -u 1" (I missed -u 1 in earlier tests.)
CONCLUSION:
The issue doesn't occur with NPMSPEED(NORMAL) or with DEFPSIST(YES) (no matter what tools i use to GET the msgs)
Issue occurs with NPMSPEED(FAST) or with DEFPSIST(NO) when the getting app abends( amqsget). |
|
| Back to top |
|
|
|
|
