| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Message Group and Channel Initiator |
« View previous topic :: View next topic » |
| Author |
Message
|
| MrVMax |
 Posted: Thu Jun 16, 2016 9:19 am Post subject: Message Group and Channel Initiator |
 |
|
Novice
Joined: 01 Aug 2013
Posts: 23
|
Hi all,
I'm in the need of some help to identify the best way to use message grouping against a remote queue.
This is the environment:
OS Linux fedora release 22 kernel 4.04-301.fc22.x86_64
Mqseries 7.5.0.2
2 Queue Manager QM1 and QM2
1 remote Queue RQ1 on QM1 that points to the local queue LQ2 on QM2 througth the XMITQ TXQ.QM2 and CHANNEL QM1.TO.QM2
2 applications:
1) PGM1 that writes on RQ1 4 messages in group and commit after the
put of the forth message that has the MQGS_LAST_MSG_IN_GROUP
2) PGM2 that writes on RQ1 one single message at time and commit;
The channel has 30 as DISCINT and normally is inactive
The programs are the sample code with little or no change.
When the PGM1 puts the first 3 messages on RQ1 without putting the last message of the group, the messages go directly to the TXQ.QM2 queue without starting the channel even if We have observed that the Queue Manager QM1 sends a message with MQTM structure on the SYSTEM.CHANNEL.INITQ ... and this is what I supposed it would happen all the messages are on the right queues but unavailable for MQGET.
When the PGM2 puts a message to the same remote queue the message goes to the same trasmission queue without putting a new message with MQTM structure on the SYSTEM.CHANNEL.INITQ to rise the channel.
So the channel isn't supposed to start automatically till the PGM1 puts the forth message on the queue with MQGS_LAST_MSG_IN_GROUP.
May be this is the supposed way to work ... but if we start the channel
througth runmqsc command the message written by PGM2 reaches the
LQ2 on QM2... so we are searching if there is a way to avoid waiting for the
PGM1's MQGS_LAST_MSG_IN_GROUP to let the other program to start communication.
Thanks in advance for any advice. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Jun 16, 2016 9:41 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
the trigger to start the channel does not fire until prog1 commits its messages OR some other app puts/commits a msg into xmitq.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Jun 16, 2016 8:45 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Putting all the messages of a group inside a syncpoint is an appropriate thing to do anyway, and it will additionally mean that the channel won't see them to move until they are all there.
That said, channels don't move messages in groups, they treat each message as a single message, so you cannot make the channel move all your messages from one group inside one batch. You must ensure that your getting application at the remote end does a get with complete group.
Remember also that channels are not generally dedicated to one application, so your channel may already be running due to messages from another application going down the same route.
Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| MrVMax |
| Posted: Fri Jun 17, 2016 1:40 am Post subject: |
|
|
Novice
Joined: 01 Aug 2013
Posts: 23
|
Thanks to Bruce and Morag I perfectly agree with what Morag wrote in her 3 sentences but not with the one of Bruce, but anyway the question is slightly different, but I suppose my statements mislead You about the target that I would acquire.
I would like to know if there is a way to make the sender channel QM1.TO.QM2 start through the channel initiator program in the case described:
the SENDER channel QM1.TO.QM2 is in the INACTIVE state.
PGM1 on QM1 starts to send messages in group to QM2 Queue LQ2 using the channel QM1.TO.QM2 at time X and will continue for sometime and commit the transaction after sending of Y messages last of them has the MQGS_LAST_MSG_IN_GROUP attribute at time X + 30.
PGM2 on QM1 sends a message (NOT IN GROUP) committed at time X+10 to the same remote Queue Manager through the same channel QM1.TO.QM2
The focus is on the same xmit queue and channel for both the communications,
the presence of a TM message on the SYSTEM.CHANNEL.INITQ after the first message sent by PGM1 (also if the trigger message is not available for read) prevents the Queue Manager to put a second TM and available message to the SYSTEM.CHANNEL.INITQ at time X+10 making the channel start wait to the time X+30 while it could more efficiently starts at time X+10.
This seems to me not inline with the fact that if I start the channel with a manual runmqsc command the messages committed by PGM2 runs through the network to their destination.
If the intention is to preserve the order of messages the manual start of the channel and following runs of PGM2 messages is inconsistent.
If the intention is to make the PGM2 available messages run the missed generation of the second TM message is inconsistent.
Am I wrong in those sentences or there is a way to avoid the useless wait or the supposed way to work is that one ? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jun 17, 2016 5:29 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| MrVMax wrote: |
PGM1 on QM1 starts to send messages in group to QM2 Queue LQ2 using the channel QM1.TO.QM2 at time X and will continue for sometime and commit the transaction after sending of Y messages last of them has the MQGS_LAST_MSG_IN_GROUP attribute at time X + 30. |
For purposes of precision and clarity...
User application programs to NOT send a message across channels to a destination queue.
1. User application programs MQPUT a message on a local transmission queue.
2. The sending side MCA MQGETs a message from a transmission queue, and transmits a message on the channel to the receiving side MCA.
3. The receiving side MCA receives a message from the channel, and MQPUTs a message to a destination queue.
Three discrete activities, three discrete "transactions," three discrete units-of-work, are involved.
[edit]
Multiple apps on the sender side may concurrently MQPUT a message to the transmission queue, and messages from those apps may/will be interleaved with messages from the other apps. The sender side MCA transmits messages as they are made available (MQCMIT) in the transmission queue.
[/edit]
Does PGM1 use the SYNCPOINT option when it MQPUTs a message on the transmission queue? If so, the message cannot be consumed (MQGET) by the sending side MCA until PGM1 MQCMITs the message.
What TRIGTYPE does you transmission queue specify?
Does your sender channel specify BATCHSIZE? If so, the MCA may be delayed in sending messages down the channel until n messages arrive to compose the batch.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Fri Jun 17, 2016 12:38 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
MrVMax,
I am not aware of a way to close the window you mention. I think one could come up with a good argument that MQ should behave differently regarding trigger messages and transactions for transmission queues than for other queues. After all, the Queue Manager already makes some adjustments for triggering such as not requiring a process object.
For those that don't know triggering that well and are following this thread. When MQ generates a trigger message it is created in the same transaction (effectively) as the message that caused the trigger. When that message is committed the trigger message is also committed. This means that the trigger message doesn't 'arrive' on the initiation queue, and therefore the channel starts, before messages are available to process. However, the odd thing is that if the message is backed out the trigger message is still committed. This is to ensure that any trigger messages lost in the window MrVMax mentions are still processed.
However, one could argue that since the queue manager is going to send the trigger anyway and we know that channels are likely to hang around for some considerable time that in the case of a transmission queue the trigger message should be generated outside of a unit of work anyway. The danger of this, of course, is that if the transaction lasts longer than channel disconnect interval then you get a missed trigger event. So, perhaps some combination of the two is more appropriate.
I guess the question is how often, in reality, the window MrVMax mentions is actually seen in real life, or is this an intellectual exercise. In reality once a channel starts it should run for for a long time so the actual start events are generally very low compared to the number of transmitted messages. In addition multi message transactions tend to be fairly short lived not spanning 30 seconds or more. However, if a customer really does have a scenario where this is happening a lot then it might be worth talking with MQ Development to see whether a slight modification to transmission queue triggering could solve the problem.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jun 17, 2016 6:23 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Increasing DISCINT should address messages beyond the first triggering channel start. A simple MQSC script command can initially start the channel at qmgr startup.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| MrVMax |
| Posted: Sun Jun 19, 2016 8:50 am Post subject: |
|
|
Novice
Joined: 01 Aug 2013
Posts: 23
|
First I thank PaulClarke because He centered the point of the problem ...
As everyone knows Murphy's Law stay all the time waiting for condition in which the damage it could cause is the maximum ...
Try to immagine what would happen if PGM1 should wait to send the last message in group on an event outside the control of the MQ administrator like the end of a sequence BEGIN BACKUP ... END BACKUP on an oracle database:
Of course this is a catastrophic vision, but connected to a real case. In this scenario the channel could start a lot of time later it should have.
So, since it's clear that it could append, and that there isn't a way beside that to use active channels almost all the time (having more than one hundred channels active at the same time doesn't attract me) to avoid the potential trouble I started to think how to solve it.
I've 3 very different ideas about that:
1) embedding into the shell script that starts the PGM an MQSC script command to start the channel at the end of the PGM;
2) insert inside the 'C' libraries that programmer uses, inside the exit routine a a call to MQPUT of a TMC2 format message to the initq to start the channel;
3) try to find if an exit routine on MQCMIT Api could resolve all.
it is clear that the complexity of the solution is increased by an order of magnitude for the transition from one solution to the next one and I think i will choice the 2 solution, but the idea of having the solution almost inside the MQ system instead of depends on the implementation of the programs still attract me.
Is really hard to solve the problem with an API Exit ?
I've no problem with the knowledge of the documented MQ structure or the 'C' language itself. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jun 19, 2016 10:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Developers should not involve themselves with channels and channel operation.
The preferred channel state is RUNNING, but RUNNING only indicates that the channel is either currently transmitting a batch of messages, OR that it believes it can do so the next time it tries. The channel could fail at the next attempt. What then?
What, in you scenario, will you or your apps do if the channel fails for external reason - TCP protocol error, equipment error, cat cable cut or unplugged?
If your answer is that you will rely on MQs built-in channel management and error recovery, then you should rely on it from the outset.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Sun Jun 19, 2016 11:37 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
MrVMax,
Your level of concern seems to indicate that you are expecting this to happen often. How long, really, are your transactions ? Why are they that long and is there something you can do in your application to shorten them? Having really long transactions (in terms of elapsed time) is never a good idea.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jun 19, 2016 12:51 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| PaulClarke wrote: |
MrVMax,
Your level of concern seems to indicate that you are expecting this to happen often. How long, really, are your transactions ? Why are they that long and is there something you can do in your application to shorten them? Having really long transactions (in terms of elapsed time) is never a good idea.
Cheers,
Paul. |
Have you identified precisely where the delay or delays is/are?
For example, is the delay caused by the end-user not completing his/her input? If this is the case, transaction managers can often impose a timeout, and if exceeded roll-back the entire transaction.
Is the delay in DB processing? If so, perhaps indices are being built in-flight, rather than by DBAs.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| MrVMax |
| Posted: Sun Jun 19, 2016 2:11 pm Post subject: |
|
|
Novice
Joined: 01 Aug 2013
Posts: 23
|
Well the environment is not really simple ...
I'm in charge of the administration of a MQ infrastructure of a customer of my company and have no access to the administration of databases or operating systems.
The customer has a lot of software suppliers for different branch of his organization most of them completely lacking any knowledge of MQ but with good knowledge of database and SQL.
To simplify their work and to extend the use of MQSeries inside the customer, we developed user functions, in all the databases used, to transform the rows of a table in MQ messages and sent them to one QM that works as the HUB of the MQ network. In the opposite way on the arrive of messages at the final destination we convert messages into rows of a db table.
Who writes on the input tables and who reads the output tables are application from other company that usually administers their databases from themselves so what I'm searching to do is find a way to prevent that some "error" or "event" outside our control could stop message flows not really involved in the event.
Of course agreement between administrators of the different servers are established about service time, backup time etc, but when the companies and people involved are a lot and every company has its own needs it could happen that someone forget what happen in the communication infrastructure while heavy load task works on database.
I hope this could explain my concerns about the strange behavior of triggered channel start with message group. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jun 19, 2016 2:39 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| MrVMax wrote: |
I hope this could explain my concerns about the strange behavior of triggered channel start with message group. |
Well, not so far. Messages arrive in the xmitq in the order they were MQPUT by applications. The first message MQPUT outside SYNCPOINT will fire the trigger.
| MrVMax wrote: |
The channel has 30 as DISCINT and normally is inactive |
From: https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q015650_.htm
| Quote: |
| If you want to keep channels up continuously, set DISCINT(0) but remember that triggering remains the preferred option. Some shops need to minimize response times during the business day and so set DISCINT to a value that allows the channels to time out at night but generally keeps them running all day. If for some reason you have triggering set up right and the channels go down prior to DISCIINT you should be able to check in the error logs for the reason why |
.
What is in your error logs for this channel?
How often do messages arrive on this channel? Continuously? Every minute? Every 5 minutes or so? Every hour or so? DISCINT of 30 is too low. Set it to a value so that the channel goes INACTIVE only after extended idle periods.
What is your channel HBINT value?
What is your channel BATCHINT value?
This is a worthwhile read for you - MD0C: WebSphere MQ - Keeping Channels Up and Running: http://www-01.ibm.com/support/docview.wss?uid=swg24006699
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
