| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Multiple Sender/Receiver Channels betweem same 2 QMgrs |
« View previous topic :: View next topic » |
| Author |
Message
|
| duffMan |
 Posted: Tue Jul 30, 2002 5:05 am Post subject: Multiple Sender/Receiver Channels betweem same 2 QMgrs |
 |
|
Voyager
Joined: 03 Jun 2002
Posts: 75
|
Hi Folks,
I have a question regarding the configuration of multiple channels between the same two Queue Managers.
I have read bits and pieces of the intercommunications guide and I can't find anything which says that this cannot be done. Hell, I even tried it out and everything seemed to work fine. The manual actually suggests it for certain performance reasons.
I'm now trying to do this though a couple of firewalls and am getting TCP/IP errors, yet both channels will stay RUNNING for a period of time. They will then abnormally end and restart themselves. I know that it is likely some 'other' problem not the channel defenitions, however the MQ specialist on the other end (the company we are trying to communicate with via MQ) said "you cant define more than one channel pair between two queue managers without using different port numbers."
This prompted me to look at support pac MA86 "MQSeries and Firewalls" written by Paul F. Sehorne. The very first page suggests that only one conversation can exist on a network with a specific signature. The signature being xxxx(aaaa) yyyy(bbbb)pppp
where xxxx is the source IP and yyyy is the target IP and aaaa and bbbb are ports and pppp is the protocol.
So an example is if I have a queue manager at IP address A on port P1 and another queue manager at address B on port P2 over tcp the signature for the SENDER (from A will be):
A(P1) B(P2)TCP
The signature for the RECEIVER (from A will be):
B(P2) A(P1)TCP
If we have multiple senders from A to B, theoretically we have more than on conversation with the same signature.
Does this make any sense? If two senders were defined would they use the same conversation signature, and hence the same tcp connection?
In case you're wondering the TCP errors I'm getting are:
my RECEIVERS are ending abnormally with TCP TIMEOUT (X'0')
my SENDER is ending abnormally with TCP/IP(SEND) 10054 (X'2746')
MQ (me) , Application, Network, and "the other company" are now trading blame. This is fun.
Thanks in advance. |
|
| Back to top |
|
 |
| mrlinux |
| Posted: Tue Jul 30, 2002 6:16 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
There is no issue with having multiple channels from one QMGR to another
we have had that is one our enviroment for at least 2yrs without issues.
Our reason for doing it was we had batch channel and a fast channel. Batch channel was setup to handle large messages so that small messages which needed processing could get through without waiting on bigger messages to be processed.
I know we could have used message priority, but this was simpler to manage.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries
Last edited by mrlinux on Tue Jul 30, 2002 6:34 am; edited 1 time in total |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Jul 30, 2002 6:24 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Yes, your MQ specialists are wrong on this one. We actually ran a production system for a fortune 500 this way. To keep transmit queue levels low and throughput high on a very active queue manager, we had 8 remote queue definitions pointing to 8 different transmit queues, which used 8 different sender channels, pointing to 8 different receiver channels on another queue manager, all on the same port. For several reasons this allowed us to maintain a higher throughput than having everything go through a single sender channel.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| mrlinux |
| Posted: Tue Jul 30, 2002 6:35 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
Well firewalls may have timers on the connections to terminate them after
x amount of time.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| duffMan |
| Posted: Tue Jul 30, 2002 6:57 am Post subject: |
|
|
Voyager
Joined: 03 Jun 2002
Posts: 75
|
Thanks for the quick responses, that is also what I thought.
The support pac document kind of made me doubt myself.
We've checked our firewall and see nothing wrong, so it is probably "them"  |
|
| Back to top |
|
|
| mrlinux |
| Posted: Tue Jul 30, 2002 7:27 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
You could try sending a dummy message every 10 minutes to see if they will stay going.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| oware |
| Posted: Tue Jul 30, 2002 11:54 am Post subject: Define Multiple Channels |
|
|
Novice
Joined: 28 Jan 2002
Posts: 23
|
Hi all,
We have an application on MQ server box which retrieves message from queue Q1 defined in queue manager QM1 and inserts it into the database. Initially I Inserted up to 1000 messages into Q1 and then ran my application. I found that my application retrieves messages from Q1 (using MQGET function) and inserts into the database at rate of 9 messages per second.
Our client wants to increase the through put so that our application can retrieve 20-30 messages per second from Q1. I fine tuned my application and even MQ series attributes to achieve this result but I was not successful in doing so.
I am thinking of defining 2/3 channels for QM1 so that I can run 2/3 instances of my application which can access queue q1 and I can retrieve and insert messages into the database at 30 msg/sec.
Can anybody provide me a clue to achieve this?
Thanks in advance.
Prashant. |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Jul 30, 2002 1:04 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
If this is just a question of throughput, then you don't need to add additional channels to the system. Keep everything the way it is now, but run additional copies of your application, all getting messages from Q1 and doing the database updates. It sounds like the major bottleneck here is the processing rate of the database updates, rather than the amount of time it takes the messages to arrive on Q1 (which is what additional channels would solve).
Also, your question seems to be somewhat separate from the original thread, so I'm going to separate it and make it a new topic...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Jul 30, 2002 1:06 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Whoops.. Doesn't look like I have that level of granularity in terms of moving things around... So I guess we'll just leave it here for now! 
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| jc_squire |
| Posted: Tue Jul 30, 2002 1:31 pm Post subject: |
|
|
Centurion
Joined: 14 Apr 2002
Posts: 105
Location: New Zealand
|
Hi,
You would probably get more responses if you listed this question in the API support forum - maybe Brandon will be kind enough to move it for you ...... 
FYI - we have developed applications handling messages in excess of 250 msgs per second with default MQ attributes so suggest you seriously look at the application/db. You should try to pin point the cause of the delay e.g. are you doing db updates in units of work - it could be that your application is waiting too long for your db to complete the updates before handling the next message. The delay could also be hardware related e.g. CPU, memory, hard disk etc. What was your reason for defining more chls? I don't think chls are your problem here - defining more chls will not improve the processing of your application and db updates and you do not need a chl per instance of the application that you are running.
You can start multiple instances of your application getting messages off of the same queue but this will probably not improve performance on a single CPU server. Presume you are using the MQI - on open calls ensure your application specifies MQOO_INPUT_SHARED which enables multiple applications to open a queue. Simply start up a number of instances and see whether it improves performance. Naturally you have to ensure that your db can accept connections from all the instances you are starting.
Also - if your first instance is running continuously (not triggered) you can use triggering to start an X number of instances of your application when the queue depth reaches a certain threshold.
Hope this helps.
Regards
_________________
J C Squire
IBM Certified Specialist - MQSeries |
|
| Back to top |
|
|
| jc_squire |
| Posted: Tue Jul 30, 2002 1:35 pm Post subject: |
|
|
Centurion
Joined: 14 Apr 2002
Posts: 105
Location: New Zealand
|
ahhh .... looks like Brandon was a step ahead of me .... got waylaid with my reply - not good doing these things at work.
_________________
J C Squire
IBM Certified Specialist - MQSeries |
|
| Back to top |
|
|
| oware |
| Posted: Tue Jul 30, 2002 2:09 pm Post subject: |
|
|
Novice
Joined: 28 Jan 2002
Posts: 23
|
Hi Squire, bduncan,
Thanks for your replies.
Actually all the messages retrieved from Q1 are in XML format. I connect to DB and then execute stored procedures from my appl. by passing XML and other parameters.
To confirm whether it’s because of DB or MQ.
1]] I kept my code as it is i.e. connecting to DB, building parameters for stored procedure except I commented execution of stored procedure from my appl. Now every time I get msg. from Q1 I log time in my log files in HH:MM:SS format to make sure how many messages are retrieved from Q1 per sec. I ran my appl. and these time again the result were same i.e. 10 msg /sec.
2]] I ran two instances of my application on Q1 with one channel. I checked my log files and I showed me that one instance of my appl. retrieved 5 msgs /sec while other application retrieved 4 msgs/sec in same timeframe. So overall its 9 msgs/sec.
FYI: I am doing a non destructive get first and then a destructive get on Q1 to remove messages from Q1.
Till I get any solution, I am planning to build another application which will only do destructive get on Q1 without any database connection.
Thanks in advance,
Prashant.
[/b] |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Jul 30, 2002 3:57 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Prashant,
Why are you doing a non destructive get followed by a destructive get? I've seen many times where people go down this path and it ends up destroying the scalability of your system. If you plan on processing a message, you should remove it from the queue. I don't see any purpose in doing these browse/get architectures. Part of what slows you down is when you do the destructive get, you need to make sure you remove the same message you just browsed, which means doing an MQGET while matching against MsgId, which can certainly slow you down if the queue depth is high.
The best way to architect this system is to do a single destructive MQGET under syncpoint. Do the database update, followed by MQCMIT. If you want to make it more bulletproof, use XA, and place the database update within the same unit of work. Now you are only doing a single MQGET, and you don't need to match against anything, you simply remove the first message on the queue. If you use XA you can also profit from the ability to make your unit of work span multiple messages. For instance, you'll process 100 messages before issuing the MQCMIT, which will further increase performance.
Another question, which language is your application coded in? If you are using perl or some other interpreted language, it's going to be much slower than C. If you are really looking for high throughput, you should consider recoding it in C/C++.
But I really think it's your browse/get scheme that is killing performance. If you move to a single MQGET, you should see major improvements.
Also, I notice you did a test where you took the database update out of the equation, but what about running a test where you take MQ out of the equation and simply test the rate at which you can do database updates? Perhaps stick the XML in a flat file on the local filesystem, have your application read it into memory, and then do the database updates as fast as possible. This may reveal some inefficiencies in your database update algorithm, database settings, disk i/o, etc...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| oware |
| Posted: Wed Jul 31, 2002 6:43 am Post subject: |
|
|
Novice
Joined: 28 Jan 2002
Posts: 23
|
Thanks Brandon.
I really appreciate your suggestion and I am planning to change my code to use sync point and MQCMIT. But can you please clarify these doubts for me.
1-> when I read a message from Q1 using SYNC point and when I try to insert it into the database and find that my DB connection is broken, then will my message be lost from Q1 even if I don’t give MQCMIT. In this case I would definitely want to preserve my message on the queue.
FYI - Q1 is set to persistent
2-> you mentioned that I can read 100 msg at a time before issuing commit. Now again Say in the middle of processing of 100 msgs. say at 50th msg I lost my DB connection. In this case I would like my 50 msg which I was able to insert in DB to be removed from queue while remaining 50 msgs I want it to be in Q1 since my DB connection was broken. Is this possible in this logic.
I may be stupid not to understand what you mean by 'XA', can you describe it for me please.
Thanks once again for spending your precious time to reply.
Prashant. |
|
| Back to top |
|
|
| bduncan |
| Posted: Wed Jul 31, 2002 8:43 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Prashant,
No worries; I'm here to help everyone... To answer your questions:
1) This is the simple case. Let's say you are just doing commits after every message, and you aren't using XA. So you remove a message from the queue under syncpoint. You attempt to do your database update, and see the connection isn't available. Depending on the language you are using, you'll either catch an exception, get a bad return code, etc., which tells your application the database isn't available. Now you need to decide what you want your application to do. You can either retry connecting to the database, or if you want to give up trying, you can issue an MQBACK which will roll back the message onto the queue. In a similar situation, let's say you get the message, but instead of the database being unavailable, your application crashes before doing the database update. Well, the queue manager is smart enough to detect this, and because you got the message under syncpoint, it will automatically be rolled back. Of course, keep in mind that you don't always want to roll back. Let's say you get a message, try to do the database update, and get a database error saying the SQL statement violates a constraint. Well, this means the SQL in the message is bad, and retrying isn't going to change that. If you rolled back the message in this case, it would go back to the front of the queue, and on the next MQGET, you would end up with the same message again, and keep rolling it back, effectively causing an infinite loop. We call these poison messages.
2) To answer this, I need to explain XA. It's basically "transaction resource coordination". Now, I've explained how you can have units of work with MQSeries by using syncpoint. If you are familiar with databases, then you know you can have units of work on databases as well. But if you have a single application doing both MQ work and database work, those units of work will be separate. In other words, you'll need to issue two separate commits, or two separate rollbacks. This creates the potential for getting out of sync: what happens when the MQ commit succeeds, but then you go to do the database commit and it fails? The database update will be rolled back, but the MQ work has already been committed and there's nothing you can do about it. So we use XA to essentially combine those two units of work into a single one. And you only issue a single commit, which is atomic. If either of the underlying commits fail, the entire unit of work is backed out. When you use XA, you need a transaction coordinator. The coordinator takes care of all the finer points of XA, and in your case, either MQSeries or the database can take on this roll (assuming you are using a major database like Oracle, DB2, etc). I don't want to go into too much detail because entire books have been written on XA, but you can consult the MQSeries manuals to learn more about it. So this explanation should answer your second question. What would actually happen is on the 50th message, the insert fails, and you'd issue a rollback call. The 49 database updates you've already done would be backed out, and the 50 messages you removed from the queue would be returned to it.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
