| Author |
Message
|
| vijiraghav |
 Posted: Mon Feb 23, 2004 11:16 pm Post subject: MQClient problem |
 |
|
Novice
Joined: 11 Nov 2003
Posts: 18
|
have the following MQclient-MQserver-MQclient architecture setup in our office.
Multiple (around 20) call center agents which are MQ Client applications installed on various Win 2000 machines (users) generating request (both enquiry and transaction posting) messages put (MQPUT) onto an input queue (INPUT.IB.QUEUE) and waits (MQGET with Wait option) for the response message from output queue (OUTPUT.IB.QUEUE). These queues are available in a queue manager QM1 on another WIN 2000 machine i.e. MQServer.
There is a single client (MQ CLIENT) application which is continuous running on the HP-UX (UNIX) server machine which has the backend application system (where the actual data is available for enquiry and posting i.e. it is a file system and the data are in the form of files). This application is always connected to the MQServer queue manager, gets the message from OUTPUT.IB.QUEUE, processes it, generates response messages and puts (MQPUT) to INPUT.IB.QUEUE.
All the clients use the same server conection channel CH1 which is defined on each agent as well as HP-UX backend machine pointing to the MQServer machine in the form of environment variable and using a single port 1414.
The front-end clients (agents) pick up the response messages (corresponding to request each has sent) and displays on the screen.
The problem is: When mutiple clients try to put messages, they do not get the response within the wait interval time (which is configurable on each agent machine) sometimes. The application error log shows the reason code 2033. This is not happening always and sometimes, the timeout happens. Roughly out of 100 messages 1 message times out. I have even increased the timeout from 15 seconds to 30 seconds, but the result is more or less same. The timed out response messages lie in the OUTPUT.IB.QUEUE.
I want to know why the messages arrive late to the output queue. Is it because of the backend client application processing speed or due to network traffic or multiple clients trying to send messages through a single port 1414 ? How to resolve the problem? |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Tue Feb 24, 2004 4:56 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
Is it because of the backend client application processing speed....
|
Probably
| Quote: |
Is it because ... (of) network traffic?
|
Remote possability. How far apart are the clients to the server? LAN? MAN? WAN? Internet?
| Quote: |
Is it because... (of) multiple clients trying to send messages through a single port 1414 ?
|
No. The # of clients is not enough to cause an issue.
| Quote: |
How to resolve the problem?
|
The first step is to prove how long the back end app takes to do its thing.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| vijiraghav |
| Posted: Tue Feb 24, 2004 5:37 am Post subject: MQClient problem |
|
|
Novice
Joined: 11 Nov 2003
Posts: 18
|
Thanks Peter.
When only one client is connected and tries to send the message to backend through MQServer, the response is received within less than a second speed or just in a second. This is evident from the application log written in each of the client pc. When multiple clients are connected and many try to post the message, some of them are not getting the response in time and in such case, the error message with code 2033 is recorded in the log. This happens at highly unpredictable time interval which is difficult to predict.
The clients and MQ Server are located on the same floor separated by a very short distance. They are connected by high-speed LAN (I think 100 Mbps speed).
Vijiraghav |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Feb 24, 2004 7:00 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
You still didn't answer how long the back end takes. If it only takes 1 second, a single client is handle quickly. If 20 hit it at the same time, the 20th request will take 20 seconds.
And maybe some of those 20 may take longer than 1 second, in which case it takes even longer.
MQ is always blamed first, when most of the time its one of the apps on either side of the link that is slow.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| gunter |
| Posted: Tue Feb 24, 2004 1:45 pm Post subject: |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
The described architecture isn't designed for perforemance.
- read with 20 application from one queue is only possible if you use correlatiion-id or message-id
- client-connections slower than server-connections
- messages are persistent ?
- you are using syncpoints ?
If it's your application, use one queue for one application and use reply-to-queue.
Put the Queuemanager on the HP-UX and use direct connection and avoid persistent messages if possible. |
|
| Back to top |
|
|
| vijiraghav |
| Posted: Tue Feb 24, 2004 10:17 pm Post subject: MQClient Problem |
|
|
Novice
Joined: 11 Nov 2003
Posts: 18
|
Thanks Mr. Peter,
The backend normally takes 1 second (approx) i.e. in 1 second the requesting application gets the response. This is true when only one client is in operation. As told by you, when multiple clients simultaneously put the messages, some of them get timed-out. I have just decided to increase the time-out period to 30 seconds or 45 seconds and observe whether there is any improvment (reduction in the rate of timed-out responses).
Mr. Gunter,
Thanks for your responses and suggestions.
Here are the answers for your questions:
read with 20 application from one queue is only possible if you use correlatiion-id or message-id
Ans: Yes. The response generating program uses the message-id of the request messages and moves to correlation-id of the response message, as the agent (requesting applications) is able to pick the right response among the many messages that arrive to the output queue
client-connections slower than server-connections
Ans: We suspect the same
messages are persistent ?
Ans: Yes. They are persistent
you are using syncpoints ?
Ans: I have no idea about it as the source code is not with us. It has been developed by the vendor of the backend banking system and they have given only the executable.
The idea of installing MQ on HP-UX and creating the complete setup are quite right but we cannot do it now due to cost and time factors and the necessity of uninterrupted running of the applications. This should have been in the beginning itself.
Vijiraghav |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Feb 25, 2004 5:44 am Post subject: Re: MQClient Problem |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| vijiraghav wrote: |
Thanks Mr. Peter,
The backend normally takes 1 second (approx) i.e. in 1 second the requesting application gets the response. This is true when only one client is in operation. As told by you, when multiple clients simultaneously put the messages, some of them get timed-out. I have just decided to increase the time-out period to 30 seconds or 45 seconds and observe whether there is any improvment (reduction in the rate of timed-out responses).
|
Let's do some simple math. 1 message takes 1 second to process. 60 messages takes 60 seconds to process. If you have 40 clients putting messages on the queue, then 40 seconds will elapse until the last client gets it's response.
So, if you know how many clients you have, you know how long to set the timeout to ensure that NO messages timeout.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Wed Feb 25, 2004 6:02 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| It sounds to me like you need to consider adding another instance of the program to service the input requests (and maybe even redesigning the program). You might want to consider triggering by depth. That way the number of instances running would be semi-configurable. |
|
| Back to top |
|
|
| gunter |
| Posted: Thu Feb 26, 2004 1:42 am Post subject: |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
Did you check your queue-depth? It's possible that you have a problem with queueing. If any MQGET is slower then the MQPUT you will get more and nore messages in the queue.
MQPUT 10 msg/sec, MQGET 5 msg/sec, you will get 5 msg more per second and an additional delay 1 sec. After 30 second you have 30 seconds delay and 150 msg in the queue. |
|
| Back to top |
|
|
| vijiraghav |
| Posted: Thu Feb 26, 2004 2:49 am Post subject: MQClient Problem |
|
|
Novice
Joined: 11 Nov 2003
Posts: 18
|
I have been monitoring the queue from which the client (agents) have to get (MQGET) the message. The delayed messages (which arrive after the timeout period) get accumulated (approx. 40 to 50 messages every day) and have to be cleared every day. As the response program uses messageid-correlid algorithm, the agent program will be able to pickup the correct response even though there are many messages in the queue.
vijiraghav |
|
| Back to top |
|
|
| gunter |
| Posted: Thu Feb 26, 2004 4:32 am Post subject: Re: MQClient Problem |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
You have 20 applications and one server, it's also possible, that the problem is on the other queue.
Find out, where the lost time is. Use the trace, if possible or monitor the queues.
Do you know the Sample API exit ? If you install this for the listener, you will get a trace for all Messages. Maybe you have to modify it to avoid trace you not need. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 26, 2004 5:09 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
No need for traces, like Jeff said, this is simple math. The backend process is very slow. Either increase the time the front end clients wait, speed up the back end process, or like bower said, have multiple instances of the back end app.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| vijiraghav |
| Posted: Fri Feb 27, 2004 5:11 am Post subject: MQClient Problem |
|
|
Novice
Joined: 11 Nov 2003
Posts: 18
|
Thanks Mr. Peter Porkay
I have increased the timeout interval to 45 seconds (that's all maximum best I can do at this time) and the timeout occurrence has reduced a little bit now. Regarding the increase the speed of backend program or increase the no. of instances, I need to consult with the vendor of the backend program.
vijiraghav |
|
| Back to top |
|
|
|
|
