| Author |
Message
|
| Pringle |
 Posted: Tue Jan 15, 2013 7:33 am Post subject: c# MQRC_SSL_INITIALIZATION_ERROR when multi-threading |
 |
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
Hi,
We have an application that allows communication with WebSphere MQ. It provides a UI where some basic queue properties can be entered:
* Queue Manager
* Queue Channel
* Machine Name
* Queue Port
* Queue Name
* SSL Cert Store
* SSL Cipher Spec
One of the users of our application is hitting the error MQRC_SSL_INITIALIZATION_ERROR (2393). In order to get to the root cause of this issue, I have now written a tiny test app to boil the issue down to its simplest form.
There are only two methods that interact with the queue:
| Code: |
public static void OpenQueue(QueueProperties queueProperties, ref MQQueueManager queueManager, ref MQQueue queue)
{
Hashtable props = new Hashtable();
props.Add(MQC.SSL_CERT_STORE_PROPERTY, queueProperties.SSLKeystorePath);
props.Add(MQC.SSL_CIPHER_SPEC_PROPERTY, queueProperties.SSLCipherSpec);
props.Add(MQC.CHANNEL_PROPERTY, queueProperties.QueueChannel);
props.Add(MQC.HOST_NAME_PROPERTY, queueProperties.MachineName);
props.Add(MQC.PORT_PROPERTY, queueProperties.QueuePort);
queueManager = new MQQueueManager(queueProperties.QueueManager, props);
queue = queueManager.AccessQueue(queueProperties.QueueName, MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING | MQC.MQOO_SET_ALL_CONTEXT);
}
public static void CloseQueue(MQQueue queue, MQQueueManager queueManager)
{
if (queue != null && queue.OpenStatus)
{
queue.Close();
}
if (queueManager != null)
{
queueManager.Disconnect();
}
}
|
Outside these methods I repeatedly call OpenQueue and then CloseQueue using the queue properties they have entered. They connect successfully (using SSL properties) to two different queues and everything is fine when it is run in a single thread.
The exact same queue properties however fail when multi-threading is introduced. On two separate threads, I instantiate the MQQueueManager and MQQueue objects in each and then make many OpenQueue/CloseQueue calls with random short thread sleeps in the middle, to mimic the real world application doing some work (each of the two threads is connecting to a different queue on different a server). Every time we hit the 2393 error.
It's very strange because opening/closing the connection 1000s of times in quick succession works without a problem in single threaded mode, so there can't be a fundamental problem with the SSL settings. Is there a bug in the IBM .net dlls that means the MQQueueManager gets confused as to which SSL properties to use?
They are using WebSphere MQ Client 7.0.1.8, and connecting to MQ Servers with various different versions from 6.0.2.4 to 7.0.1.8.
Any help would be greatly appreciated. I don't actually have access to their MQ environment, but does anyone have any ideas of what could be going wrong or how we could rectify it? Multi-threading support is a vital part of our application, and we want to be sure that this is supported by WebSphere MQ. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Jan 15, 2013 7:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Are you sure you're using the multi-threaded versions of the MQ libs?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Pringle |
| Posted: Tue Jan 15, 2013 7:48 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
| No, we are just referencing amqmdnet.dll and amqmdxcs.dll. Are there other dlls we should be referencing that support multi-threading? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 15, 2013 7:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
There are in other languages. I'll leave someone with more .NET than me to speak authoritatively.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jan 15, 2013 10:42 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you tried defining one hash table per connection? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Pringle |
| Posted: Wed Jan 16, 2013 3:45 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
Do you mean instantiating the hash table once before I enter the loop of closing/opening connections many times? I.e reusing a single instance per thread? Because I am already creating a hash table per connection.
Not sure why this would help but I can certainly give it a try. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 16, 2013 3:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Pringle wrote: |
Do you mean instantiating the hash table once before I enter the loop of closing/opening connections many times? I.e reusing a single instance per thread? Because I am already creating a hash table per connection.
Not sure why this would help but I can certainly give it a try. |
I think he wants you to make sure that every thread has at least one hashtable, and ideally that every connection has a unique hashtable.
It's not as clear from the code you've posted, but it seems like you're already doing that.
If you add some debugging to dump out the contents of the hashtable just before the connect, does that provide additional information on what fails when you use more than one thread? |
|
| Back to top |
|
|
| Pringle |
| Posted: Wed Jan 16, 2013 4:12 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
Some more details on the scenario might help:
Thread 1:
Connect to Queue Manager ABC on machine name XXX
Close connection
Connect to Queue Manager ABC on machine name XXX
Close connection
Connect to Queue Manager ABC on machine name XXX
Close connection
...etc... (always using identical hash table properties - so whether defined once before loop or each time connection is opened shouldn't make a difference)
Thread 2:
Connect to Queue Manager DEF on machine name YYY
Close connection
Connect to Queue Manager DEF on machine name YYY
Close connection
Connect to Queue Manager DEF on machine name YYY
Close connection
...etc...(again all hash table properties, i.e. queue name, channel, SSL, etc stay unchanged)
Running just thread 1 in isolation works fine, and the same with thread 2. However start both threads and you get 2393 errors straight away.
An older version of our application didn't used to have this problem, and I'm going through the code changes with a fine-toothed comb to see what has changed. One minor difference is that we now set MQC.MQOO_SET_ALL_CONTEXT in AccessQueue whereas we didn't in the past. Could this cause any problems with multi-threading/SSL? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 16, 2013 4:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You should open and close connections as infrequently as possible.
So ideally all threads should
begin
Connect
... do everything it needs to
... reconnect if the connection drops somehow
disconnect
end
But that won't affect your 2393 issue.
Again, FJB_Saper's suggestion is that the thread should look like
begin
create new hashtable
populate new hashtable
connect
... do everything it needs to
... reconnect if the connection fails
disconnect
end
Again, it's not 100% clear, but it seems that you are already doing that, at least to the part about creating a new hashtable for each thread.
I would also make sure that every thread sets MQC.TRANSPORT_PROPERTY to MQC.TRANSPORT_MQSERIES_CLIENT in the hashtable. |
|
| Back to top |
|
|
| Pringle |
| Posted: Wed Jan 16, 2013 4:53 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
Thanks for the reply. Unfortunately for historical/legacy reasons we're tied to the way our system has been implemented in terms of communicating with various different queuing systems, WebSphere MQ being one of several. Our API forces each queue implementation to work in this way where the connection to the queue is opened and closed after each operation. In an ideal world we would re-write the entire app to work in a way that fits better with WebSphere MQ but that isn't feasible.
Anyway back to the 2393, is there a reason why MQC.TRANSPORT_MQSERIES_CLIENT should be used? It is currently hard-coded to MQC.TRANSPORT_MQSERIES (it is configurable in our main application though - but wasn't in the old version that didn't exhibit these SSL errors).
I am creating a new hashtable per connection/thread. Unfortunately we are stuck with this paradigm:
begin
connect with newly populated hashtable
... do nothing
disconnect
connect with newly populated hashtable
... do nothing
disconnect
connect with newly populated hashtable
... do nothing
disconnect
connect with newly populated hashtable
... do nothing
disconnect
...etc...
end
But could change it to this if it really would make a difference:
begin
create new hashtable
populate hashtable
connect
... do nothing
disconnect
connect
... do nothing
disconnect
connect
... do nothing
disconnect
connect
... do nothing
disconnect
...etc...
end |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 16, 2013 5:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Pringle wrote: |
| Thanks for the reply. Unfortunately for historical/legacy reasons we're tied to the way our system has been implemented in terms of communicating with various different queuing systems, WebSphere MQ being one of several. Our API forces each queue implementation to work in this way where the connection to the queue is opened and closed after each operation. In an ideal world we would re-write the entire app to work in a way that fits better with WebSphere MQ but that isn't feasible. |
You can implement your thread class to cache the queue manager object. That way, even if the caller says "connect" a lot, you don't always connect.
| Pringle wrote: |
| Anyway back to the 2393, is there a reason why MQC.TRANSPORT_MQSERIES_CLIENT should be used? It is currently hard-coded to MQC.TRANSPORT_MQSERIES (it is configurable in our main application though - but wasn't in the old version that didn't exhibit these SSL errors). |
Because you're making a CLIENT connection, not a bindings connection.
That's what MQC.TRANSPORT controls..
http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/topic/com.ibm.mq.doc/un10480_.htm |
|
| Back to top |
|
|
| Pringle |
| Posted: Wed Jan 16, 2013 9:24 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
Yes you're right, something along those lines should work.
But the 2393 error would happen regardless, as it occurs on the first iteration of the two threads. I changed it so that MQC.TRANSPORT_MQSERIES_CLIENT was set in the hash table on both threads but this didn't help.
Any other suggestions on what could be causing this issue? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 16, 2013 9:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Pringle wrote: |
Yes you're right, something along those lines should work.
But the 2393 error would happen regardless, as it occurs on the first iteration of the two threads. I changed it so that MQC.TRANSPORT_MQSERIES_CLIENT was set in the hash table on both threads but this didn't help.
Any other suggestions on what could be causing this issue? |
That somehow your threads don't have the right values for your queueProperties object? i.e. that you are starting the thread before the object is fully populated....
That somehow the keystore is being locked by one thread?
It'd be worth taking an MQ client trace and trying to sort through it to see if it shows something useful. |
|
| Back to top |
|
|
| Pringle |
| Posted: Thu Jan 17, 2013 2:03 am Post subject: |
|
|
Novice
Joined: 12 Nov 2012
Posts: 15
|
The queueProperties object definitely has the right properties and is fully populated. The identical code with identical queue properties runs fine when I don't start both threads at the same time.
The two queue managers are using two different key stores, so that rules out the second possibility I think.
I have looked up tracing in the documentation and found this: http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzav.doc%2Fun10680_.htm. I will get the client to try this and see what the results are.
Thanks for the advice. |
|
| Back to top |
|
|
| tri.vu |
| Posted: Wed Feb 06, 2013 12:54 pm Post subject: |
|
|
Newbie
Joined: 01 Feb 2013
Posts: 9
|
| Did you have any static variables declaration for queueManager / queue? |
|
| Back to top |
|
|
|
|
