ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Accessing multiple messages from a Q concurrently

Post new topic  Reply to topic
 Accessing multiple messages from a Q concurrently « View previous topic :: View next topic » 
Author Message
samird
PostPosted: Thu Apr 10, 2003 3:27 am    Post subject: Accessing multiple messages from a Q concurrently Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified


We have an application wherein data entered from web-site is stored in a database table. We need to do the following:

(i) first know when a new record has been created/updated in that table

(ii) once a new record has been created / updated in the table, create a message in a specified format and write that message onto the IBM MQ.

(iii) We need to create hundreds of such messages per minute

(iv) There is another application on another server which has some APIs written in C. Whenever a message arrives on the Q, this application needs first read from the Q, call a relevant C API for every message and then once successful, remove the message from the Q.

How can I go about achieving this requirement. Also, how do I access multiple messages from the Q concurrently?.
Back to top
View user's profile Send private message
mqonnet
PostPosted: Thu Apr 10, 2003 5:25 am    Post subject: Reply with quote

Grand Master

Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.

You can achieve this in the simplest form by having 2 apps working on the same queue.

App A puts a message with specified format on Queue X.

App B is doing a waited get on Queue X. If you want to get a specific message you could use msgid, correlid etc with respective match options. If you first wish to browse a message and then process it. YOu can do this as well by doing a browse and then doing destructive get if it matches your criterion.

Another solution of course would be to use triggering.

To answer your last question. From a stand along single threaded application there is no way you can get 2 messages off the queue concurrently. You can achieve this though using threaded apps on the platform threads are supported. But again there are limitations as to what message, what queue can be accessed by multiple threads of the same app.

Hope this helps.

Cheers
Kumar
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator
Back to top
View user's profile Send private message Send e-mail Visit poster's website
samird
PostPosted: Thu Apr 10, 2003 5:37 am    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

If there is a daemon process running - i.e. a program whose purpose is to receive the message from the Q and then call a C API to perform some database related operations after deconstructing elements from the message, can I use MQ triggering.

Do i need to have this program running in multiple threads if i wish each message to be processed as soon as it arrives(of course the trigger will do the invocation)?.

What will be the most ideal way of being able to process multiple messages as concurrently as possible? will having multiple queues be any better ?
Back to top
View user's profile Send private message
mqonnet
PostPosted: Thu Apr 10, 2003 6:24 am    Post subject: Reply with quote

Grand Master

Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.

You can virtually trigger anything you want. A program of course.

Running the program in multiple threads or running many instances of the same app or anything else is more of a design question.

You have to bear in mind that MQ CANNOT process messages simultaneoulsy. It can only do that in a threaded environment if and only if the messages are of different sizes and one thread takes longer to process than the other. Under normal circumstances if you have 100 messages and 10 apps with 10 threads, you CANNOT retrieve all the messages at once. MQ DOES NOT SUPPORT THIS. As mq processess messages sequentially.

So, your best bet would be to have more queues. But again the thumb rule is that you create a queue based on purpose. Say, request, replies etc. Hence you should have to strike a balance between all of these.

Just out of curiousity, why do you plan to access all the messages simultaneously. You need to remember that MQ is pretty fast and processes 100's of messages per second. Which caters to the need of most, if not all, the app scenario and design. So, the question of concurrently retrieving messages does not hold any significance. :).

Hope this helps.

Cheers
Kumar
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator
Back to top
View user's profile Send private message Send e-mail Visit poster's website
samird
PostPosted: Thu Apr 10, 2003 6:53 am    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

Thanks Kumar .

I still need a few more clarifications. What do u imply by MQ being able to process more than 100 messages per second?. See my concern is that I've a 3rd party application that performs database operations in a particular manner. I don't know the logic written in that application.

What I know is that every message is like an C API call to that application. So, I need to call an API for every message I get. I am not at the moment, sure as to how much time each API call will take to execute and finish. Therefore wanted a way in which I could call the APIs in parallel, by reading more than one message from the Q at a time.
But now that u say that || processing i.e. || reading from the Q is not possible, wonder what I can do. Can the MQ triggers work at this rate too..ie.. 100s of trigger executions per minute?
Back to top
View user's profile Send private message
mqonnet
PostPosted: Thu Apr 10, 2003 7:00 am    Post subject: Reply with quote

Grand Master

Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.

Firstly i think you should refer to the manuals for detailed clarifications. App programming ref and guide. Also triggering.

Secondly, i will try to explain what you want to achieve.

You are not clear as to what you mean by "API call". I would assume that you are talking about MQ API call, and in this case it is MQGET(looking at the scenario that you posted). MQGET does not take long and there are performance statistics on the IBM web site that you are search for and look on. When i said MQ process 100's of messages per second, i meant API calls such as MQget/put etc. So, in your case, if you say you are getting 100's of messages in your queue. YOu should first design how to handle it and then write an app and test out. If an only if you believe that your apps are unable to retrieve messages faster, you should starting thinking of alternatives that you put up on this post. Which i doubt you would need to.

As for triggering apps is concerned, reading the triggering manual would help you understand it better. There are various ways to trigger an app. Depth, every etc. You have to again design this according to your need.

Hope this helps.

Cheers
Kumar
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator
Back to top
View user's profile Send private message Send e-mail Visit poster's website
samird
PostPosted: Thu Apr 10, 2003 8:51 pm    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

Thanks Kumar.

A clarification to what u said. Actually we need to call two kinds of APIs once messages have arrived on the Q. MQGET to read the message and then some C APIs in a 3rd party billing system that takes field values from the message and does some database operations.

In our system we currently use two servers - server 1 has an auction site which writes to a database. Server 2 has a billing sysem. The Q should reside somewhere in between (on a 3rd server) so that both applications can read or write to that common Q as appropriate.

Since we are going to have only one Q really, what will that be called as? does the terminology of local and remote Queues apply here? Do we need to have transmission queues and channel initiation queues (as they are required in a typical local<-> remote Q configuration)?.

Also can we use Oracle packages to write to the Q and a C program (using MQ APIs) to read the same message(written by Q) from the Q. Will that work?
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Fri Apr 11, 2003 5:31 am    Post subject: Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

You may want to have two separate queues for easier management - one for each application to read from. That way, if one queue has messages sitting on it longer than expected, you know for sure which application isn't able to read from the queue.

If both applications are connecting to the same queue manager, you are correct that you don't need channels and etc. You may want initiation queues, and trigger monitors, in order to start up an application in response to a message being put on a queue though.

Any program can read a message on a queue that was written by any other program in any other language, but only if both programs know what the message will look like. For instance, you can have a COBOL program read a message produced by a Java program - but only if the message contains data that the COBOL program can interpret (so, not a Java object).
Back to top
View user's profile Send private message
samird
PostPosted: Wed Apr 16, 2003 2:02 am    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

Thanks.

Considering that we can read messages from the MQ sequentially only, if I need to read say 200 messages from the Q, will the MQGET command need to executed 200 times? So, will the sequence of steps be:
MQCONN
200 MQGETs
MQDISCONN
How costly (in terms of time) is it to do an MQGET?
Back to top
View user's profile Send private message
bower5932
PostPosted: Wed Apr 16, 2003 5:30 am    Post subject: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

MQCONN is the most expensive call. MQOPENs will be next. In comparison, MQGET/MQPUT will be fairly fast. You'll get into trouble with MQGET if you have to issue it twice because the buffer that you supplied on the first MQGET wasn't large enough. There are quite a few performance related SupportPacs. You might want to look them over:

http://www-3.ibm.com/software/integration/support/supportpacs/
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
samird
PostPosted: Wed Apr 16, 2003 8:08 pm    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

What is the best possible method by which we can retrieve multiple messages so that our performance (in terms of time taken) is good. Also, for every message we have to call APIs of a 3rd party billing system.

Therefore, between the time we get the message and until the billing system finishes processing each message, what is the ideal way to store each message after it has been read from the Q.

Will it be a good approach to read a number of messages from the Q(considering that MQGET is fairly fast), store the deconstructed elements from the message in the database and get the API execution to happen in a separate program.
Back to top
View user's profile Send private message
bower5932
PostPosted: Thu Apr 17, 2003 6:41 am    Post subject: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

I'll admit that I haven't been following this thread to closely. However, it looks like you are trying to do a two-phase commit type of operation. I think you should be doing something like:

MQCONN, DB connect
MQOPEN the queue
Loop:
MQBEGIN
MQGET in syncpoint
DB update in syncpoint
API call to 3rd party (do they have a syncpoint option?)
Either commit or backout
Next
MQCLOSE
MQDISC

This will keep your message gets, db updates, 3rd party calls all together.
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
Chris123
PostPosted: Thu Apr 17, 2003 6:48 am    Post subject: Reply with quote

Novice

Joined: 14 Apr 2003
Posts: 11

Samird,

Am I missing something here.

1) You have an auction site that writes to a database
2) You have a billing system that has a C API
3) When a new row appears in your database you need to call the billing system API

Where does a queuing system come into this? Why not have a service or daemon that does the following:

1) Get all records that have not been marked as 'billed'
2) For each record
a) Call the billing system API
b) Mark that record as 'billed'
3) Wait a period of time
4) Go back to 1)

Will this do what you what? It's simple and has no need for multi-threading.

Cheers,

Chris
Back to top
View user's profile Send private message
samird
PostPosted: Sun Apr 20, 2003 7:42 pm    Post subject: Reply with quote

Novice

Joined: 10 Apr 2003
Posts: 11
Location: unspecified

The data from the auction site is written to the MQ by another system. For every message in the MQ, I've to call the billing system.
Back to top
View user's profile Send private message
Keka
PostPosted: Tue Apr 22, 2003 6:08 am    Post subject: Reply with quote

Voyager

Joined: 28 Dec 2002
Posts: 96

Since everybody was suggesting towards Triggering and Multi threading, I thought I will add my 2 cents too..
If I understand it right then My idea would be
To Have a Starter program waiting on the queue for ever and pick up the mesage as soon as it arrives and depending on the Header or what ever to decide on which C API needs to be called. Then call another program( process) to process that C API call. Here my assumption is you have a separate program for each API.By this way you can control how many instances of each API you can have and the processing also will be faster as these are two different processes. You can have your load balancing logic in the starter program.
As for making sure you don't lose the message until you process it, two phase commit with the persistant message type is a good way to go for it.. But remember the two phase commit and persistant messages will add to your performance bottle necks.

Hope this helps..
_________________
Keka
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » Accessing multiple messages from a Q concurrently
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.