| Author |
Message
|
| pressy |
 Posted: Mon Jul 10, 2006 2:08 am Post subject: WebsphereMQ with JMS |
 |
|
Newbie
Joined: 10 Jul 2006
Posts: 7
|
Hi
I'm a newbee to websphereMQ, this is general question and would like to have some reference materials or tips on that.
I'm trying to design a batchframe work system, using WSApplication Server,WebsphereMQ and JMS technologies.
1. Approximately there will be around 1000000 records needs to be processed daily .
2. These records may be categorized according to the functional group.
There are approximately 25-30 functional groups i'm having.
3. There is a possibility that , 2 or more functional groups batches will be running at a same time
4. We are also planning to have clustering in Websphere Application Server.
Now in the configuration of MQ,
a. Is it better for each functional group i have a Queue defined and a Listener defined.
or
For all the Functional Groups intially i have 1 Queue and 1 Listener and then this listener will inturn sends the messages to other queues for processing of the messages.
b. what kind of transactional managements are sugeested.
c. What kind of Clustering techniques might be helpful in the load balancing..
d.What kind of error configurations are best suited for this environment.
I have many more questions but this is the start... request you to point to the right design or solutions.
--pressy  |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Jul 10, 2006 2:35 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
WAS, MQ, and JMS are not designed to implement batch processing. They are designed to implement single-transaction processing.
You should design your system to put the smallest possible piece of data into one transaction. So if you have 100000 records, then each record should be it's own transaction - you should not even think of trying to do one transaction over 100000 records. Even if each record is only one byte long - this is going to be a big load on the transactional management capabilities of your system.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| pressy |
| Posted: Mon Jul 10, 2006 11:39 pm Post subject: |
|
|
Newbie
Joined: 10 Jul 2006
Posts: 7
|
| Quote: |
WAS, MQ, and JMS are not designed to implement batch processing. They are designed to implement single-transaction processing.
You should design your system to put the smallest possible piece of data into one transaction. So if you have 100000 records, then each record should be it's own transaction - you should not even think of trying to do one transaction over 100000 records. Even if each record is only one byte long - this is going to be a big load on the transactional management capabilities of your system |
Hi
Probably i should give more explanation for this..
I'm defenitely not interested in putting all the records in to a single transaction I'm considering here the following type of design.
MessageSender---->Queue1------->MDB1[listenes messages on queue1]
MDB1-------------->Queue2[specificto functional group]
MDB1--------------->Queue3[[specificto functional group]
So as you can see one MDB's can listen to all the messages and based on the message type, puts the message into the respective functional queue, then an MDB will listen to that queue and process that message.
This is what i'm planning , now i will split the no of work records into atomic unit of works so that my transaction loads are reduced.
The thing what i'm looking is how to increase the speed of Message Consuming when compared with Message Producing.
This is in addition to the questions i have posted originally.
--Pressy |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jul 11, 2006 12:41 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| jefflowrey wrote: |
| You should design your system to put the smallest possible piece of data into one transaction. So if you have 100000 records, then each record should be it's own transaction |
Interested to know more about your thinking here. MQ tries hard to avoid I/O until the MQCMIT. Therefore if you can do (say) 10 small things within a single UOW then this will go faster than doing the same 10 small things under 10 separate UOWs. A similar principle is applied to message batching on normal channels: ie. the commit processing is done once per batch rather than once per message. |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jul 11, 2006 12:45 am Post subject: Re: WebsphereMQ with JMS |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| pressy wrote: |
2. These records may be categorized according to the functional group.
There are approximately 25-30 functional groups i'm having. |
Will this mean 25-30 different styles of processing the messages too? If so, it will probably be a good idea to have 25-30 "input" queues to handle the distinct types. Ideally you will also have 25-30 application programs watching the 25-30 queues. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jul 11, 2006 3:05 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| mvic wrote: |
| jefflowrey wrote: |
| You should design your system to put the smallest possible piece of data into one transaction. So if you have 100000 records, then each record should be it's own transaction |
Interested to know more about your thinking here. MQ tries hard to avoid I/O until the MQCMIT. Therefore if you can do (say) 10 small things within a single UOW then this will go faster than doing the same 10 small things under 10 separate UOWs. A similar principle is applied to message batching on normal channels: ie. the commit processing is done once per batch rather than once per message. |
It's always better to program conservatively, and then adjust for performance if needed.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
