| Author |
Message
|
| krishna1228 |
 Posted: Wed May 29, 2013 1:45 am Post subject: How can we avoid duplicate mesages in case of mutiprocessing |
 |
|
Novice
Joined: 28 May 2013
Posts: 18
|
| Im using jave client program to read the messages and persisting these messages in hadoop using flume multi processing here im getting duplicate messages how can i avoid this? |
|
| Back to top |
|
 |
| zpat |
| Posted: Wed May 29, 2013 1:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Why use another message store?
Simply get under sycnpoint from MQ (do NOT browse).
When ready - commit the message to remove it from queue.
MQ will not present the same message more than once unless backed out (check MQMD.back value to detect this). |
|
| Back to top |
|
|
| krishna1228 |
| Posted: Wed May 29, 2013 2:08 am Post subject: |
|
|
Novice
Joined: 28 May 2013
Posts: 18
|
| If i use syncpoint and commit it is decreasing my performance by half.... |
|
| Back to top |
|
|
| zpat |
| Posted: Wed May 29, 2013 2:16 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Process more than one message per commit. Maybe 100 or 200 for example.
If you don't use syncpoint you are at risk of getting a message destructively before successfully processing it in the application - that's your choice.
Worry about performance when your code works properly and after you have made the tuning changes that I suggested. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 29, 2013 12:30 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| krishna1228 wrote: |
| If i use syncpoint and commit it is decreasing my performance by half.... |
I doubt very much that it is MQ that is slowing you down that much.
With a server connection, 1 commit per (let's say 1,000 messages, could also be 100) and messages under 2k, I have seen achieved performance of over 7,000 TPS in JMS TextMessage.... (shame I didn't care about the payload, just reading the queue destructively
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| krishna1228 |
| Posted: Thu May 30, 2013 1:35 am Post subject: |
|
|
Novice
Joined: 28 May 2013
Posts: 18
|
Hi,
Currently we are getting 30 msgs/sec which means 1800/min only could you please let me know what action need to be taken in order to increase the performance.....
Note :
All messages are of 4kb |
|
| Back to top |
|
|
| zpat |
| Posted: Thu May 30, 2013 1:40 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
|
| Back to top |
|
|
|
|
