Author |
Message
|
krishna1228 |
Posted: Wed May 29, 2013 1:45 am Post subject: How can we avoid duplicate mesages in case of mutiprocessing |
|
|
Novice
Joined: 28 May 2013 Posts: 18
|
Im using jave client program to read the messages and persisting these messages in hadoop using flume multi processing here im getting duplicate messages how can i avoid this? |
|
Back to top |
|
 |
zpat |
Posted: Wed May 29, 2013 1:53 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Why use another message store?
Simply get under sycnpoint from MQ (do NOT browse).
When ready - commit the message to remove it from queue.
MQ will not present the same message more than once unless backed out (check MQMD.back value to detect this). |
|
Back to top |
|
 |
krishna1228 |
Posted: Wed May 29, 2013 2:08 am Post subject: |
|
|
Novice
Joined: 28 May 2013 Posts: 18
|
If i use syncpoint and commit it is decreasing my performance by half.... |
|
Back to top |
|
 |
zpat |
Posted: Wed May 29, 2013 2:16 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Process more than one message per commit. Maybe 100 or 200 for example.
If you don't use syncpoint you are at risk of getting a message destructively before successfully processing it in the application - that's your choice.
Worry about performance when your code works properly and after you have made the tuning changes that I suggested. |
|
Back to top |
|
 |
fjb_saper |
Posted: Wed May 29, 2013 12:30 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
krishna1228 wrote: |
If i use syncpoint and commit it is decreasing my performance by half.... |
I doubt very much that it is MQ that is slowing you down that much.
With a server connection, 1 commit per (let's say 1,000 messages, could also be 100) and messages under 2k, I have seen achieved performance of over 7,000 TPS in JMS TextMessage.... (shame I didn't care about the payload, just reading the queue destructively
Have fun  _________________ MQ & Broker admin |
|
Back to top |
|
 |
krishna1228 |
Posted: Thu May 30, 2013 1:35 am Post subject: |
|
|
Novice
Joined: 28 May 2013 Posts: 18
|
Hi,
Currently we are getting 30 msgs/sec which means 1800/min only could you please let me know what action need to be taken in order to increase the performance.....
Note :
All messages are of 4kb |
|
Back to top |
|
 |
zpat |
Posted: Thu May 30, 2013 1:40 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
|
Back to top |
|
 |
|