| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Check for duplicate data sent within the same day |
« View previous topic :: View next topic » |
| Author |
Message
|
| whydieanut |
 Posted: Thu Jan 08, 2015 5:13 am Post subject: Check for duplicate data sent within the same day |
 |
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
Hi everyone.
I am trying to avoid processing duplicate messages in my flow.
The flow receives data along with some meta data via RFH2 like:
- Subject
- File Name
For a give day, any message that has the same Subject and File Name as an earlier message from the same day needs to be treated as a duplicate.
I plan on using a Shared Row variable as a cache, to store a string (obtained by concatenating the Subject, File Name and the Date when it was received) for each message that is processed.
For each incoming message, I will check if the generated string is already present in the cache.
If not, I treat is as a fresh message and process it. Then append it to the cache.
If the generated string is already present in the cache, I treat the current message as a duplicate.
But I also need to reset the cache every day, as I am only concerned with checking duplicates within the same day.
For that, when I receive a message, I will check the first string in the cache to check the date in it. If the date is different from the current date, I reset the cache.
Is there anything glaringly wrong with this approach?
Is there a more graceful way of doing this in WMB? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jan 08, 2015 5:40 am Post subject: Re: Check for duplicate data sent within the same day |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
| Is there anything glaringly wrong with this approach? |
If anyone or anything restarts or reloads the execution group the shared variable will be lost. If this happens in the middle of the day and a duplicate message arrives after this it won't be detected.
I leave it to you to calculate the odds of this happening on your particular site and if you think the risk is acceptable.
| whydieanut wrote: |
| Is there a more graceful way of doing this in WMB? |
Store the "cache" in something that will survive an EG reload. A database is the obvious choice, but you could append your concatinated string to a file on the OS.
Depends on the risk of you losing the shared variable and the damage a duplicate will cause.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| whydieanut |
| Posted: Thu Jan 08, 2015 6:36 am Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
| Quote: |
| but you could append your concatinated string to a file on the OS. |
This is just an enhancement for an already existing flow which wasn't taking care of duplicates. So I believe avoiding the duplicates would be a value add to the existing process. Need to check with the client if occasional duplicates in case of EG reloads is acceptable.
If it is, would the performance impact of writing to a file justify taking that approach over Shared Variables? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 08, 2015 7:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
| If it is, would the performance impact of writing to a file justify taking that approach over Shared Variables? |
If the current situation is that the client suffers every duplicate, then using a shared variable to eliminate the vast majoirty of the duplicates would seem cost effective to me.
You'd only want a file (or other solution) if you absolutely had to have no duplicates or there was an on-site procedure to restart or reload components on a regular schedule that wasn't at end of day.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| whydieanut |
| Posted: Thu Jan 08, 2015 8:00 am Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
