ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » file handlibg - filter duplicate records

Post new topic  Reply to topic
 file handlibg - filter duplicate records « View previous topic :: View next topic » 
Author Message
sat_in_field
PostPosted: Thu Mar 18, 2010 2:56 am    Post subject: file handlibg - filter duplicate records Reply with quote

Novice

Joined: 23 Jan 2003
Posts: 20
Hi,

Running WMB6.1

We have a requirement to filter duplicate records from a file. The file will potentially be quite large in size and contain many records. Can WMB work on the entire file in memory.

Should a separate process external to WMB be created

We cannot ask for change in the source system.

What would be the best approach for this.

Cheers
Steve
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 18, 2010 3:56 am    Post subject: Re: file handlibg - filter duplicate records Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
sat_in_field wrote:
We have a requirement to filter duplicate records from a file. The file will potentially be quite large in size and contain many records. Can WMB work on the entire file in memory.


Define "large".

WMB can, but the question is can the box it's running on support that level of memory usage? There have been a number of posts in here about handling large messages which may be useful.

sat_in_field wrote:
Should a separate process external to WMB be created


It's probably quicker and easier to implement. Most sort utilites come with some kind of summary/de-duplicate function and it'll save a lot of WMB code.

<shameless_plug>z/OS sort certainly does</shameless_plug>
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Mar 18, 2010 4:14 am    Post subject: Re: file handlibg - filter duplicate records Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447
sat_in_field wrote:
Should a separate process external to WMB be created


No. WMB can handle this in many different ways. Very few of them require holding the entire file in memory.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » file handlibg - filter duplicate records
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
  
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.