ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportMQ , FIle Max Size to handle without any issue

Post new topicReply to topic
MQ , FIle Max Size to handle without any issue View previous topic :: View next topic
Author Message
IIBTRACKER01
PostPosted: Wed Apr 12, 2017 1:59 am Post subject: MQ , FIle Max Size to handle without any issue Reply with quote

Newbie

Joined: 12 Apr 2017
Posts: 4

Hello Experts,
We have a requirement wr in we get a file of size 60 MB (CSV) of 800,000 record , where in we need to split these record based on some field and put for a queue and transform once transformed we place the message in queue again . Now queue will have consider 600,000 record . Now we need to combine and send for a file.
File Input ==> Compute[Split FIle] ==> MQ1
MQ1 ==> Compute[TRansform File by some field value] ==> MQ2
MQ2 ==> COmpute[Combine to single] ==> File
Based on this have set of questions
1) File Can handle upto 60 MB easily ? ( as per IBM it can handle upto 100MB)
2) MQ can handle this much record easily ?( as per IBM it can handle 100MB , by chnaging channel and Queue def)
3) If i parse CSV message with DFDL , 60 MB file will reach more than 100 MB , can this be handled in compute node to split the file
4) How can i combine 600000 record to single file ( ANy logic/Approach )
If you see any issue wrt to this design please let me know. (MQ is required inbetween of MQ for future use)
I know IIB is not the easy solution for this .. But we are trying to acihev it here.
Back to top
View user's profile Send private message
DeadPool
PostPosted: Wed Apr 12, 2017 2:20 am Post subject: Reply with quote

Apprentice

Joined: 27 May 2016
Posts: 30

OK so some questions here

Are all the records in the files the same?

If they are not the same when you put them back in a file are you splitting by record type into separate files?

Records are in CSV format but what transformations are likely to be required? just content and stay in CSv or into another format?

Why use MQ1 and MQ2

This can be done in a single flow File In read by record transform and write by record. (this does depend on if record content is the same) If you do this then the size limit in MQ will not come into effect however, if you need to use MQ then as long as the queue / channel settings are correct this should not be an issue as IIB will consume / write at a fast rate so the queue should not fill up and then teh only issue is whether the os can handle the numbers

which brings into effect Persistence? will these messages be Persistent or not?

Combining goes back to original question of record content and splitting

If it is just a case of combining then File out node will handle that.

Would suggest you try building this on a small scale with a file with 5 records so you can see the process
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Apr 12, 2017 4:46 am Post subject: Re: MQ , FIle Max Size to handle without any issue Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24452
Location: Ohio, USA

IIBTRACKER01 wrote:
1) File Can handle upto 60 MB easily ? ( as per IBM it can handle upto 100MB)


If IBM says it can, it can. I'd be more concerned with the resources available on whatever IIB is running on. Especially if you need all this done in a timely fashion.

IIBTRACKER01 wrote:
2) MQ can handle this much record easily ?( as per IBM it can handle 100MB , by chnaging channel and Queue def)


See my answer above, which applies equally here.

IIBTRACKER01 wrote:
3) If i parse CSV message with DFDL , 60 MB file will reach more than 100 MB , can this be handled in compute node to split the file


See my answer above - you'll need a lot of memory. There's also specific advice in the IIB documentation concerning handling large files in a memory efficient manner you should review.

IIBTRACKER01 wrote:
4) How can i combine 600000 record to single file ( ANy logic/Approach )


Figure out how you know you've consumed all the records (I bet it's never exactly 600,000). Store all the records until this criteria is met. Write out all the records into a file.

IIBTRACKER01 wrote:
If you see any issue wrt to this design please let me know.


You're trying to filter a file of 800,000 records into a file of 600,000 records. Big whoop. Usual comments about using files apply here.

IIBTRACKER01 wrote:
(MQ is required inbetween of MQ for future use)


Well that's a throwaway comment of no value. As the other poster correctly states, you don't need MQ and it's making your design problems worse. What "future use"? What would you change? Why would you put a file on a queue then try and turn the messages back into a file? It's nonsensical.

IIBTRACKER01 wrote:
I know IIB is not the easy solution for this


It's easier than with some other platforms.

IIBTRACKER01 wrote:
But we are trying to acihev it here.


Why? How did you come up with IIB if you have apparently decided it's not easy with IIB? Why not use something that allows an easy solution for your requirement? You enjoy complaining? Or what the ability to blame the software when you fail to deliver? What?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
IIBTRACKER01
PostPosted: Wed Apr 12, 2017 9:05 pm Post subject: File and MQ Handling Reply with quote

Newbie

Joined: 12 Apr 2017
Posts: 4

Are all the records in the files the same?

Yes Same file

Records are in CSV format but what transformations are likely to be required? just content and stay in CSv or into another format?

Just content example time needs to be mdoifed and key field mapped to another field


Why use MQ1 and MQ2

Client Arch .. they need MQ only fuutre indivual transcation they want to send for different system

which brings into effect Persistence? will these messages be Persistent or not?

Non Persistent

If it is just a case of combining then File out node will handle that.

For combining i need to put limit , else the file size will go bigger and bigger

Resource Available :

16GB Ram , windows server 2012

Let me rephrase my query:

Actually will read that big file 60 MB from file input node and split it individual transaction (in ESQL compute node) and send for MQ. (Hope this is achievable or do you see any issue will consider file handling best practices then also i am bit scared here as i am doing split based on field value example

Have tets,tets,111,tets
tets,tets,111,tets
tets,tets,222,tets
tets,tets,222,tets
tets,tets,333,tets
tets,tets,333,tets? will send these 111,222, 333 as single message to MQ)

MQ1(single transaction consider 5 line of CSV record) will pick and do transformation and send for another MQ2 (This should be straight forward)

Now from MQ2 , we need to put for file again by combining the bulk set of messages
Back to top
View user's profile Send private message
DeadPool
PostPosted: Wed Apr 12, 2017 11:22 pm Post subject: Reply with quote

Apprentice

Joined: 27 May 2016
Posts: 30

I will repeat my original question

Why do this in multiple flows

SIngle flow can output to both file and queue

The messages that are re-combined are they the original messages or the new ones ie 111,222,333
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Apr 13, 2017 4:35 am Post subject: Re: File and MQ Handling Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24452
Location: Ohio, USA

IIBTRACKER01 wrote:
16GB Ram , windows server 2012


I've seen bigger Windows servers.

IIBTRACKER01 wrote:
Hope this is achievable


It is.

IIBTRACKER01 wrote:
will consider file handling best practices


On a server that size, very wise.

IIBTRACKER01 wrote:
then also i am bit scared here as i am doing split based on field value


Is the file sorted by split value, or do you need to accumulate records as you go? What exactly is scaring you?

IIBTRACKER01 wrote:
MQ1(single transaction consider 5 line of CSV record) will pick and do transformation and send for another MQ2 (This should be straight forward)


Because you've consolidated the message or you're using an MQ group?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportMQ , FIle Max Size to handle without any issue
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.