| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| file streaming in WMB v7, how does it work ? |
« View previous topic :: View next topic » |
| Author |
Message
|
| ydsk |
 Posted: Wed Dec 29, 2010 7:18 am Post subject: file streaming in WMB v7, how does it work ? |
 |
|
Chevalier
Joined: 23 May 2005
Posts: 410
|
I did a simple test to see if streaming really works when reading big files in WMB v7 using a File Input Node, as mentioned by some others in the forum.
Here is my test msgflow:
---------------------------------------------------
FIN--> Compute--> FON
FIN: BLOB, 'Whole File' is a record, rest all default
Compute Node ESQL: CopyEntireMessage, and set OutputLocalEnvironment.Destination.File.Name/Directory to appropriate values to just move the file from one dir to another, NO processing involved. 'Compute Mode' is set to 'All'
FON: 'whole file', rest all default.
---------------------------------------------------
This flow could transfer a 70 MB file from a source dir to a target dir (on the same machine - there is NO ftp involved).
But when a 105 MB file is dropped, the file got moved to mqsibackout directory.
Now I don't understand how streaming works in WMB v7, when a huge file is read by FIN. If streaming works, the file should be moved by my msgflow irrespective of its size (within some limits of course). Can some experts comment on this ?
This is related to my other post on splitting huge XML files, but the issue here is different because there is NO xml involved, it's just BLOB. This thread is just about STREAMING. That's why I created a separate thread/post for the issue. Sorry if it isn't acceptable.
thanks
ydsk. |
|
| Back to top |
|
 |
| ydsk |
| Posted: Wed Dec 29, 2010 7:24 am Post subject: |
|
|
Chevalier
Joined: 23 May 2005
Posts: 410
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 29, 2010 7:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The blob parser is NOT a streaming parser.
If you want this to work, configure it to read the file in fixed-length pieces, and connect the end of data terminal to the finish file terminal.
Then the file will only get moved in fixed-length pieces and only the amount of memory for the fixed-length piece will be used.
The previous discussions about handling this better in XMLNSC were based on the fact that XMLNSC *is* a streaming parser. This means that it can handle the fact that FileInput can pass it a handle to a bitstream that has not been fully read, and then the Parser can only access those points of the bitstream that it needs to at a given time. The delete statements you were given in that other thread were to remove the pieces of parsed data from memory after use. It's, however, possible that you're trying to do this on something like Broker v6.0.0.1 which may not actually have streaming capabilities in it. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Dec 29, 2010 7:40 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| It's, however, possible that you're trying to do this on something like Broker v6.0.0.1 which may not actually have streaming capabilities in it. |
OP seems clear that some level of WMBv7 is in use.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ydsk |
| Posted: Wed Dec 29, 2010 7:55 am Post subject: |
|
|
Chevalier
Joined: 23 May 2005
Posts: 410
|
Thanks mqjeff for the info on BLOB parser.
Our environment is all WMQ v7.0.1, and WMB v7.0.0.1 on Windows 2008 R2 servers.
I didn't see the streaming behavior with XMLNSC parser either, as I mentioned already in the other thread.
My real issue is in the other thread (unable to split a huge xml file into smaller chunks with the SAME structure but with a smaller number of repeating elements in each) so I'll leave the discussion in this thread here.
thanks
ydsk
Last edited by ydsk on Wed Dec 29, 2010 7:58 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 29, 2010 7:57 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| It's, however, possible that you're trying to do this on something like Broker v6.0.0.1 which may not actually have streaming capabilities in it. |
OP seems clear that some level of WMBv7 is in use. |
Okay. I didn't look up the other thread.
And I suppose I also deserve  for forgetting that FileInput wasn't introduced until 6.1. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
