| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Unzip file with multiple xml files |
« View previous topic :: View next topic » |
| Author |
Message
|
| new2wmbdev |
 Posted: Thu Apr 12, 2012 10:10 pm Post subject: Unzip file with multiple xml files |
 |
|
Newbie
Joined: 12 Apr 2012
Posts: 5
|
Hi,
Please excuse as this is my first post.
We have an issue where we need to unzip an file with multiple xml files and process them in the flow.
I have started using compute node and calling java functions, but it's seems to complicated with thread management etc.
1. What is the better way/flow to process such files.
2. Can we use Unzip plugin to process multiple files.
Any ideas/pointers to what is a correct approach might.
Thank you all for you replies. |
|
| Back to top |
|
 |
| Esa |
| Posted: Thu Apr 12, 2012 10:43 pm Post subject: Re: Unzip file with multiple xml files |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| new2wmbdev wrote: |
I have started using compute node and calling java functions, but it's seems to complicated with thread management etc.
|
Be very careful with threads. I guess you are familiar with this and have read paragraph 'Restrictions on java routines'.
A message flow instance is always run in one thread, and you would be on a safer ground if your java routine did not start any threads of its own.
I would write a java function that decompresses only one ZipEntry at a time and returns it as a blob. And call it in a loop so many times there are files in the zipfile, propagating the resulting blob to processing nodes after each time the function has been called. (In fact, I am going to implement something similar myself today)
The best solution would be to do it in a java compute node, because it gives you better ways to control flow performance, but if you know how the product works you can get the same performance with ESQL + java function, too.
I am not familiar with the UnZip plugin, but I don't think you would get a well-performing solution with it, at least in your case of several files to be unzipped. |
|
| Back to top |
|
|
| new2wmbdev |
| Posted: Fri Apr 13, 2012 8:27 am Post subject: Unzip file with multiple xml files |
|
|
Newbie
Joined: 12 Apr 2012
Posts: 5
|
Esa,
Thank your response. I have a couple of more questions on this,
1. what would be the input to JavaCompute Node, the zip file path or the contents of the Zipfile as a BLOB stream.
2. If using BLOB, how do we convert it into ZipFile of java.util.zip.ZipFile.
Any guidance is greatly appreciated.
Would be able to share the process/ pseudo code you were going to implement for this issue.
Thank you again. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Apr 13, 2012 9:09 am Post subject: Re: Unzip file with multiple xml files |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| new2wmbdev wrote: |
1. what would be the input to JavaCompute Node, the zip file path or the contents of the Zipfile as a BLOB stream. |
InputRoot.BLOB.BLOB
| new2wmbdev wrote: |
| 2. If using BLOB, how do we convert it into ZipFile of java.util.zip.ZipFile. |
You don't. Create a ByteArrayInputStream for the BLOB, wrap it with a ZipInputStream, open a ZipEntry and stream its contents to a ByteArrayOutputStream...
| new2wmbdev wrote: |
Would be able to share the process/ pseudo code you were going to implement for this issue.
|
Actually, I did something else. I created a java class that renames a file inside the zip file, and called id trough an ESQL function. Worked OK. Basicly a lot like what you will have to do, because a ZipEntry cannot be renamed, you will have to create a new ZipEntry and stream the contents into it. That will uncompress and decompress the file, but if you apply streaming, it does not allocate much extra memory. I used a 16 kB buffer. 5 MB zipfile got processed in 2,5 seconds. |
|
| Back to top |
|
|
| new2wmbdev |
| Posted: Sat Apr 14, 2012 6:30 pm Post subject: Unzip file with multiple xml files |
|
|
Newbie
Joined: 12 Apr 2012
Posts: 5
|
Esa,
Thank you for your guidance. I was able to unzip the file and read each file separately .
I am now faced with another issue(please excuse my repeated queries).
After reading each file using ZIPFile Entry, when I used propagate to pass the message as xml to next node, the original zip file name is getting passed over and effectively overwriting the files except for the last one.
Here is the code snippet that I am using to propagate.
I need to send the xml file name in the zipfile across to the next node and NOT the ZIP file name.
| Code: |
MbMessage inMessage = assembly.getMessage();
MbMessage outMessage = new MbMessage(inMessage);
MbMessageAssembly outAssembly = new MbMessageAssembly(assembly,
outMessage);
// Unzip Code
While (files in ZIpfile){
newMsg.finalizeMessage(MbMessage.FINALIZE_NONE);
out.propagate(newAssembly;
} |
I have looked at infocenter and as I am new to this, I might be missing something very obvious which has been eluding me…
Thanks in advance. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Apr 14, 2012 11:44 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Look again at the Java WMB sample...
What you want to do is read the blob corresponding to the file entry, set that blob to the outMessage and then propagate and continue with the next entry file.
Your example as shown here does none of these things.
You might need a little bit more training or help from an experienced mentor... If you had difficulties with reading the zip file content, consider message broker an advanced Java topic, (not for the faint of heart) 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Apr 15, 2012 2:12 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| fjb_saper wrote: |
Your example as shown here does none of these things.
|
This part does:
| new2wmbdev wrote: |
| // Unzip Code |
Obviously new2wmbdev did not take my advice to stream each entry separately but chose to inflate them all into a huge collection object. Must have had his reasons: the zipped fiels are tiny, memory is cheap, skills missing or the workload of the incident management team must be secured...
However, his current problem has nothing to do with unzipping, with the exception of capturing the name of each ZipEntry.
@newtowmbdev: It's time for you to start reading a holy script called the InfoCenter. Navigate to Message Broker webpages, find the infocenters under support/documentation (they are a bit difficult to find, use all your imagination), select the Infocenter with the same version number of your broker runtime. Then find documentation of FileOutput node and read it. You will find the info you need under 'Local Environment overrides' or something like that, but it does not hurt to read everything.
the InfoCenters |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
