| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Does RfhUtil Write No Headers exist in the XMS API? |
View previous topic :: View next topic |
| Author |
Message
|
| fjb_saper |
 Posted: Thu Apr 28, 2022 3:58 am Post subject: |
 |
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| psychodelic_drumboy wrote: |
We are sent, for example, three iBytes messages with names such as:
- "zip_file_28_04_2022_1_of_3.SPLIT"
- "zip_file_28_04_2022_2_of_3.SPLIT"
- "zip_file_28_04_2022_3_of_3.SPLIT"
These files, on their own, cannot be opened by tool like WinZip.
|
That's expected. What happens to the file that is less than the max bytes? Can you open that one?
| psychodelic_drumboy wrote: |
The utility simply combines the contents of these 3 x SPLIT files (I call that "reconstituting") into a single file and names that file .ZIP. I didn't write the utility, so note that this is only my interpretation based on the end results.
|
I did not see any code about reassembling the files...
| psychodelic_drumboy wrote: |
When the messages are downloaded via my code, the ZIP file created by the utility will not open and will not unzip.
There is still something unique to how RFHUtil works, that is different to my supplied code. I have no idea what it is.
|
This suggests that the fault lies within the VB code.
Also I hope this is not Visual Basic but Visual Basic dot Net...
VB MQ plugin is no longer supported... VB.Net is supported...
| psychodelic_drumboy wrote: |
I don't know why 47.6 MB is the maximum permitted size of a message in this environment. I also don't know if that is a limitation of MQ, or a constraint the supplier of the messages has mandated.
As you can tell - I don't know very much! I'm sorry about that. |
First look if your code allows you to open a single message file.
Then look at how the multiple message files are reassembled. You did not supply code for that and there might be errors there... 
_________________
MQ & Broker admin |
|
| Back to top |
|
 |
| psychodelic_drumboy |
| Posted: Thu Apr 28, 2022 5:25 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| fjb_saper wrote: |
Also I hope this is not Visual Basic but Visual Basic dot Net...
VB MQ plugin is no longer supported... VB.Net is supported... |
So, it's definitely VB.NET!
| fjb_saper wrote: |
Then look at how the multiple message files are reassembled. You did not supply code for that and there might be errors there... |
I totally understand why you might be questioning whether our "reassembly" utility is the root cause, but the reason I would dismiss that, is because we have proven, that when the messages are downloaded via RFHUTIL, the resassembly tool results in a working file.
I know this might sound like a naive question, but, is there not a MS DOS command line utility I could use to pull down the iBytesMessages from a queue to a target file system location?
It seems (to me) like such an obvious thing to want to do. I wonder why it requires so much hand-crafted code. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 28, 2022 1:31 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
I also would make sure to flush the output stream before closing it...
Has to do with internal buffering and such...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Tue May 03, 2022 6:05 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| fjb_saper wrote: |
I also would make sure to flush the output stream before closing it...
Has to do with internal buffering and such... |
Sure okay so I'll try this:
| Code: |
Dim fsZipFile As FileStream = New FileStream(strMQMessageOutputFileDestinationFilePath, FileMode.Create)
'Write the contents of the Byte Array to the File via the FileStream object
fsZipFile.Write(buffer, 0, buffer.Length)
fsZipFile.Close()
fsZipFile.Flush()
fsZipFile = Nothing
|
Although I think the Dim statement surely must flush also (?), but I'm open to all suggestions right now.
I've also been given the tool that joins the files together, so I should be able to report the results of a code change more quickly now.
Let's see if flushing and destroying (e.g. setting to nothing) the FileStream object solves it. :fingerscrossed: |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue May 03, 2022 3:45 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2494
Location: Melbourne, Australia
|
New FileStream does not do any data I/O on the file content. All it does is create / open the file.
Close() does an implicit flush to the physical file if there are pending writes.
Flush() after Close() is pointless.
_________________
Glenn |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 12:09 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| bruce2359 wrote: |
| psychodelic_drumboy wrote: |
| … this still results in files that cannot be reconstituted. |
Please explain “reconstituted”. From what? To what? |
I have located the author of this utility which we call the split file utility.
It's in fact, very simple, it just issues this MS DOS command:
| Code: |
| copy /b eg_file_*.split combined_file.zip |
So
eg_file_0001.split
eg_file_0002.split
eg_file_0003.split
eg_file_0004.split
would be combined into new file combined_file.zip
/B - Used to copy binary files. Applies to the filename preceding it and to all following filenames. Copied files will be read by size (according to the number of bytes indicated in the file`s directory listing). An end-of-file mark is not placed at the end of the copied file. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 1:04 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| gbaddeley wrote: |
New FileStream does not do any data I/O on the file content. All it does is create / open the file.
Close() does an implicit flush to the physical file if there are pending writes.
Flush() after Close() is pointless. |
Understood. So I've just tried
| Code: |
fsZipFile.Write(buffer, 0, buffer.Length)
fsZipFile.Flush()
fsZipFile.Close()
|
..but I'm afraid it didn't make any difference. Regardless of what option I throw at the XMSC.WMQ_MESSAGE_BODY attribute, the combined file does not open / unzip. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 1:50 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
For one particular message on the queue, the queue item in MQ Explorer
states:
- fileName WHATEVER.ZIP_0002_of_0004.SPLIT
- Size = 50,000,176
In the data tab of the message
- Data length 824
- Total length 50000176
- Format <BLANK>
- coded character set identifier 1208
- Encoding 546
Should I be expecting the size of the message in MQ, and the size of the downloaded file to match? None seem to result in a file of size 50,000,176
WMQ_MESSAGE_BODY_JMS - size 50,000,001 / size on disk 50,003,968
WMQ_MESSAGE_BODY_MQ = size 50,000,177 / size on disk 50,003,968
WMQ_MESSAGE_BODY_UNSPECIFIED = 50,000,001 / size on disk 50,003,968
Edit: Ignore that, we've removed the code loading contents into a string. |
|
| Back to top |
|
|
| hughson |
| Posted: Wed May 04, 2022 2:16 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand
|
I'm curious whether you ever tried my suggestion in the first response to this thread - the queue attribute?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 2:24 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| hughson wrote: |
I'm curious whether you ever tried my suggestion in the first response to this thread - the queue attribute?
Cheers,
Morag |
Hi, it's a fair question.
The answer is, that I assumed I'd need to execute it on the server.
I don't have access to connect to that server. The server & queues were set up by IBM.
I can only connect via MQ Explorer. Have I misunderstood? Could I execute the command from my machine, or use MQ Explorer to apply the change to the queue? |
|
| Back to top |
|
|
| hughson |
| Posted: Wed May 04, 2022 2:39 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand
|
| psychodelic_drumboy wrote: |
| I can only connect via MQ Explorer. Could I execute the command from my machine, or use MQ Explorer to apply the change to the queue? |
You do either. You can use runmqsc -c or MQ Explorer to change a queue attribute. Whichever you prefer.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 3:11 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| hughson wrote: |
| psychodelic_drumboy wrote: |
| I can only connect via MQ Explorer. Could I execute the command from my machine, or use MQ Explorer to apply the change to the queue? |
You do either. You can use runmqsc -c or MQ Explorer to change a queue attribute. Whichever you prefer.
Cheers,
Morag |
On your suggestion, I have altered the "Property Control" attribute, which I determined is how PROPCTL is set via the UI.
It was set to "Compatibility", so I changed it, as you recommended, to "None", and ran the download & combine code again.
Regrefully, it didn't affect the outcome. The API code still downloaded messages without throwing an exception, but the copy /b command did not result in a useable file.
Again, I tried all 3 x WMQ_MESSAGE_BODY options.
Can you clarify, do you think the messages need to be re-sent to the queue, or does re-running the code against the existing queue/messages, as I have done, suffice? |
|
| Back to top |
|
|
| hughson |
| Posted: Wed May 04, 2022 4:32 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand
|
| psychodelic_drumboy wrote: |
| Can you clarify, do you think the messages need to be re-sent to the queue, or does re-running the code against the existing queue/messages, as I have done, suffice? |
It is hard for us to say without you showing us some of the message. I have been assuming that you have an MQRFH2 header on the front of the message and that is what is messing you up, but unless you show us the message, we can do little more than guess.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 5:30 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Well here's the result of a Binary Compare, between working files downloaded by RFHUtil and the files downloaded by the XMS API.
I use Beyond Compare to perform a "hex compare".
WMQ_MESSAGE_BODY_MQ Option - 177 extra bytes in the VS version compared to the RFHUTIL version, some evidence of RF Header in the data, 8 line items difference. I can actually see the message's named property values in the data.
WMQ_MESSAGE_BODY_JMS Option - 1 extra byte difference in size, otherwise, identical. The extra byte is represented as "00", and appears at the very end of the file.
WMQ_MESSAGE_BODY_UNSPECIFIED Option - Exactly the same as JMS Option, difference of 1 byte, existing only only 1 line of binary data.
So the RFH UTIL FILE data looks like:
ED EB F0 00 60 DA 24 C5 C4 C5 84 7C A4 00 10 AF
The data from the API-generated files looks like:
ED EB F0 00 60 DA 24 C5 C4 C5 84 7C A4 00 10 AF 00
If that extra byte wasn't in the file, those files would be identical to the file downloaded by RFHUTIL. I still wonder if my visual studio code is causing the fault here.
It's almost as if
Dim buffer(msg.BodyLength) As Byte
should be
Dim buffer(msg.BodyLength-1) As Byte
or
fsZipFile.Write(buffer, 0, buffer.Length)
should be
fsZipFile.Write(buffer, 0, buffer.Length-1)
Any thoughts? Could I be onto something? |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed May 04, 2022 6:12 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
I am very pleased to say that changing the file stream code to:
| Code: |
| fsZipFile.Write(buffer, 0, buffer.Length - 1) |
Has resolved the issue.
I have undone the other suggestions made in this thread, both to the code and the queue configuration, and the code still works correctly.
The default value of the destination XMSC.WMQ_MESSAGE_BODY property is UNSPECIFIED, so I don't even need to add that to the original code base.
I would like to thank you for your assistance with this - this has been out of action for around two years, my colleagues will be delighted to see it working again. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
