| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Does RfhUtil Write No Headers exist in the XMS API? |
View previous topic :: View next topic |
| Author |
Message
|
| psychodelic_drumboy |
 Posted: Thu Apr 07, 2022 3:54 am Post subject: Does RfhUtil Write No Headers exist in the XMS API? |
 |
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Hello and good afternoon to you fine and learned people...
I have written a tremendous piece of code that pulls iBytesMessage items from an MQ queue, in a spritely fashion.
However, I need to implement the option that you see in RfhUtil, under the write menu, which states "No Headers". The provider of the messages has insisted on the format, and won't alter it...even when I ask nicely!
We didn't historically need to use that option, but now we do, otherwise the resultant file is unusable.
So my question is, where does an equivalent to the "no headers" option in the RfhUtil tool reside within the IBM.XMS API?
I'm using version 9.2.5 of the IBMXMSDotnetClient by IBM, as found via the NuGet route in Visual Studio.
Thanks in advance, I know you guys will probably have the answer to this one.. |
|
| Back to top |
|
 |
| hughson |
| Posted: Thu Apr 07, 2022 8:12 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
In fact you might be able achieve this without changing your application code at all.
If you change the queue definition in this way:-
| Code: |
| ALTER QLOCAL(q-name) PROPCTL(NONE) |
This option will cause any RFH2 user property headers to be removed from the message before they are given to an application which cannot deal with them. Any applications reading from this queue that can handle message properties should be unaffected by this setting.
Having said that, IBM.XMS API is one of those APIs which can handle message properties, so this option may be over-ridden by the the XMS API asking for the message properties.
Worth trying anyway.
If you do want to change your application, you might be looking for the XMSC_WMQ_MESSAGE_BODY property with value XMSC_WMQ_MESSAGE_BODY_MQ, but I am not an XMS expert, so someone else might jump in with a better answer.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Fri Apr 08, 2022 12:11 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Morag, many thanks for your detailed response.
I am trying the .NET code change first, by adding this additional line in my code:
| Code: |
| destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_MQ) |
We are now examining the effect on the output files; I will report back as soon as I know.
If this fails, we'll see if we can alter the queue definition using the command you mentioned. I almost certainly don't have access to run that command, so I would need to find the person who can. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Fri Apr 08, 2022 1:34 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Ok, so I ran the code with all three possible WMQ_MESSAGE_BODY options
- Not set (documentation says default is XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED)
- XMSC_WMQ_MESSAGE_BODY_JMS
- XMSC_WMQ_MESSAGE_BODY_MQ
Unfortunately, none of these options resulted in a working set of files.
Interestingly, when I did a binary file compare with the BeyondCompare tool, I actually saw no header data when using XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED, but header data when XMSC_WMQ_MESSAGE_BODY_MQ was used.
I am now obtaining a set of files that definitely work, using RfhUtil, so that:
1 - I can confirm that the messages on the queue are in working order
2 - I can maybe spot differences between the RFHUTIL version and the versions with various WMQ_MESSAGE_BODY options.
I'll let you know what differences I find, but thanks for your help so far. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Fri Apr 08, 2022 1:54 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| psychodelic_drumboy wrote: |
1 - I can confirm that the messages on the queue are in working order
|
They're not! Apologies, I should have performed this stage beforehand.
We've been given duff data. We should see 4 x messages on the queue, where 1,2 + 3 are the same size. That is not the case, so please ignore my findings so far, your code change suggestion may still be the right one. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed Apr 13, 2022 12:37 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Ok so we've been sent four new mesages from our supplier, which, size-wise, look valid.
I'm again trying all three possible " XMSC_WMQ_MESSAGE_BODY" options, to see if any of those address our issue.
I will report back shortly - my colleague is now validating the downloads. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 14, 2022 12:44 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Define the destination as "queue://QMGR/QNAME?targetClient=1"
The QMGR name is optional. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Fri Apr 22, 2022 4:07 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| fjb_saper wrote: |
Define the destination as "queue://QMGR/QNAME?targetClient=1"
The QMGR name is optional. |
Thanks fjb_saper. I will try that. The API certainly accepts the targetClient argument, I can see the messages coming down right now.
I've been provided with a fresh set of messages but my wider team haven't yet verified that they're valid (as I mentioned, the last set were not). |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Tue Apr 26, 2022 5:39 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Ok, so I have tried a number of options, recommended in this thread, to resolve the issue.
To be clear, we are sent IbytesMessages which are actually zip files; files larger than 47.6MB are divded into 47.6 MB segments.
On the queue right now I have the following iBytes messages:
- 1 x 16MB message (that, when renamed to .ZIP, should unzip without error)
- A set of 4 x divided zip messages (3 x 47.6 MB, 1 x 47.4MB)
- A set of 3 x divided zip messages (2 x 47.6 MB, 1 x 44 MB)
We use an EXE utility that glues the divided zips together into one zip file. I didn't write the utility, but I believe it just pulls all the data from the multiple groups of files into a single file and names it .ZIP
I have tried all three XMSC_WMQ_MESSAGE_BODY options:
- XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED
- XMSC_WMQ_MESSAGE_BODY_JMS
- XMSC_WMQ_MESSAGE_BODY_MQ
..but none result in working file sets.
When using the XMSC_WMQ_MESSAGE_BODY_MQ against the 16MB "single file message", this results in an invalid zip, but the other two options result in a file that will unzip.
I have also tried the "queue://QMGR/QNAME?targetClient=1" argument, and tried both UNSPECIFIED and JMS options. Neither resulted in file sets that could be reconstituted. I ruled out XMSC_WMQ_MESSAGE_BODY_MQ due to the finding above.
Lastly, my colleague has manually downloaded the messages with RFHUTIL, using the "No Headers" option in the Write Menu - those messages have been found to be usable, so we have a valid set of messages on our queue - we just can't get them down via the XMS API.
So, we still don't have the answer yet.
Do you have any other questions about what I've tried so far?
Any assistance you can provide on this would be very much appreciated. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Tue Apr 26, 2022 5:58 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
I think at this point it would be useful for me to share my code. I've chosen to write it in Visual Basic.
Please note that this code does not, at any time, throw an exception, but the resultant files (as described above) don't turn out to be valid.
I've stripped out the other message types from the excerpt because that's not relevant to what we're talking about. The messages in this case are all IBytesMessage.
| Code: |
Imports System
Imports System.IO
Imports IBM.XMS
Imports System.Diagnostics
Public Class clsMQ
'MQ Connection Setings Specified By User
Private mstrQueueManager As String
Private mstrHostName As String
Private mstrDestinationURI As String
Private mstrDestination As String
Private mstrMQErrorMessage As String = ""
'Local variable to count messages received
Private mIntMessagesReceived As Integer = 0
'The specified prefix for each file
Private mstrFilePrefix As String = ""
Private mstrFileExtension As String = ""
'Deletion Property
Private mBlnDeleteFromQueue As Boolean = False
Public Enum enBinaryDownloadType
enWMQ_MESSAGE_BODY_JMS = 0
enWMQ_MESSAGE_BODY_MQ = 1
enWMQ_MESSGAE_BODY_UNSPECIFIED = 2
End Enum
Private mIbytesDownloadType As enBinaryDownloadType
Public Property iBytesOption As enBinaryDownloadType
Get
Return mIbytesDownloadType
End Get
Set(value As enBinaryDownloadType)
mIbytesDownloadType = value
End Set
End Property
Public Sub RetrieveFiles()
'Create MQ Objects so they can be destroyed in the Finally block
Dim blnErrorOccurred As Boolean = False
Dim Consumer As IMessageConsumer = Nothing
Dim session As ISession = Nothing
Dim destination As IDestination = Nothing
Dim connection As IConnection = Nothing
Dim cfConnectionFactory As IConnectionFactory = Nothing
Dim strFileNameFromMsg As String = ""
Try
'Create an XMS Factory Object
Dim FactoryFactory As XMSFactoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ)
'Create a ConnectionFactory Object
cfConnectionFactory = FactoryFactory.CreateConnectionFactory()
'this variable will contain the full path of any file downloaded from MQ
Dim strMQMessageOutputFileDestinationFilePath As String = ""
'This variable will be used to evaluate whether the MQ Message Output file exists
Dim fiMQMessageOutputFile As FileInfo = Nothing
'Set various Connection Factory properties
cfConnectionFactory.SetStringProperty(XMSC.WMQ_HOST_NAME, Me.HostName)
cfConnectionFactory.SetIntProperty(XMSC.WMQ_PORT, 1414)
cfConnectionFactory.SetStringProperty(XMSC.WMQ_CHANNEL, "SYSTEM.DEF.SVRCONN")
cfConnectionFactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, 1)
cfConnectionFactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, Me.QueueManager)
cfConnectionFactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, 0)
'Create a new Iconnection object via the Connection Factory
connection = cfConnectionFactory.CreateConnection()
'Create a sesion via the Connection Object, using ClientAcknowledge mode
'ClientAcknowledge is being used because it allows us to control precisely
'when a message should be removed from the queue
session = connection.CreateSession(False, AcknowledgeMode.ClientAcknowledge)
'Create a destination using the Session Object
destination = session.CreateQueue(Me.mstrDestinationURI)
destination.SetIntProperty(XMSC.DELIVERY_MODE, 1)
Select Case Me.iBytesOption
Case enBinaryDownloadType.enWMQ_MESSAGE_BODY_JMS
destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_JMS)
Case enBinaryDownloadType.enWMQ_MESSAGE_BODY_MQ
destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_MQ)
Case enBinaryDownloadType.enWMQ_MESSGAE_BODY_UNSPECIFIED
destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_UNSPECIFIED)
Case Else
'do nothing
End Select
'Create a consumer using the Session & Destination Objects
Consumer = session.CreateConsumer(destination)
connection.Start()
'IMessage is the base class that is returned from the Consumer's Receive method
Dim recvMsg As IMessage = Nothing
Do
' Retrieve message from Queue, using the XML file's specified time out
'ReceiveNoWait will run until
recvMsg = Consumer.ReceiveNoWait
If recvMsg Is Nothing Then
'No more messages left to process, exit loop
Exit Do
Else
'Increment File Counter
mIntMessagesReceived += 1
'Obtain filename from MQ Message Meta Data (if available)
strFileNameFromMsg = If(Not recvMsg.PropertyExists("fileName"), "", recvMsg.GetStringProperty("fileName"))
If TypeOf (recvMsg) Is IBytesMessage Then
'Binary Message
Dim msg As IBytesMessage = CType(recvMsg, IBytesMessage)
Dim buffer(msg.BodyLength) As Byte
msg.ReadBytes(buffer)
'The PrepareDestinationFile Function will generate a unique file name for the new file
'and ensure that the file does not already exist on the drive
strMQMessageOutputFileDestinationFilePath = PrepareDestinationFile(strFileNameFromMsg)
'A FileStream object is needed to write a binary array to a file
Dim fsZipFile As FileStream = New FileStream(strMQMessageOutputFileDestinationFilePath, FileMode.Create)
'Write the contents of the Byte Array to the File via the FileStream object
fsZipFile.Write(buffer, 0, buffer.Length)
fsZipFile.Close()
End If
End If
Loop 'Next message in MQ Queue
'Close connection to MQ
connection.Close()
Catch mqe As XMSException
blnErrorOccurred = True
'MQ Related Error Handler
'Any errors which occurred during code execution will be logged to the event log as Warnings
mstrMQErrorMessage = "Reason:" & mqe.Message.ToString & " / " & "Reason Code:" & mqe.Source.ToString()
Exit Sub
Catch ex As Exception
blnErrorOccurred = True
'General (Non-MQ Related) Error Handler
'Any errors which occurred during code execution will be logged to the event log as Warnings
Finally
'Final Event Log entry will be either Information or Error, depending on whether
'any unexpected errors occurred during code execution
'destroy all objects
If Not Consumer Is Nothing Then
Consumer = Nothing
End If
If Not destination Is Nothing Then
destination = Nothing
End If
If Not session Is Nothing Then
session = Nothing
End If
If Not connection Is Nothing Then
connection = Nothing
End If
If Not cfConnectionFactory Is Nothing Then
cfConnectionFactory = Nothing
End If
End Try
End Sub
Private Function PrepareDestinationFile(ByVal strFileNameFromMsg As String) As String
Try
Dim fiDestinationFile As FileInfo
If strFileNameFromMsg.Length = 0 Then
'Generate unique file name using specified file prefix & suffix, specified in XML config file
PrepareDestinationFile = Me.MessageFileDestination & Me.FilePrefix & System.DateTime.Now.ToString("yyyyMMddhhmmss") & "_" & mIntMessagesReceived.ToString() & "." & Me.FileExtension
Else
'use file destination found in MQ Message
PrepareDestinationFile = Me.MessageFileDestination & strFileNameFromMsg
End If
'Ensure destination file does not exist
fiDestinationFile = New FileInfo(PrepareDestinationFile)
If fiDestinationFile.Exists Then fiDestinationFile.Delete()
Catch ex As Exception
PrepareDestinationFile = ""
End Try
End Function
End Class
|
Do you see any obvious errors here?
The best case scenario would be that I'm simply doing something wrong when either reading the message data from MQ into the "buffer" byte array, or writing the "buffer" ByteArray data into the file (in the fsZipFile object)
It's also apparent that I load the data into a variable called content and then don't use it.
| Code: |
content As String = Text.Encoding.UTF8.GetString(buffer)
|
This can be removed.
Last edited by psychodelic_drumboy on Wed May 04, 2022 4:21 am; edited 1 time in total |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Apr 27, 2022 9:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I am not seeing you writing/producing the message. Just consuming the message.
Forget any problems with the message. The errors have already been made.
Forget anything about the destination set up as you are consuming the message and not producing it...
If you are producing it, forget all about setting the destination properties.
The only thing you need to do is:
session.createQueue("queue://qmgr/qname?targetClient=1").
Read the message as an IBytesMessage.
I would however question why your are reading the message as UTF-8?
It's a zip file and not a String!!!
| Code: |
| Dim content As String = Text.Encoding.UTF8.GetString(buffer) |
.
Also I have no idea what this does as a pointer to your buffer?
May be comment out that line and see what you get?
Can you share how the message is produced?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Wed Apr 27, 2022 11:36 pm Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
Hi,
So to be clear...
| Quote: |
| Just consuming the message. |
Correct, the solution only concerns itself with reading messages sent by others.
| Quote: |
| I would however question why your are reading the message as UTF-8? It's a zip file and not a String!!! |
| Quote: |
| Also I have no idea what this does as a pointer to your buffer? |
This line of code is redundant. You're right that I should remove it; Maybe it's affecting the integrity of the buffer object?
I'll try again, with it removed, and report back. If that's all it is, then that would be great. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Thu Apr 28, 2022 2:15 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
I'm sorry to report that even with the "?targetClient=1" value in the WMQ_HOST_NAME, and with the redundant line of code removed, this still results in files that cannot be reconstituted.
Is there anything I could look for, in the resultant files, that could provide an angle as to what's wrong? It's possible that I could get a copy of the working files downloaded by RHFUTIL, and perform some kind of comparison. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 28, 2022 2:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| psychodelic_drumboy wrote: |
| … this still results in files that cannot be reconstituted. |
Please explain “reconstituted”. From what? To what?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| psychodelic_drumboy |
| Posted: Thu Apr 28, 2022 3:09 am Post subject: |
|
|
Novice
Joined: 07 Apr 2022
Posts: 20
|
| bruce2359 wrote: |
| psychodelic_drumboy wrote: |
| … this still results in files that cannot be reconstituted. |
Please explain “reconstituted”. From what? To what? |
We are sent, for example, three iBytes messages with names such as:
- "zip_file_28_04_2022_1_of_3.SPLIT"
- "zip_file_28_04_2022_2_of_3.SPLIT"
- "zip_file_28_04_2022_3_of_3.SPLIT"
These files, on their own, cannot be opened by tool like WinZip.
The utility simply combines the contents of these 3 x SPLIT files (I call that "reconstituting") into a single file and names that file .ZIP. I didn't write the utility, so note that this is only my interpretation based on the end results.
When the messages are downloaded via RfhUtil, with the "No Headers" option selected, the ZIP file created by the utility will open and unzip.
When the messages are downloaded via my code, the ZIP file created by the utility will not open and will not unzip.
There is still something unique to how RFHUtil works, that is different to my supplied code. I have no idea what it is.
I don't know why 47.6 MB is the maximum permitted size of a message in this environment. I also don't know if that is a limitation of MQ, or a constraint the supplier of the messages has mandated.
As you can tell - I don't know very much! I'm sorry about that. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2, 3 Next |
Page 1 of 3 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
