| Author |
Message
|
| Svipdag |
 Posted: Tue Jul 15, 2014 12:01 am Post subject: Dfdl dynamic encoding is only set for first message? |
 |
|
Newbie
Joined: 14 Jul 2014
Posts: 6
|
Hi all,
At work, we are in the progress of migration from wmb7 to ib9, and I am for the first time creating a message model in dfdl for an inhouse EDI format (TDS format). After having solved several problems, I'm stuck on one problem. The thing is that this flow should be able to process messages with different CCSID (they can be at least both 278=EBCDIC and 819=ISO-Latin-1), so the encoding the dfdl parser uses in runtime must be dynamically set. Well, I found out that to achieve this, you should set
| Code: |
binaryFloatRep="{dfdl:binaryFloatRep}"
byteOrder="{$dfdl:byteOrder}"
encoding="{$dfdl:encoding}" |
in the dfdl:format. I have set this in my globally defined format, which my default format references in the dfdl schema.
So, does it work? Yes and no. When I test with a EBCDIC message directly after deploy in my integration server, it is parsed ok. And If I test with a ISO-Latin-1 message directly after deploy, it is parsed ok. BUT if I try a second message, with different encoding compared to the first, I get a parsing error on the first element in my message, as it can't figure out the initator. This is exacly what happened when I first had set the wrong encoding in my schema.
My interpertation is that the dfdl global encoding variable that the parser uses, is set the first time a message enters the flow, and set from the MQMD CCSID. But for subsequent messages, it behaves as if the parser uses the old value for encoding - it is not reset for each message!
Should it be like this? I suppose no, and the reason to this behaviour is that I have missed something. But what?
I post schema details in my next post. |
|
| Back to top |
|
 |
| Svipdag |
| Posted: Tue Jul 15, 2014 12:02 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2014
Posts: 6
|
This is my defined format, defined in a separate schema file:
****************************
| Code: |
<xsd:annotation>
<xsd:appinfo source="[url]http://www.ogf.org/dfdl/[/url]">
<dfdl:defineEscapeScheme name="EIFEscapeScheme">
<dfdl:escapeScheme escapeBlockEnd=""" escapeBlockStart=""" escapeCharacter=";" escapeEscapeCharacter=";" escapeKind="escapeCharacter" extraEscapedCharacters="%#x0D; %#x0A;" generateEscapeBlock="whenNeeded"/>
/dfdl:defineEscapeScheme>
<dfdl:defineFormat name="EIFGeneralFormat">
<dfdl:format binaryFloatRep="{dfdl:binaryFloatRep}" byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" alignment="1" alignmentUnits="bytes" occursCountKind="fixed" calendarCenturyStart="53" calendarCheckPolicy="lax" calendarDaysInFirstWeek="4" calendarFirstDayOfWeek="Monday" calendarLanguage="en-US" calendarObserveDST="yes" calendarPattern="yyyy-MM-dd'T'HH:mm:ss" calendarPatternKind="implicit" calendarTimeZone="UTC" choiceLengthKind="implicit" decimalSigned="yes" documentFinalTerminatorCanBeMissing="yes" emptyValueDelimiterPolicy="initiator" escapeSchemeRef="tns:EIFEscapeScheme" fillByte="0" floating="no" ignoreCase="no" initiatedContent="no" initiator="" leadingSkip="0" lengthKind="delimited" lengthUnits="characters" nilValueDelimiterPolicy="initiator" outputNewLine="%CR;%LF;" representation="text" separator="" separatorPosition="infix" separatorSuppressionPolicy="anyEmpty" sequenceKind="ordered" terminator="" textBidi="no" textBooleanFalseRep="false" textBooleanJustification="left" textBooleanPadCharacter="%SP;" textBooleanTrueRep="true" textCalendarJustification="left" textCalendarPadCharacter="%SP;" textNumberCheckPolicy="lax" textNumberJustification="right" textNumberPadCharacter="%SP;" textNumberRep="standard" textNumberRounding="pattern" textNumberRoundingMode="roundUp" textPadKind="none" textStandardBase="10" textStandardDecimalSeparator="." textStandardExponentRep="E" textStandardGroupingSeparator="," textStandardInfinityRep="Inf" textStandardNaNRep="NaN" textStandardZeroRep="" textStringJustification="left" textStringPadCharacter="%SP;" textTrimKind="padChar" textZonedSignStyle="asciiStandard" trailingSkip="0" truncateSpecifiedLengthString="no">/dfdl:format>
</dfdl:defineFormat>
</xsd:appinfo>
</xsd:annotation> |
********************************
And this is my default format declaration in my message model schema file (which imports the first file), the one that is used all over this schema file where all my message types and elements are defined:
************************************
| Code: |
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format ref="EIF:EIFGeneralFormat"/>
</xsd:appinfo>
</xsd:annotation> |
*****************************
I have tried to set the encoding locally on each element instead, but I got another problem. And I don't think it should be the right solution anyway.
I'm thankful for any ideas! |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jul 15, 2014 3:01 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Your diagnosis is correct - I am fairly sure that this is a defect.
I am also fairly sure that it has been fixed. So you should make sure that you are on the latest fix pack. If you are already on IIB 9.0.0.1 then you should check the list of published fixes, and select the latest fix that mentions DFDL.
http://www-01.ibm.com/support/docview.wss?uid=swg27041010
You probably need http://www-01.ibm.com/support/docview.wss?uid=swg1IC97526 but I would advise you to take the latest ( highest numbered ) DFDL fix or else wait a couple of weeks for IIB 9.0.0.2.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 15, 2014 4:24 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I'm already running IIB 9.0.0.2. It was released last week.
http://www.mqseries.net/phpBB2/viewtopic.php?t=67904
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Svipdag |
| Posted: Tue Jul 15, 2014 11:16 pm Post subject: |
|
|
Newbie
Joined: 14 Jul 2014
Posts: 6
|
Ok, I have now installed 9.0.0.2, restarted my broker and verified that the version property reads 9.0.0.2.
Unfortunately, the problem persists! 
Isn't this a very basic error? This can't be the first IB9 flow needing to be able to process multiple character encodings?
This is why I think it is strange if this is a IB9 defect. There isn't some setting I have missed? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 15, 2014 11:24 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Time for a PMR. Mention this thread in the PMR.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| jmenning |
| Posted: Fri Feb 27, 2015 12:52 pm Post subject: |
|
|
Newbie
Joined: 27 Feb 2015
Posts: 8
|
Hello,
We are running into the same issue in the context of the HL7 messages defined in the IBM HealthCare pack. HL7 messages are allowed to have various delimiters, which the Healthcare pack stores in a variable. However, the first message through the flow seems to permanently set the delimiter variable. If subsequent messages does not have the same set of delimiters, the parsing fails. The only resolution at this point is to restart the Integration Server (execution group), which is obviously suboptimal.
Just wondering if the PMR opened above bore any fruit, or if others have suggestions.
Best,
Jesse |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 27, 2015 1:05 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmenning wrote: |
| if others have suggestions. |
Have you tried using any of the HL7 schemas on GitHub? Do these show the same symptom?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jmenning |
| Posted: Mon Mar 02, 2015 11:29 am Post subject: |
|
|
Newbie
Joined: 27 Feb 2015
Posts: 8
|
Vitor,
Thanks for the helpful reply.
I just tried out the github schemas and get the same results (The github and HealthCare pack schemas are very similar).
The issue with both appears to be IBM's DFDL parser implementation: once dfdl:setVariable is invoked for a particular variable, that variable value is shared across all instances of the flow, and cannot be modified until the integration server is restarted. I'm sure this increases speed, but definitely at a hit to flexibility. I was hoping that there was an option to change that functionality.
dfdl:newVariableInstance looked promising, but unfortunately isn't supported in the current version of IIB.
Jesse |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 02, 2015 11:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmenning wrote: |
| The issue with both appears to be IBM's DFDL parser implementation: once dfdl:setVariable is invoked for a particular variable, that variable value is shared across all instances of the flow, and cannot be modified until the integration server is restarted. I'm sure this increases speed, but definitely at a hit to flexibility. I was hoping that there was an option to change that functionality. |
| smdavies99 wrote: |
| Time for a PMR. Mention this thread in the PMR |
As the OP hasn't posted any response to what came out of their PMR (or indeed ever posted since then) I think you need your own PMR. This will confirm:
- it's a defect that was supposed to be fixed in 9.0.0.2 but apparently isnt't
- it's a defect that was supposed to be fixed in 9.0.0.2 but didn't make the cut, so you need to wait for 9.0.0.3 or apply an APAR
- it's functioning as designed and you need an RFE
For my money, if @kimbert thinks this is a defect not a feature then it's a defect. Given who kimbert is and what he does.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Mar 02, 2015 12:13 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| jmenning wrote: |
Vitor,
The issue with both appears to be IBM's DFDL parser implementation: once dfdl:setVariable is invoked for a particular variable, that variable value is shared across all instances of the flow, and cannot be modified until the integration server is restarted. I'm sure this increases speed, but definitely at a hit to flexibility. I was hoping that there was an option to change that functionality.
|
You explanation struck a chord. It could be similar/the same as this
http://www-01.ibm.com/support/docview.wss?uid=swg1IT07285&myns=swgws&mynp=OCSSKM8N&mync=E&cm_sp=swgws-_-OCSSKM8N-_-E
If you aren't already signed up then i'd recommend that you do so.
https://www.ibm.com/support/mynotifications
you get an almost daily list of issues/workarounds and when fixed' email. Very useful.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| jmenning |
| Posted: Thu Mar 05, 2015 10:38 am Post subject: |
|
|
Newbie
Joined: 27 Feb 2015
Posts: 8
|
| Thank you very much, I'll sign up for the email and keep this post updated with the result of the PMR. |
|
| Back to top |
|
|
| shanson |
| Posted: Tue Mar 10, 2015 6:01 am Post subject: |
|
|
Partisan
Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley
|
|
| Back to top |
|
|
| jmenning |
| Posted: Tue Mar 10, 2015 5:21 pm Post subject: |
|
|
Newbie
Joined: 27 Feb 2015
Posts: 8
|
| That is indeed the APAR I'm looking for. Many thanks! |
|
| Back to top |
|
|
| jmenning |
| Posted: Thu Mar 19, 2015 9:56 am Post subject: |
|
|
Newbie
Joined: 27 Feb 2015
Posts: 8
|
| Just a brief addendum, per IBM support this fix will also be included in WMB V8.0.0.6. That fixpack is set to be delivered 3Q 2015. |
|
| Back to top |
|
|
|
|
