| Author |
Message
|
| kotagiriaashish |
 Posted: Wed Apr 03, 2013 11:29 am Post subject: Handling Packed Data Coming From AS400 |
 |
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
We are receiving a mix of packed and text data in a same record from AS400 machine... I defined message set to felicitate packed data as HEXBINARY in message set and normal text as string.... the problem is we wrote a custom java program to convert the incoming packed data to long and are getting some output but the application teem claims the numbers are wrong.
| Code: |
public class PackedDecimal {
public static java.lang.Long packedtostring(byte[] packed)
{
java.lang.Long data = null ;
try{
data = PackedDecimal.parse(packed);
}
catch(Exception E){
//data = E.toString();
}
return data;
}
public static long parse(byte[] pdIn) throws Exception {
// Convert packed decimal to long
final int PlusSign = 0x0C; // Plus sign
final int MinusSign = 0x0D; // Minus
final int NoSign = 0x0F; // Unsigned
final int DropHO = 0xFF; // AND mask to drop HO sign bits
final int GetLO = 0x0F; // Get only LO digit
long val = 0; // Value to return
for (int i = 0; i < pdIn.length; i++) {
int aByte = pdIn[i] & DropHO; // Get next 2 digits & drop sign bits
if (i == pdIn.length - 1) { // last digit?
int digit = aByte >> 4; // First get digit
val = val * 10 + digit;
// System.out.println("digit=" + digit + ", val=" + val);
int sign = aByte & GetLO; // now get sign
if (sign == MinusSign)
val = -val;
else {
// Do we care if there is an invalid sign?
if (sign != PlusSign && sign != NoSign){
//throw new Exception("OC7");
}
}
} else {
int digit = aByte >> 4; // HO first
val = val * 10 + digit;
// System.out.println("digit=" + digit + ", val=" + val);
digit = aByte & GetLO; // now LO
val = val * 10 + digit;
// System.out.println("digit=" + digit + ", val=" + val);
}
}
return val;
} // end parse()
// Test the above
public static void main(String[] args) throws Exception {
byte[] pd = new byte[] { 0x19, 0x2C }; // 192
System.out.println(PackedDecimal.parse(pd));
pd = new byte[] { (byte) 0x98, 0x44, 0x32, 0x3D }; // -9844323
System.out.println(PackedDecimal.parse(pd));
pd = new byte[] { (byte) 0x98, 0x44, 0x32 }; // invalid sign
System.out.println(PackedDecimal.parse(pd));
}
}
|
can anyone suggest?? |
|
| Back to top |
|
 |
| kimbert |
| Posted: Wed Apr 03, 2013 11:40 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
That is not a very useful description of the problem. Please answer the following questions:
- which version of WMB are you using?
- why are you not using the MRM parser ( if on v7 ) or the DFDL parser ( if on v8 ) to decode the packed decimals?
- specifically, what do the 'application team' think is wrong with your numbers? Please give an example. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Apr 03, 2013 11:41 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I don't really want to try and read and mentally debug that java code.
I'll suggest that using Java code to unpack data or parse data in broker is the kind of thing that should only be done as a last resort.
I'll suggest that a Broker message definition - either MRM or DFDL- should be configurable to understand what data is packed and what data is not packed and have enough information to unpack it for you.
I'll also suggest that if you can provide more information about what data is being sent in, and what that is being unpacked into, and why the developers think it's "wrong", then you may get better assistance. |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Wed Apr 03, 2013 11:44 am Post subject: |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| kimbert wrote: |
That is not a very useful description of the problem. Please answer the following questions:
- which version of WMB are you using?
- why are you not using the MRM parser ( if on v7 ) or the DFDL parser ( if on v8 ) to decode the packed decimals?
- specifically, what do the 'application team' think is wrong with your numbers? Please give an example. |
We are running on Version 6.1 we are using MRM to convert the packed data to Hexadecimal
From As400 - > Ÿ¶Úgª
When Message Set parses it -> 9FB6DA67AA
Our Java Program OP -> 106174068
expected value -> 76215875
we are completely in dark |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Wed Apr 03, 2013 12:17 pm Post subject: |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| mqjeff wrote: |
I don't really want to try and read and mentally debug that java code.
I'll suggest that using Java code to unpack data or parse data in broker is the kind of thing that should only be done as a last resort.
I'll suggest that a Broker message definition - either MRM or DFDL- should be configurable to understand what data is packed and what data is not packed and have enough information to unpack it for you.
I'll also suggest that if you can provide more information about what data is being sent in, and what that is being unpacked into, and why the developers think it's "wrong", then you may get better assistance. |
we tried setting the Physical Type to PackedDecimal
and Local Element Type to xsd:long
we get a parsing exception. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Apr 03, 2013 12:28 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| From As400 - > Ÿ¶Úgª |
Those are characters. Please explain how we are supposed to interpret them? |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Wed Apr 03, 2013 12:31 pm Post subject: |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| kimbert wrote: |
| Quote: |
| From As400 - > Ÿ¶Úgª |
Those are characters. Please explain how we are supposed to interpret them? |
They sent us a sample data file for our internal testing... thats what we see when we open the file... according to the mapping speck these characters are supposed to be packed data.. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Apr 03, 2013 10:47 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| kotagiriaashish wrote: |
They sent us a sample data file for our internal testing... thats what we see when we open the file... according to the mapping speck these characters are supposed to be packed data.. |
As one puppet said to another, 'That's not the way to do it'
{only the Brits will understand that}
Being sent a file like that is not really going to help. There could have been all sorts of codepage conversions done along the way without you knowing about it. Even in the application that you use to display the data...
The ONLY way for sure is to do one of the following
1) Get a character by character dump of the data from the AS400 PLUS knowing what the CCSID of the data is that you are reading
2) Get the 'sample' data placed in an accessible location on the AS400 and YOU collect it with again knowing what CCSID you should read the data with.
If you 'fix' your code to work with the sample dataset who knows what will happen when you start working with real ones. IMHO there is probably very little chance of it working.
You should really be using a Message Model to assist the parsing of the data (as recommended by previous posters). not doing so is frankly just asking for trouble.
I once did some work for a PC Maker. They said process this XML data. Ok, I said, I'll use the freely available parser from Oracle. No you won't said the PC Maker. I had to write my own XML Parser. When it went live we spent the next 6 months tweaking the parser to cover all the variances in the data. do you really want to have to do the same thing?
(The PC Maker is still in business but is trying to take itself private)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Wed Apr 03, 2013 11:32 pm Post subject: |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| smdavies99 wrote: |
| kotagiriaashish wrote: |
They sent us a sample data file for our internal testing... thats what we see when we open the file... according to the mapping speck these characters are supposed to be packed data.. |
As one puppet said to another, 'That's not the way to do it'
{only the Brits will understand that}
Being sent a file like that is not really going to help. There could have been all sorts of codepage conversions done along the way without you knowing about it. Even in the application that you use to display the data...
The ONLY way for sure is to do one of the following
1) Get a character by character dump of the data from the AS400 PLUS knowing what the CCSID of the data is that you are reading
2) Get the 'sample' data placed in an accessible location on the AS400 and YOU collect it with again knowing what CCSID you should read the data with.
If you 'fix' your code to work with the sample dataset who knows what will happen when you start working with real ones. IMHO there is probably very little chance of it working.
You should really be using a Message Model to assist the parsing of the data (as recommended by previous posters). not doing so is frankly just asking for trouble.
I once did some work for a PC Maker. They said process this XML data. Ok, I said, I'll use the freely available parser from Oracle. No you won't said the PC Maker. I had to write my own XML Parser. When it went live we spent the next 6 months tweaking the parser to cover all the variances in the data. do you really want to have to do the same thing?
(The PC Maker is still in business but is trying to take itself private) |
I will keep that in mind... but can anyone suggest if this is what as400 packed data looks like when opened in a normal windows text editor?? because the data flow is like
PKMS -> AS400 -> MQ/MB since the data being sent is not obeying Packed data converts and parsers... im a bit doubtful |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 04, 2013 8:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Your packed decimal has at least been converted once as text.
Why? because the hex you showed does not represent a packed decimal (comp-3).
A packed decimal looks like
999999999X where X is any of C,F,or D and 9 represents digits. This is not what your hex looks like. Make sure the data does not get converted by a channel.
Also the broker needs to parse the unconverted data. So no conversion flag checked on the input node!.
Make sure when importing the copy book that you get big Endian checked.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Apr 05, 2013 3:16 am Post subject: Re: Handling Packed Data Coming From AS400 |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| kotagiriaashish wrote: |
| We are receiving a mix of packed and text data in a same record from AS400 machine. |
The application team creating the message needs to specify which bytes are character data, which bytes are non-character data, and how it encodes the bytes for each element in the message.
| kotagiriaashish wrote: |
| From As400 - > Ÿ¶Úgª |
As kimbert points out, displaying non-character data as if it were character data is not meaningful. It gets worse if the characters displayed are control characters, or if the non-character data doesn't actually map to any available characters.
| kotagiriaashish wrote: |
| When Message Set parses it -> 9FB6DA67AA |
As fjb_saper points out, this is not packed decimal. Confirm with the application team what bytes they actually sent. If the bytes sent don't match the bytes received, try to narrow down where that happens, and why.
| kotagiriaashish wrote: |
| Our Java Program OP -> 106174068 |
This is not quite the output that the above Java program would generate for X'9FB6DA67AA'.
| kotagiriaashish wrote: |
| expected value -> 76215875 |
Confirm with the application team how they encoded the bytes they actually sent.
| kotagiriaashish wrote: |
| according to the mapping speck these characters are supposed to be packed data |
The bytes in question can either be characters or they can be "packed data". They can't be both.
| kotagiriaashish wrote: |
| but can anyone suggest if this is what as400 packed data looks like when opened in a normal windows text editor? |
That's like asking what a picture sounds like, if routed to the speakers, rather than to the display. But if a normal windows text editor were to display AS400 "packed data", it might look something like that. |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Fri Apr 05, 2013 7:37 am Post subject: |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| fjb_saper wrote: |
Your packed decimal has at least been converted once as text.
Why? because the hex you showed does not represent a packed decimal (comp-3).
A packed decimal looks like
999999999X where X is any of C,F,or D and 9 represents digits. This is not what your hex looks like. Make sure the data does not get converted by a channel.
Also the broker needs to parse the unconverted data. So no conversion flag checked on the input node!.
Make sure when importing the copy book that you get big Endian checked.
Have fun |
Thanks for the suggestion i will check with the application team |
|
| Back to top |
|
|
| kotagiriaashish |
| Posted: Fri Apr 05, 2013 7:39 am Post subject: Re: Handling Packed Data Coming From AS400 |
|
|
Disciple
Joined: 06 Aug 2011
Posts: 165
|
| kotagiriaashish wrote: |
| kotagiriaashish wrote: |
| Our Java Program OP -> 106174068 |
This is not quite the output that the above Java program would generate for X'9FB6DA67AA'.
|
Thats what we are getting when we pass the packed data to our programme... i will check with the AS400 team and update if the misunderstanding gets resolved. |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Apr 06, 2013 1:00 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
A packed decimal looks like
999999999X where X is any of C,F,or D and 9 represents digits |
Actually that would be an external decimal. A packed decimal encodes one decimal digit in every nibble ( 4 bits ) with the sign being indicated by a special value in either the first or last nibble.
There are many conventions for packed decimal data, and the DFDL parser ( v8 required ) can handle almost any variant. The MRM parser can handle a wide range of packed decimal styles, and it may be able to handle this one. But we don't know yet, because you have not supplied the required information. It is absolutely vital to provide the following information
a) the exact bytes of the packed decimal field, exactly as they are received by the message flow. Nothing else will do.
b) the signed decimal number that those bytes represent |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Apr 07, 2013 7:05 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| kimbert wrote: |
| Quote: |
A packed decimal looks like
999999999X where X is any of C,F,or D and 9 represents digits |
Actually that would be an external decimal. A packed decimal encodes one decimal digit in every nibble ( 4 bits ) with the sign being indicated by a special value in either the first or last nibble.
There are many conventions for packed decimal data, and the DFDL parser ( v8 required ) can handle almost any variant. The MRM parser can handle a wide range of packed decimal styles, and it may be able to handle this one. But we don't know yet, because you have not supplied the required information. It is absolutely vital to provide the following information
a) the exact bytes of the packed decimal field, exactly as they are received by the message flow. Nothing else will do.
b) the signed decimal number that those bytes represent |
Thanks Kimbert for clarifying the language. I was referring to the comp-3 MF format which is sometimes called packed decimal on the MF. Thanks for letting us know to call it EXTERNAL Decimal.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
