| Author |
Message
|
| mindspace |
 Posted: Tue May 01, 2018 11:54 am Post subject: Mixed data inpu, ASCII and EBCDIC hex data parsing with DFDL |
 |
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
Hi,
I am struggling on how to handle below situation..
Input: Message format similar to ISO8583-87, with additional header elements(ISO VISA NET-BASE I Technical Specifications)
Data entering into IIB:
160101005800000085640400000000000000000000090302C22000000800000000800020080000021045495160977726310426191349005255F8F1F1F6F1F9F0F0F5F2F5F5220400F102C5F20BF4F3F04040404040404040
Breakup of above message:
Header - 160101005800000085640400000000000000000000090302
Bitmaps - C2200000080000000080002008000002
Data -
ActNumber_Field2-104146516097772631(prefix 10 means 16 digits)
Date_field7-0426191349
Trace Number -005255
Reference number_field 37- F8F1F1F6F1F9F0F0F5F2F5F5
Rest of the data - 220400F102C5F20BF4F3F04040404040404040
This data is received over TCP input node, this data has mixed format, some of them are in string format(Act number, date and trace number) and some of the data is in Hex format(Reference number- ebcdic,fixed length)
I am using the standard DFDL model provided by IBM for ISO8583-87 and trying to make changes to suit my needs.
I have set TCP input node to BLOB, and then in compute node trying to modify the data...before sending to RCD node with DFDL.
So the BLOB content after TCP node is
31363031303130303538303030303030383536343034303030303030303030303030303030303030303030393033303243323230303030303038303030303030303038303030323030383030303030323130343534393531363039373737323633313034323631393133343930303532353546384631463146364631463946304630463546324635463532323034303046313032433546323042463446334630343034303430343034303430343034300d0a
In compute node, from the blob, leaving the fixed length header part, I am able to convert bitmaps back to ascii-niebble-to Blob in the output message, in this way,ISO DFDL is correctly parsing the bitmaps..i.e RCD node is recieving the orignial data for bitmaps which is c2200000080000000080002008000002 in BLOB, then rest of the fields are also parsing succesfully except the ones which are AN like the field Reference_number_field37(12 AN EBCDIC; 12 bytes)
How to handle this efficiently? By usinge esql or is their anyway with in DFDL?
Last edited by mindspace on Wed May 02, 2018 10:12 am; edited 1 time in total |
|
| Back to top |
|
 |
| timber |
| Posted: Tue May 01, 2018 2:22 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| How to handle this efficiently? By usinge esql or is their anyway with in DFDL? |
You can do it either way.
a) Parse the bitmap as an array of hexBinary with representation='text' and write it out using a different DFDL model which has representation='binary'
b) extract the bytes using ESQL, do the conversion, write the BLOB to outputRoot.BLOB.BLOB and then parse using the standard DFDL model.
I've done it both ways - the choice is yours! |
|
| Back to top |
|
|
| mindspace |
| Posted: Tue May 01, 2018 4:03 pm Post subject: |
|
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
| timber wrote: |
| Quote: |
| How to handle this efficiently? By usinge esql or is their anyway with in DFDL? |
You can do it either way.
a) Parse the bitmap as an array of hexBinary with representation='text' and write it out using a different DFDL model which has representation='binary'
b) extract the bytes using ESQL, do the conversion, write the BLOB to outputRoot.BLOB.BLOB and then parse using the standard DFDL model.
I've done it both ways - the choice is yours! |
Issue is not with bitmaps, i've followed the second(esql conversion) method as mentioned by you. But with the data fields, some data fields in the message are ebcdic blob and some or ascii blob.. please see the input message ....ive mentioned about field 37 which is ebcdic hex(F8F1F1F6F1F9F0F0F5F2F5F5-- F8 - 8, F1 - 1 etc) - ..how to handle such data using dfdl? I mean how to convert F8 to 8? |
|
| Back to top |
|
|
| timber |
| Posted: Wed May 02, 2018 1:24 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
Apologies - the long lines made the end of your last line invisible in my browser, so I missed the most important part of your question. Sounds as if you are simply asking how to convert a particular sequence of EBCDIC characters into a number.
Have you looked in detail at all of the number representations that are available in DFDL?
https://www.ibm.com/support/knowledgecenter/en/SSKM8N_8.0.0/com.ibm.dfdl.spec.doc/dfdl_13_6.htm
Looks like a textNumberRep="standard" with encoding set to the appropriate EBCDIC encoding name...but don't take my word for it. |
|
| Back to top |
|
|
| mindspace |
| Posted: Wed May 02, 2018 6:06 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
No luck, I tried different code pages but DFDL is not able to parse EBCDIC fields. I may be missing some thing here....
EBCDIC Hex values such as F8, F1 are being converted to BLOB in TCP input node as 4638, 4631, and same is being fed to RCD node DFDL, here I have set datatype to unsignedint, representation to standard, text and tried different encodings. |
|
| Back to top |
|
|
| timber |
| Posted: Wed May 02, 2018 6:21 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| DFDL is not able to parse EBCDIC fields. I may be missing something here... |
I can assure you that DFDL is perfectly capable of parsing and writing all kinds of EBCDIC fields. What you mean is that you have not been able to work out how to parse your EBCDIC numeric fields.
| Quote: |
| EBCDIC Hex values such as F8, F1 are being converted to BLOB in TCP input node as 4638, 4631 |
I don't know what that means. Are you saying that a single byte with value 0xF8 (EBCDIC '8') is being passed to DFDL as two bytes with values 0x46, 0x38? Or are you just describing how the BLOB gets displayed in your trace file?
Either way, the TCPIPInput node is just emitting whatever arrives on the wire; it does not change the BLOB in any way.
Please answer that second question carefully - I cannot advise you unless I know exactly what byte values are being passed to DFDL. |
|
| Back to top |
|
|
| mindspace |
| Posted: Wed May 02, 2018 6:37 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
| Thats right, from the debug mode, I can see that input F8 is represented as X'4638 in BLOB and same is being sent to DFDL. |
|
| Back to top |
|
|
| mindspace |
| Posted: Wed May 02, 2018 6:43 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
| other thing I noticed is that if I pass 'F8' (instead of X'4638) in blob message(alter the blob in compute node) to DFDL and with proper IBM500 ebcdic encoding..DFDL is successfully parsing it to '8' |
|
| Back to top |
|
|
| mindspace |
| Posted: Wed May 02, 2018 7:49 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2013
Posts: 26
|
I need to do some processing in esql same as bitmaps, so that actual data received (F goes to DFDL not X'4648, but for data fields in the message, which vary their positions and again not all fields are in EBCDIC..how to do this? so only specific fields if present must be converted like field_37 |
|
| Back to top |
|
|
| timber |
| Posted: Thu May 03, 2018 2:59 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
It sounds as if some previous application has (possibly accidentally) expanded the EBCDIC digits into their hex-[b]ASCII[/i] representation and written the resulting ASCII string into the ISO8583 field. Nobody else represents text numbers like that, so your only option is to
- parse it as hexBinary using DFDL
- CAST the resulting BLOB to CHARACTER using ESQL
- CAST the resulting string to INTEGER using ESQL. |
|
| Back to top |
|
|
| Dyanoo |
| Posted: Thu Oct 20, 2022 2:52 am Post subject: |
|
|
Newbie
Joined: 20 Oct 2022
Posts: 4
|
Hi, Can you please explain how you are handling BCD values for prefix length in DFDL..
ActNumber_Field2-104146516097772631(prefix 10 means 16 digits)
how you converted 10 to 16 in DFDL? |
|
| Back to top |
|
|
| timber |
| Posted: Thu Oct 20, 2022 6:51 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| @Dyanoo: Please open a new thread - your question is quite different from this one. |
|
| Back to top |
|
|
|
|
