| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| IIB9: DFDL, Random Junk Characters in incoming files |
« View previous topic :: View next topic » |
| Author |
Message
|
| akil |
 Posted: Tue Dec 23, 2014 9:53 pm Post subject: IIB9: DFDL, Random Junk Characters in incoming files |
 |
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
Hi
We have a message flow that receives flat files from a Credit Card system , we've modelled the file structures using DFDL, and all works fine, except on occasions where we find some non-printable characters in the file .
These characters appear with no specific pattern and come anywhere in the file, sometimes multiple times.
The 2 kinds of flows
FileInput (validationOn ,record identification: FixedLength) --> compute -->..
FileInput (validationOn ,record identification: ParsedRecordSequence) --> compute -->..
When these characters come in, the validation fails .
Question: Is it possible to escape / filter out certain characters in the bitstream globally, that would skip these characters and pass only the rest ahead?
_________________
Regards |
|
| Back to top |
|
 |
| McueMart |
| Posted: Wed Dec 24, 2014 1:21 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
I think you'll have to either tell your source system to not send junk characters in the data, or have a pre-processor flow which strips any junk data out the files.
I'm pretty sure there is no way to tell broker to disregard certain byte ranges in input data. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Dec 24, 2014 2:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Everytime I hear the term "junk characters", I suspect it is simply a problem with code page conversion. Or sometimes it can be control characters than unicode uses.
Rarely, if ever, do applications generate "junk" characters. You need to look at the unconverted values in hexadecimal and try to work out what they are.
Sometimes it is simply mis-described data. For example a program generates UTF-8 data, but then sets the MQMD.CCSID to ASCII.
MQ (or the application) doesn't know this and tries to process some Unicode control characters or escape characters as ASCII and can't do it properly - hence the so-called junk characters.
Whereas if the message was correctly described as UFT-8 in the CCSID (and especially if MQ does the conversion using MQGMO_CONVERT) - all would be well.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Dec 26, 2014 4:20 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| These characters appear with no specific pattern and come anywhere in the file, sometimes multiple times. |
I agree with zpat. As a professional engineer, writing software for financial systems, it is your responsibility to understand the input format. All of it.
I accept that the input format sometimes does not have a written specification ( although that would be surprising in this case ). But it should still be possible to examine these 'junk characters' and make an educated guess about why they are present. Then you can design a message model ( DFDL ) and message flow logic that will accurately process the input data.
| Quote: |
| When these characters come in, the validation fails |
Please quote the exact error message issued by DFDL.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 26, 2014 7:33 am Post subject: Re: IIB9: DFDL, Random Junk Characters in incoming files |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| akil wrote: |
Hi
We have a message flow that receives flat files from a Credit Card system , we've modelled the file structures using DFDL, and all works fine, except on occasions where we find some non-printable characters in the file .
These characters appear with no specific pattern and come anywhere in the file, sometimes multiple times.
|
That sounds like files generated by an ATM system that contains an optional binary part that needs to be evaluated separately from the rest of the file/record.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
