| Author |
Message
|
| firoj.badsa |
 Posted: Fri May 06, 2011 1:43 am Post subject: FileInput Node: Special Character |
 |
|
Centurion
Joined: 16 Feb 2007
Posts: 104
|
Hi,
I have flow like FileInput -> Compute -> HTTPRequest.
FileInput node doing a FTP to a remote server and trying to send a the message over HTTPRequest. But Webservice was failing for some reason. When I did a debug, I saw FileInput node adding some special character at the begining of the first line(). And the error I am getting like 'Invalid byte 1 of 1-byte UTF-8 sequence'. can somebody help here.
thanks in advance,
Firoj |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri May 06, 2011 1:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Have you told FileInput to perform the ftp in ascii mode or in binary mode?
Have you told FileInput to process the file in the correct CCSID? |
|
| Back to top |
|
|
| artykito |
| Posted: Fri May 06, 2011 4:19 am Post subject: |
|
|
Apprentice
Joined: 24 Jun 2010
Posts: 33
|
Check that jeff told you.
Anyway, did you check the input file if is it correct? I had a problem with a file sometime ago with a special char that I saw like a space in the input file and like other weird char when the FileInput node parsed the message, but it was the "Abort Output" <AO> character. I detected it with Notepad++ converting the message to other format...maybe your problem is similar.
I have one question for whoever, I don't have too much experience with web services, what's the difference between use SOAPRequest and HTTPRequest nodes? thanks! |
|
| Back to top |
|
|
| firoj.badsa |
| Posted: Fri May 06, 2011 9:09 pm Post subject: |
|
|
Centurion
Joined: 16 Feb 2007
Posts: 104
|
Thanks for your reply Jeff. I have created a new file(.TXT) and copied the content of the previous file and it looks fine now. So, the issue could be there were some issue while the source application creating the file. Actually the original file is a .TXT file and there are renaming it to .CSV before asking me to process the data. Maybe issue is coming while the renaming happens.
thanks,
Firoj |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat May 07, 2011 3:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| firoj.badsa wrote: |
| Thanks for your reply Jeff. I have created a new file(.TXT) and copied the content of the previous file and it looks fine now. So, the issue could be there were some issue while the source application creating the file. Actually the original file is a .TXT file and there are renaming it to .CSV before asking me to process the data. Maybe issue is coming while the renaming happens. |
The renaming almost certainly has nothing to do with the problem.
The problem as stated is that you are receiving a file with a Byte Order Mark on it, and are using a code page that does not expect a Byte Order Mark.
One reason for this could be that you have transfered the file as FTP ASCII, which has caused a code page conversion.
Another reason for this could be that you have specified a Unicode CCSID that does not require a BOM and they have created the file with a CCSID that does. |
|
| Back to top |
|
|
| firoj.badsa |
| Posted: Sat May 07, 2011 5:54 am Post subject: |
|
|
Centurion
Joined: 16 Feb 2007
Posts: 104
|
Hi Jeff,
Actually the transfer method I was using is Binary then I got the special character . But just for a change I changed it to ASCII then I got '???' at the beginning!.
And Input Message Parsing Tab I have selected
CCSID: Broker System Default
Message Encoding: Broker System Determined.
thanks,
Firoj |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat May 07, 2011 8:15 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| firoj.badsa wrote: |
Hi Jeff,
Actually the transfer method I was using is Binary then I got the special character . But just for a change I changed it to ASCII then I got '???' at the beginning!.
And Input Message Parsing Tab I have selected
CCSID: Broker System Default
Message Encoding: Broker System Determined.
thanks,
Firoj |
| mqjeff wrote: |
| The problem as stated is that you are receiving a file with a Byte Order Mark on it, and are using a code page that does not expect a Byte Order Mark. |
Read again very carefully what mqjeff said. If you do not understand any of those terms, (like byte order mark) google them and then ask here for more clarification.
Your last attempt to fix this is showing more guesswork than actual reflection. You have a CCSID problem there. You need to deal with it.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| firoj.badsa |
| Posted: Sat May 07, 2011 9:17 am Post subject: |
|
|
Centurion
Joined: 16 Feb 2007
Posts: 104
|
| I did not understood what guesswork I did.. I just explained what I did .. |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat May 07, 2011 12:42 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
mqjeff is 100% correct. Your input message has a BOM.
| Quote: |
| I did not understood what guesswork I did.. I just explained what I did .. |
There is a standard process for solving problems like this:
1 Try to understand the problem
2 Design a fix for the problem, based on the output of step 1.
Vitor ( edit: fjb_saper actually) thinks that you did this
1. Ask a question on this forum
2. Ignore the ( correct ) answer, and try some random change to see whether it works.
I think I agree with fjb_saper. And I agree with his suggestion:
| Quote: |
| Read again very carefully what mqjeff said. If you do not understand any of those terms, (like byte order mark) google them and then ask here for more clarification. |
Last edited by kimbert on Sat May 07, 2011 2:12 pm; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat May 07, 2011 12:52 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kimbert wrote: |
| I think I agree with Vitor. |
Presumably in some other thread, however. |
|
| Back to top |
|
|
| firoj.badsa |
| Posted: Sat May 07, 2011 8:19 pm Post subject: |
|
|
Centurion
Joined: 16 Feb 2007
Posts: 104
|
Ok.. ok. let me clarify .. all I was saying is I have tried all the things only on friday .. and surely I have to look for the BOM on monday . I can not afford to ignore any suggestion when I need a solution  . |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun May 08, 2011 6:16 am Post subject: Re: FileInput Node: Special Character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| firoj.badsa wrote: |
And the error I am getting like 'Invalid byte 1 of 1-byte UTF-8 sequence'. ...
And Input Message Parsing Tab I have selected
CCSID: Broker System Default |
The receiver needs to use the same ccsid to read the file that the sender used to write the file. Confirm that the sender wrote the file correctly, and that the receiver read the file correctly.
While some applications will use a BOM for UTF-8, WMB does not expect one, so the sender or receiver may need to remove it.
| firoj.badsa wrote: |
| I have created a new file(.TXT) and copied the content of the previous file and it looks fine now. |
Copying the file introduces a new receiver, sender, and (possibly converted) file between the original sender and receiver. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun May 08, 2011 8:55 am Post subject: Re: FileInput Node: Special Character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| rekarm01 wrote: |
| firoj.badsa wrote: |
And the error I am getting like 'Invalid byte 1 of 1-byte UTF-8 sequence'. ...
And Input Message Parsing Tab I have selected
CCSID: Broker System Default |
The receiver needs to use the same ccsid to read the file that the sender used to write the file. |
Which means that the FileInput node needs to use the correct CodedCharSetID. Which may not be Broker System Default. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun May 08, 2011 11:42 am Post subject: Re: FileInput Node: Special Character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
Although ...- if the message contains a UTF-8 BOM, this suggests that the sender intended to write UTF-8
- if the error complains about a malformed UTF-8 sequence, this suggests that the receiver attempted to read UTF-8
So, if the File Input node ccsid is correct, check the other relevant properties on the Input Message Parsing tab (message domain, set, type, format, encoding). If there's still a problem, it's likely the bytes are corrupt; the next step is to check them at each point between sender and receiver, to narrow down where the corruption(s) occurred. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun May 08, 2011 4:58 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| The receiver could entirely have meant UTF-16 BOM and the receiver attempted UTF-16 NO BOM. |
|
| Back to top |
|
|
|
|
