| Author |
Message
|
| j1 |
 Posted: Wed Dec 02, 2015 3:50 pm Post subject: Reading Records from a file using FileRead Node |
 |
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
Hi,
I have a requirement to read records from a file as they are being appended on an ongoing basis. It looks like this kind of use case may be possible using the FileRead Node but the documentation is not clear how the FileRead node could be made to do this periodically to read off new records appended to a file. Would using a Timer node with a FileRead node be the answer?
Thanks in advance |
|
| Back to top |
|
 |
| zpat |
| Posted: Wed Dec 02, 2015 11:10 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
That is not a requirement. That is a possible (and weak) design.
Ask them to use MQ messages if possible.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Mon Apr 25, 2016 12:10 am; edited 1 time in total |
|
| Back to top |
|
|
| timber |
| Posted: Thu Dec 03, 2015 2:21 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| I have a requirement to read records from a file as they are being appended on an ongoing basis. |
What happens if the FileRead node tries to read the file before the 'append' has completed?
| Quote: |
| Would using a Timer node with a FileRead node be the answer? |
Only if there is no other design that can possibly work, and my question above has been explored carefully.
Why does it have to be done this way? Can the application that appends to this file write to a new file instead? |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 03, 2015 3:30 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Files are transactionally unsafe.
Even if the developers have not heard of MQ, have they heard of JMS?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Mon Apr 25, 2016 12:09 am; edited 1 time in total |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Dec 03, 2015 5:26 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
Can the application stop living in the 1970's and use messaging instead of files for transactions?
|
If the system is in South Asia then it seems that 'Files Rulez Ok'
We had to implement a system where it seemed that every interface was FTP. One of the connections was even going to be picked up by an application and sent on by MQ!
My thoughts at the time were:
Find a brick wall and bassh you head until you can see that it makes perfect sense.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| timber |
| Posted: Thu Dec 03, 2015 7:29 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
For whatever reasons, file-based integration is alive and well in most parts of the world, including the USA and the UK. I agree that MQ is superior, but getting an entire organisation to move away from file-based integration is a difficult sell.
The second-best isolutiuon s to get people to realise that FTP is best done using a managed file transfer tool like MQFT...which also provides a migration path towards MQ messaging in the medium term. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 03, 2015 7:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
No-one is suggesting removing all files, but each time some complex solution involving files is proposed - it needs to be challenged, otherwise it could get out of control with layer on layer of workarounds that have transactional integrity gaps
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Mon Apr 25, 2016 12:08 am; edited 1 time in total |
|
| Back to top |
|
|
| j1 |
| Posted: Fri Dec 04, 2015 3:17 am Post subject: |
|
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
The problem here is that the messages are in MQ to begin with, as PCF formatted messages. Not being much of a C programmer or familiar with PCF, I could only hack some sample code to write the PCF messages to a file..and that's where the problem lies,... to get them back onto a queue as XML messages (not PCF). Was hoping IIB could come to the rescue here  |
|
| Back to top |
|
|
| j1 |
| Posted: Fri Dec 04, 2015 3:21 am Post subject: |
|
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
| besides, as timber points out, this would not be a too uncommon use case - having to read messages off a file that's being written to and put them to a queue. I was thinking of using something like 'Q' tool or qload but all of those have a problem maintaining an offset (to read only newly appended messages) |
|
| Back to top |
|
|
| timber |
| Posted: Fri Dec 04, 2015 5:01 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| Not being much of a C programmer or familiar with PCF, I could only hack some sample code to write the PCF messages to a file |
That doesn't make a lot of sense. Why can you not write the messages to an MQ queue?
| Quote: |
| besides, as timber points out, this would not be a too uncommon use case - having to read messages off a file that's being written to and put them to a queue. |
I actually said 'file-based integration is alive and well in most parts of the world'. I don't see how you can interpret that to mean what you said. Please don't misquote me in order to support your (decidedly shaky) position. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Dec 04, 2015 5:22 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| j1 wrote: |
| The problem here is that the messages are in MQ to begin with, as PCF formatted messages. Not being much of a C programmer or familiar with PCF, I could only hack some sample code to write the PCF messages to a file..and that's where the problem lies,... to get them back onto a queue as XML messages (not PCF). Was hoping IIB could come to the rescue here |
Why can't you have IIB read the messages off the initial queue and dump your hacked together C code altogether?
Then your flow would be dealing with MQ messages in PCF format and simply turning them into XML messages on the new queue. No messy file interactions needed. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 04, 2015 5:26 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| j1 wrote: |
| The problem here is that the messages are in MQ to begin with, as PCF formatted messages. Not being much of a C programmer or familiar with PCF, I could only hack some sample code to write the PCF messages to a file..and that's where the problem lies,... to get them back onto a queue as XML messages (not PCF). Was hoping IIB could come to the rescue here |
First of all you show a very poor understanding of the problem.
If your spec is to translate the PCF message (presumably a response to a PCF enquiry) into an XML message than what does a file have to do in the middle???
 pcf message and understand what your spec is calling for... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 04, 2015 5:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| j1 wrote: |
| The problem here is that the messages are in MQ to begin with, as PCF formatted messages. Not being much of a C programmer or familiar with PCF, I could only hack some sample code to write the PCF messages to a file..and that's where the problem lies,... to get them back onto a queue as XML messages (not PCF). Was hoping IIB could come to the rescue here |
This is insane.
If you want IIB to come to the rescue, get IIB to read the PCF messages from the original queue and write XML to the target; if you want to hack some code (your words not mine), leave IIB out of it and keep hacking some more code to write to a queue. It's not hard - it's the code you've already hacked but running the other way.
Do not do both. This just makes it worse.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 04, 2015 6:17 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So the messages start as PCF messages on a queue.
You then write them to a file, and want to use IIB to read them from a file and then turn them into XML?
No. Don't do that.
Read the messages directly using an MQInput node. Use the MQ Java classes in a JCN to parse and work with the PCF messages. Use the IIB Java classes to create an XMLNSC output tree.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Dec 04, 2015 8:28 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There is even a post in this forum that described how to create a PCF Message using ESQL and process the response.
Take a look at it and see if that will help you eliminate files from your solution entirely.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
