| Author |
Message
|
| varunraot |
 Posted: Mon Apr 03, 2017 6:35 am Post subject: BFGSS0030W: MQ MFT error |
 |
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
Scenario: I have IIB V10 message flow which makes use of FTE output node to write the file to a directory in an another server ( Server A) with the help of agent defined( say Agent A) in that server A
Recently we came across a situation where file transfer failed( From IIB to Server) with below error
"BFGSS0030W: The agent is already acting as the source agent for the maximum number of file transfer operations and unable to queue further requests due to the queued transfer limit of 1000 being reached. The new transfer will not be carried out"
This made Agent A to go a recovery state "BFGTR0063I: Transfer ID: 414d5120514d4d464751523031202020cb0ed45823616863 entering recovery, immediately, due to recoverable error: BFGTR0059I: The transfer receiver timed out waiting for data from the source agent" in the Server A
I was able to reproduce this error by creating a bottleneck by releasing more than 1000 messages to the IIB flow.
I created an agent in the server where IIB flow has been deployed and made that agent to transfer the file to the server A with the more load( 10000 messages at one shot) and to my surprise, it did not result in above error.
Not sure what is the reason for this peculiar behavior? |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Mon Apr 03, 2017 4:01 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
The number of queued transfers will depend on how quickly they are requested, and how quickly the FTE/MFT agents are able to complete them. The limit is specified in the agent properties file.
There may also be "stuck" transfers that are unable to complete. These contribute to the queued count. Check the depth of the agents state queues.
The agent should be able to process at least 30 - 50 transfers / sec, if the files are small and the wind is blowing in the right direction.
_________________
Glenn |
|
| Back to top |
|
|
| varunraot |
| Posted: Mon Apr 03, 2017 8:21 pm Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
Here is the thing.
Scenario 1: When an attempt was made to write the file from IIB flow ( via embedded agent i.e when the message flow containing FTE output node gets deployed to an execution group) to a directory in an another server( say B), the error BFGSS0030W occurs with uncontrolled release of more than 1200 messages. Please note that the queue depth of STATE queues were sufficient. The maxQueuedTransfers property was set to 5000.
Scenario 2: I introduced an agent in the Server A( where IIB flows are deployed). IIB flow was made to transfer the file from FTE output node ( embedded agent) to the interim folder in the server A. MQ MFT monitor was created to transfer the file from the interim folder in Server A to the actual destination directory in Server B. The error BFGSS0030W did not occur in this case with uncontrolled release of 10,000 messages. Pleas note that the maxQueuedTransfers property was set to 5000 in both the agents ( A & B)
The question here is
Why was scenario 2 worked fine with more volume compared to the first scenario with less volume ? |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Apr 04, 2017 5:59 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
| Scenario 1: When an attempt was made to write the file from IIB flow ( via embedded agent i.e when the message flow containing FTE output node gets deployed to an execution group) to a directory in an another server( say B), the error BFGSS0030W occurs with uncontrolled release of more than 1200 messages. Please note that the queue depth of STATE queues were sufficient. The maxQueuedTransfers property was set to 5000. |
What is the text of the BFGSS0030W error? Was it "The agent is already acting as the source agent for the maximum number of file transfer operations and unable to queue further requests due to the queued transfer limit of 5000 being reached. The new transfer will not be carried out"
_________________
Glenn |
|
| Back to top |
|
|
| varunraot |
| Posted: Wed Apr 05, 2017 1:44 am Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
The error was "The agent is already acting as the source agent for the maximum number of file transfer operations and unable to queue further requests due to the queued transfer limit of 1000 being reached"
Please note that the threshold "1000" was being shown even though maxQueuedTransfers property was set to 5000 in the agent.properties of Server B |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Apr 05, 2017 5:22 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Did you restart the MFT agent after changing the properties file? The agent reads this at start up, and reports all the values it is using in its output log.
_________________
Glenn |
|
| Back to top |
|
|
| varunraot |
| Posted: Wed Apr 05, 2017 6:55 pm Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
| Yes. Agent has been restarted post the changes in agent.properties file for maxQueuedTransfer. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Apr 06, 2017 3:46 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| varunraot wrote: |
| Yes. Agent has been restarted post the changes in agent.properties file for maxQueuedTransfer. |
If you've properly changed this to 5000, the error message cannot possibly refer the old limit 1000. Does the output log show this set to 1000 or 5000 at startup?
_________________
Glenn |
|
| Back to top |
|
|
| varunraot |
| Posted: Fri Apr 07, 2017 6:25 am Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
| Output log show this set to 5000 at startup. I verified in the output.log post start up. |
|
| Back to top |
|
|
| varunraot |
| Posted: Tue May 08, 2018 7:52 am Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
This is one of the vulnerability of FTE output node which breaks with just 1000 files transfer limitation ( at one go) from IIB to any server !!!.
The word "reliability" is definitely not applicable for this product as we have spent considerable amount of time and money on this unreliable product.
The workaround for below problem is to introduce the maxQueuedTransfers value in agent.properties file of an embedded agent and the worst part is FTE output node never transfer files in FIFO order which is a implicit requirement for any file transfer software. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue May 08, 2018 8:02 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Moved to Managed File Transfer forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed May 09, 2018 4:24 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| varunraot wrote: |
| The word "reliability" is definitely not applicable for this product as we have spent considerable amount of time and money on this unreliable product. |
Hi varunraot,
A perception of unreliability may be due to issues with understanding proper usage, planning, design and configuration.
We have about 3,000 MFT agents doing 100,000's transfer per day of vast GB's across a diverse network, and it is very reliable.
I suggest engaging your IBM rep if you are having difficulties with this product, or post your specific concerns.
_________________
Glenn |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 10, 2018 5:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| varunraot wrote: |
| the worst part is FTE output node never transfer files in FIFO order which is a implicit requirement for any file transfer software. |
Define "FIFO" - order they are written to disc, order the file is completed on the disc, alphabetical order? What to do if the "oldest" file is larger than a newer file and the newer file is completed first? Wait for however long it takes to finish the older file?
It may be an implicit requirement (I would say assumption) of your system, and you might see what you consider FIFO with a large number of similarly sized small files with ftp or similar. But ftp doesn't guarantee the order in which files are transferred. Expecting files to be in a specific order is an affinity which is a poor design model.
For the record, I've limited experience with FTE but given how long the product's been around and the number of people that do use it, I think if it was as "unreliable" as you claim someone would have noticed by now. I agree with my associate that it's more likely to be the way you have FTE stood up & configured.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu May 10, 2018 4:32 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Vitor wrote: |
| varunraot wrote: |
| the worst part is FTE output node never transfer files in FIFO order which is a implicit requirement for any file transfer software. |
Define "FIFO" - order they are written to disc, order the file is completed on the disc, alphabetical order? What to do if the "oldest" file is larger than a newer file and the newer file is completed first? Wait for however long it takes to finish the older file? |
FTE / MFT does not guarantee FIFO order of file transfer requests. Individual transfers can independently take an indefinite period of time to complete (subject to specified timeout). MFT uses MQ as its data transport layer. MQ does not guarantee message delivery order.
Your app may have an "implicit requirement", but the designer should check that this requirement is actually met by the chosen solution.
You can put sequence numbers or date/time stamps in destination file names. The files are then marshalled by the destination application. If there are any missing files in the sequence, the app needs to wait for them to arrive, &/or alert a support team to investigate.
_________________
Glenn |
|
| Back to top |
|
|
| varunraot |
| Posted: Thu May 10, 2018 11:44 pm Post subject: |
|
|
Acolyte
Joined: 01 Jun 2011
Posts: 63
|
Few comments from my end
gbaddeley
We have about 3,000 MFT agents doing 100,000's transfer per day of vast GB's across a diverse network, and it is very reliable.
My comment: I am talking about "embedded agent" from IIB ( and not the standalone MQ MFT agent) which is not really robust when it comes to the transfer of high volume of data.
I had engaged IBM Product team for over 6 months in analyzing the behavior, pattern and root cause. This is what they concluded
"We expect that in flight transfers already in the queue will eventually complete. For transfers that experience this error (BFGSS0030W : The agent is already acting as the source agent for the maximum number of file transfer operations and unable to queue further requests,
due to the queued transfer limit of XXXX being reached. The new transfer request will not be carried out) they will need to be resubmitted (so the message flow needs to be re-run)."
In fact IBM proposed interim fix/patch to address this problem
"An enhancement to the MFT transfer process was undertaken to prevent messages from being added to the queue when the queue was already full.
This will mean that where previously if the transfer queue was full, transfers would be sent and lost track of,
they will now be rejected before they are sent, giving an exception that can be handled separately"
The above fix/patch was supposed to be in 10.0.0.10, however we did not pursue this as we had other preventive mechanism in place to address this issue.
My comment on FIFO.
Release a burst of 2000 odd files from IIB flow using FTE output node ( embedded agent) to a stand alone MQ MFT agent. All files are uniquely named with timestamp ( upto micro seconds precision). You would see that the files are not transferred in the order it is being created in the IIB embedded agent mqsitransit directory. All files were of very short size of 1 to 2 KB.
It was also admitted by IBM that "order" is not guaranteed in the case of file transfer from "IIB embedded agent" . I agree that default behavior is as pet the ascending order of the file name, however it is NOT applicable in the case of file transfer from IIB embedded agent
Hope I have clarified on this few things. |
|
| Back to top |
|
|
|
|
