Help with FME Server, HTTP Caller, and CKAN
Since updating to FME Server 2019 a number of workspaces reading or writing via the web seem to be problematic.
In this particular instance, I have a workspace that writes to CKAN at data.gov.au. Desktop works fine, but since updating our server it no longer works.
My workspace is modeled on this one (with a few minor changes to reflect some updates to the api) :
https://github.com/datagovau/ckan-api-examples/blob/master/FME/CKANUpdate.fmw
The first issue is that my HTTP Caller in desktop has the response body encoding set to 'auto-detect'. I found that Server fails with this, and after trial and error found it works when set to unicode 8-bit.
However the more serious problem is that I can't get it to transfer my file. The error looks like this:
| 63 | HTTPCaller_2__3 (HTTPFactory): HTTP/FTP transfer error: 'Failed sending data to the peer' |
| 64 | HTTPCaller_2__3 (HTTPFactory): Please ensure that your network connection is properly set up |
| 65 | HTTPCaller_2__3 (HTTPFactory): Please ensure the following proxy information is correct. The current proxy is: 'myProxy:8080' |
It would appear that the proxy is correct as the workspace is able to download the json file only two lines earlier:
HTTPCaller (HTTPFactory): HTTP transfer summary - status code: 200, download size: '8009 bytes', DNS lookup time: '0.00102 seconds', total transfer time: '0.180459 seconds' |
If anyone has any suggestions, they would be greatly appreciated....
Thanks
Andrew
Early days, but I think I may have found a solution. It looks like a bug with libcurl 7.65.2.0 which is installed with Server. Our Desktop is using libcurl 7.61.1.0. I replaced the Server version with the Desktop version and all appears to be well. This might be something Safe might want to have a look into maybe....
Hi @andrewspencer, this issue was recently brought to my attention – have there been any problems since replacing libcurl in FME Server?
And can you please let us know the build numbers of FME Server and FME Desktop that you are using? (build numbers are represented with five digits – for example, 19642)
Looking through the commit logs, we upgraded to 7.65.2 for approximately Build 19617 and Newer, (and this should have been) across all platforms and Desktop/Server.
The workspace you linked (GitHub) appears to be authored with FME 2015 (Build 15244). Did you use that workspace as a starting template? If so, I am wondering if all of the transformers were upgraded to the latest versions in 2019? (the HTTPCaller in particular has an upgrade available..)
If you can share the offending workspace with us that might help us narrow down the issue. You can submit to our Support Team if it contains sensitive information.
I'm hopeful that we can evaluate if this library upgrade will impact any other workflows. I'm sorry for the inconvenience it has caused you! (Great troubleshooting to find and replace that component, by the way!)
Hi @rylanatsafe, thanks for your reply. I've attached a workspace (we have 6 of these running on a daily schedule). Our API key and source file location have been removed, but the end result of this is here: https://data.gov.au/data/organization/nationalnativetitletribunal. The workspace on GitHub is a little old, and we need to do a little more work to flatten the JSON than we did in 2015. All transformers in our workspaces have been have been updated.
Our Desktop build is 19608, our server is 19617.
Since replacing libcurl I've had no issues, and it also resolved the response unicode/autodetect problem.
On the surface, it seemed that Server was trying to switch the traffic to http/2 before failing, whilst Desktop was happy with http/1.1. Thanks again for your interest, and let me know if you need any more info.
One tip I have is to turn on FME Debug logging, it's pretty helpful I've found with the HTTPCaller - Especially when a proxy is in use.
It's possible that a bug/issue has been fixed meaning that now data is trying to go through the configured proxy whereas before it didn't. It is strange that the issues are only with FME Server. Could it be a firewall or CORS issue?
