The other side of the moon

The problem

A colleague was recently tasked with using SFTP to upload an automated report to a customer's server. The code that generates the data runs in Julia 1.6, but there were no restrictions on where the upload code had to run.

Unfortunately our command line curl wasn't build with sftp support so we couldn't use that, and our system doesn't have sftp installed, so we couldn't use that either. The question was, whether we could use Julia's built-in LibCURL library to do the upload.

tl; dr

Yes, you can, by converting a Julia IOStream to a C pointer to the file that needs uploading.

The details

Julia uses LibCURL which is a simple wrapper around the libcurl C API. This generally means that we have to pass in C pointers for a lot of things. Now Julia can automatically do type conversions for basic types like Strings and Numbers, but more complex structs will need some work to make sure they're in the right format.

The basic libcurl code we need to reproduce is this:

curl = curl_easy_init()

curl_easy_setopt(curl, CURLOPT_URL, "sftp://user:password@server:port/path/to/file.csv") # This would be a NULL terminated string in C, but Julia does that conversion for us
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1)
curl_easy_setopt(curl, CURLOPT_PROTOCOLS, CURLPROTO_SFTP)

curl_easy_setopt(curl, CURLOPT_READDATA, file_ptr)   # We need to pass in a C file pointer here

res = curl_easy_perform(curl)
curl_easy_cleanup(curl)

The complication is that Julia uses libuv to open files, and the return value from Julia's open function is an IOStream. Fortunately, Julia has code in Libc that converts between an IO and a FILE *:

struct FILE
    ptr::Ptr{Cvoid}
end

modestr(s::IO) = modestr(isreadable(s), iswritable(s))
modestr(r::Bool, w::Bool) = r ? (w ? "r+" : "r") : (w ? "w" : throw(ArgumentError("neither readable nor writable")))

function FILE(fd::RawFD, mode)
    FILEp = ccall((@static Sys.iswindows() ? :_fdopen : :fdopen), Ptr{Cvoid}, (Cint, Cstring), fd, mode)
    systemerror("fdopen", FILEp == C_NULL)
    FILE(FILEp)
end

function FILE(s::IO)
    f = FILE(dup(RawFD(fd(s))),modestr(s))
    seek(f, position(s))
    f
end

Base.unsafe_convert(T::Union{Type{Ptr{Cvoid}},Type{Ptr{FILE}}}, f::FILE) = convert(T, f.ptr)

Using this, we can open a file in Julia, and pass it on to curl:

fh = open("file.csv", "r")   # Open the file for reading and get an IOStream
fp = Libc.FILE(fh)           # Convert the IOStream to a FILE*
curl_easy_setopt(curl, CURLOPT_READDATA, fp.ptr)

Note that in the call to CURLOPT_READDATA, we need to pass in the ptr member of the FILE struct, since that's the actual C object.

Complete example

curl = curl_easy_init()

curl_easy_setopt(curl, CURLOPT_URL, "sftp://user:password@server:port/path/to/file.csv")
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1)
curl_easy_setopt(curl, CURLOPT_PROTOCOLS, CURLPROTO_SFTP)

fh = open("file.csv", "r")
fp = Libc.FILE(fh)
curl_easy_setopt(curl, CURLOPT_READDATA, fp.ptr)

res = curl_easy_perform(curl)
curl_easy_cleanup(curl)