Remote IO: http support

[madsbk]

Support read directly from a http server like:

import kvikio
import cupy

with kvikio.RemoteFile.from_http_url("http://127.0.0.1:9000/myfile") as f:
    ary = cupy.empty(f.nbytes, dtype="uint8")
    f.read(ary)

This PR is the first step to support S3 using libcurl instead of aws-s3-sdk, which has some pros and cons:

• Pros
  • The global conda pinning issue is less of a problem.
  • We can support other protocols such as http, ftp, and Azure’s storage, without much work.
  • We avoid the free-after-main issue in aws-s3-sdk. This is huge since we would otherwise have to pass around a S3Context in libcudf and cudf to handle shutdown correctly. This is not a problem in libcurl, see https://curl.se/libcurl/c/libcurl.html under Global constants.
• Cons
  • Hard to support the AWS configuration file. We will require the user to either specify the options programmatically or through environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY .

[wence-]

C++, not python yet

[wence-]

Minor comments, overall looking good to me

[wence-]

note: Nothing can be done, because it's in the curl API, but if nbytes == CURL_WRITEFUNC_ERROR and ctx->size < ctx->offset + nbytes then curl won't notice that we returned an error here, I think.

[madsbk]

nbytes == CURL_WRITEFUNC_ERROR cannot happen, CURL_WRITEFUNC_ERROR is defined as 0xFFFFFFFF, which is greater than CURL_MAX_WRITE_SIZE.

[wence-]

Suggested change

	cdef extern from "<kvikio/remote_handle.hpp>" nogil:
	cdef cppclass cpp_RemoteEndpoint "kvikio::RemoteEndpoint":
	pass
	cdef cppclass cpp_HttpEndpoint "kvikio::HttpEndpoint":
	cpp_HttpEndpoint(string url) except +
	cdef cppclass cpp_RemoteHandle "kvikio::RemoteHandle":
	cdef extern from "<kvikio/remote_handle.hpp>" nogil namespace "kvikio":
	cdef cppclass RemoteEndpoint:
	pass
	cdef cppclass HttpEndpoint:
	HttpEndpoint(string url) except +
	cdef cppclass RemoteHandle:

?
Since RemoteEndPoint/HttpEndpoint and RemoteHandle are not used by any python cdef class names.

[madsbk]

I keep mixing C++ and Python objects when reading Cython, I think always using cpp_ makes it a bit easier?

[wence-]

question: Since we're starting a new process here to run the server, why do we also run the server in its own thread in _server?

[madsbk]

To handle the max_lifetime

[bdice]

Use an exact pinning in host: (next to cuda-version on line 77). Then it will use a run-export to get the compatible run pinning here.

[madsbk]

done

[bdice]

Use an exact pinning here -- building against a specific (possibly older) version will allow users to have anything equal or newer at runtime. See other comment below about run pinnings.

The problem you want to avoid is that this host pinning could pull libcurl 7.999 at build time (anything >=7.87), but your run pinning below is >=7.87.0 which might be incompatible. Pinning a specific version at build time and then relying on run-exports for a compatible runtime pinning is the cleanest solution here, and it's how we handle most dependencies like this.

[madsbk]

Thanks for the clarification, I understand meta.yaml a little bit better now :)