Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote IO: S3 support #479

Draft
wants to merge 72 commits into
base: branch-24.12
Choose a base branch
from
Draft

Remote IO: S3 support #479

wants to merge 72 commits into from

Commits on Sep 30, 2024

  1. Impl. basic remote handle

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a01dc53 View commit details
    Browse the repository at this point in the history

  2. RemoteHandle::read()

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a6467d5 View commit details
    Browse the repository at this point in the history

  3. ci: rangehttpserver

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    127b782 View commit details
    Browse the repository at this point in the history

  4. libcurl>=7.75.0 # Need <https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.html 

    >
    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    089397f View commit details
    Browse the repository at this point in the history

  5. cmake

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    5f1fca0 View commit details
    Browse the repository at this point in the history

  6. fix ci rangehttpserver

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    eaacacf View commit details
    Browse the repository at this point in the history

  7. cleanup

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    59595a8 View commit details
    Browse the repository at this point in the history

  8. get_file_size(): check -1

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    f803220 View commit details
    Browse the repository at this point in the history

  9. Read(): check file size

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    dd06044 View commit details
    Browse the repository at this point in the history

  10. LibCurl get/set: use lock

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    06dcb35 View commit details
    Browse the repository at this point in the history

  11. test_no_range_support

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    7501600 View commit details
    Browse the repository at this point in the history

  12. bump libcurl>=7.87.0

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    7e6315e View commit details
    Browse the repository at this point in the history

  13. ci: more libcurl>=7.87.0

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4f9ab63 View commit details
    Browse the repository at this point in the history

  14. cmake: CURL 7.87.0

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a20774c View commit details
    Browse the repository at this point in the history

  15. ci_debug

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    43421ed View commit details
    Browse the repository at this point in the history

  16. CURLOPT_FOLLOWLOCATION

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    7df0e9f View commit details
    Browse the repository at this point in the history

  17. overflow_error

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    40b4be3 View commit details
    Browse the repository at this point in the history

  18. RemoteEndpoint

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    943e30b View commit details
    Browse the repository at this point in the history

  19. cpp_RemoteEndpoint

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    e8f6b12 View commit details
    Browse the repository at this point in the history

  20. from_http_url

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    5f73677 View commit details
    Browse the repository at this point in the history

  21. Update cpp/include/kvikio/remote_handle.hpp 

    Co-authored-by: Lawrence Mitchell <[email protected]>
    @madsbk @wence-
    madsbk and wence- committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    5dbe2e5 View commit details
    Browse the repository at this point in the history

  22. setopt(): use CURLoption

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    f9873c0 View commit details
    Browse the repository at this point in the history

  23. comment: need CURL_WRITEFUNC_ERROR

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4896452 View commit details
    Browse the repository at this point in the history

  24. LibCurl: use vector

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    e53b761 View commit details
    Browse the repository at this point in the history

  25. UniqueHandlePtr

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4ba4eb4 View commit details
    Browse the repository at this point in the history

  26. get_handle() / retain_handle()

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    bf716be View commit details
    Browse the repository at this point in the history

  27. clean up

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    83774e3 View commit details
    Browse the repository at this point in the history

  28. doc

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    e884d3c View commit details
    Browse the repository at this point in the history

  29. utils: LocalHttpServer

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    6637089 View commit details
    Browse the repository at this point in the history

  30. http benchmark

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a0098e9 View commit details
    Browse the repository at this point in the history

  31. test_http_io

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0e6dc7a View commit details
    Browse the repository at this point in the history

  32. example

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    f0b2f9f View commit details
    Browse the repository at this point in the history

  33. cleanup

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    8e7e2cf View commit details
    Browse the repository at this point in the history

  34. cmake: rapids_cpm_find CURL

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    fa67219 View commit details
    Browse the repository at this point in the history

  35. cleanup

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9906a14 View commit details
    Browse the repository at this point in the history

  36. doc

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    d38121b View commit details
    Browse the repository at this point in the history

  37. cleanup

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    bdc1e22 View commit details
    Browse the repository at this point in the history

  38. Merge branch 'branch-24.10' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    51057f6 View commit details
    Browse the repository at this point in the history

  39. clean up

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    383fa4c View commit details
    Browse the repository at this point in the history

  40. docs

    @madsbk
    madsbk committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0055373 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. doc

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    628e525 View commit details
    Browse the repository at this point in the history

  2. fix virtual dtor

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8726f87 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    3222c92 View commit details
    Browse the repository at this point in the history

  4. fix benckmark

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c94121d View commit details
    Browse the repository at this point in the history

  5. clean up

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0c764ba View commit details
    Browse the repository at this point in the history

  6. libcurl.hpp

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    91d953b View commit details
    Browse the repository at this point in the history

  7. rename: RemoteFile.open_http

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    94e47ee View commit details
    Browse the repository at this point in the history

  8. doc

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    fcb9358 View commit details
    Browse the repository at this point in the history

  9. cleanup

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e8018d1 View commit details
    Browse the repository at this point in the history

  10. PushAndPopContext outside callback_device_memory()

    @madsbk
    madsbk committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    bb737b9 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    f3f46f8 View commit details
    Browse the repository at this point in the history

  2. more nvtx

    @madsbk
    madsbk committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    b8a9c0a View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Bradley Dice <[email protected]>
Co-authored-by: Lawrence Mitchell <[email protected]>
    @madsbk @bdice @wence-
    3 people authored Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9cffe1c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    792f43f View commit details
    Browse the repository at this point in the history

  3. std::function<decltype(curl_easy_cleanup)>

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    1257a9f View commit details
    Browse the repository at this point in the history

  4. getinfo: take output pointer

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    1006b24 View commit details
    Browse the repository at this point in the history

  5. STATUS

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9ba9752 View commit details
    Browse the repository at this point in the history

  6. close

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    128e3dd View commit details
    Browse the repository at this point in the history

  7. test_http_io: check is_remote_file_available

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    0df1726 View commit details
    Browse the repository at this point in the history

  8. std::ptrdiff_t

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    96af5ac View commit details
    Browse the repository at this point in the history

  9. Apply suggestions from code review 

    Co-authored-by: Bradley Dice <[email protected]>
    @madsbk @bdice
    madsbk and bdice authored Oct 3, 2024
    Configuration menu
    Copy the full SHA
    feccdd9 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'remote-io' of github.com:madsbk/kvikio into remote-io

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    119c025 View commit details
    Browse the repository at this point in the history

  11. use conda's rangehttpserver

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    7dcc8f3 View commit details
    Browse the repository at this point in the history

  12. doc

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    60f7331 View commit details
    Browse the repository at this point in the history

  13. CallbackContext ctor

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    13e0b7a View commit details
    Browse the repository at this point in the history

  14. host: libcurl==7.87.0

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    474e694 View commit details
    Browse the repository at this point in the history

  15. run: remove libcurl>=7.87.0

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9022b18 View commit details
    Browse the repository at this point in the history

  16. curl-7_87_0

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    8bbb33b View commit details
    Browse the repository at this point in the history

  17. Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    46b9300 View commit details
    Browse the repository at this point in the history

  18. kvikio/meta.yaml: libcurl==7.87.0

    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    1176b67 View commit details
    Browse the repository at this point in the history

  19. Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int… 

    …o remote-io
    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    4ed88a2 View commit details
    Browse the repository at this point in the history

  20. all of the remote-io-s3 branch 

    this is the only commit to cherry-pick
    @madsbk
    madsbk committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9287599 View commit details
    Browse the repository at this point in the history