Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify interactions between TorchServe and KServe #3378

Open
kimminw00 opened this issue Dec 26, 2024 · 0 comments
Open

Clarify interactions between TorchServe and KServe #3378

kimminw00 opened this issue Dec 26, 2024 · 0 comments

Comments

@kimminw00
Copy link

kimminw00 commented Dec 26, 2024

📚 The doc issue

When deploying PyTorch models using the pytorch/torchserve-kfs image with Kserve, I found it challenging to understand the architecture and how different processes interact with each other. Specifically, I would like to know which processes run in which pods and how resources are allocated. To optimize for large traffic volumes, it's crucial to understand how resources are allocated to each process.

As I understand, TorchServe uses Netty-based HTTP/gRPC servers, while Kserve uses Tornado-based HTTP/gRPC servers. However, when deploying with pytorch/torchserve-kfs image, it's unclear what process runs where.

Reference
https://kserve.github.io/website/master/modelserving/v1beta1/torchserve/

Suggest a potential alternative/fix

If possible, providing a high-level diagram or explanation of how the different components interact would be incredibly helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant