-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added code to allow users to specify inference parameters for Huggingface models using HuggingfaceInferenceAPI #15680
base: main
Are you sure you want to change the base?
Conversation
"The value of the stop parameter to use when generating text. This parameter" | ||
" is used to stop generating text when one of the specified tokens is generated." | ||
), | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
None of these parameters are used though? Seems like there needs to be some function to get model kwargs when using the client?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes! Sorry for the oversight. Correct me if I am wrong - link - here we can add these extra arguments. I am new to open-source contributions, I would really appreciate some guidance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, not in the metadata, but I think these options need to be passed when using the clients, like here
Line 228 in 1f62f0e
output: "ConversationalOutput" = self._sync_client.conversational( |
Probably we want a helper function to gather up the kwargs, and then each time a client is used, that function can help gather all the kwargs and send them along
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Working on it! Thanks !
Description
This PR is related to issue #15659 - I added code to allow users to specify inference parameters for Huggingface models using HuggingfaceInferenceAPI . Refer to link to see how this code integrates with huggingface.
Fixes #15659
New Package?
Did I fill in the
tool.llamahub
section in thepyproject.toml
and provide a detailed README.md for my new integration or package?Version Bump?
Did I bump the version in the
pyproject.toml
file of the package I am updating? (Except for thellama-index-core
package)Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Suggested Checklist: