community: Outlines integration

[shroominic]

In collaboration with @rlouf I build an outlines integration for langchain!

I think this is really useful for doing any type of structured output locally.
Dottxt spend alot of work optimising this process at a lower level (outlines-core written in rust) so I think this is a better alternative over all current approaches in langchain to do structured output.
It also implements the .with_structured_output method so it should be a drop in replacement for a lot of applications.

The integration includes:

• Outlines LLM class
• ChatOutlines class
• Tutorial Cookbooks
• Documentation Page
• Validation and error messages
• Exposes Outlines Structured output features
• Support for multiple backends
• Integration and Unit Tests

Dependencies: outlines + additional (depending on backend used)

I am not sure if the unit-tests comply with all requirements, if not I suggest to just remove them since I don't see a useful way to do it differently.

Quick overview:

Chat Models:

Structured Output:

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 20, 2024 9:14am

[cpfiffer]

Either I am stupid or the Integration docs lint / build (pull_request) check is broken... It showes ValueError: Document /home/runner/work/langchain/langchain/docs/docs/integrations/chat/outlines.ipynb is missing headers:

• Installation

But the header is definitlely there!

I'm surprised by this too -- the header is included for sure.

[cpfiffer]

Ah, I see. I believe the order of credentials and installation should be switched, as per this doc.

[shroominic]

Ah, I see. I believe the order of credentials and installation should be switched, as per this doc.

I tried this and run the script locally but still didn't work ...

[cpfiffer]

Could y'all wait a bit before merging? I want some Outlines people from @dottxt-ai to take a look at this to see how we feel about the interface.

[rlouf]

Looks good to me 🫡

[vbarda]

Thanks for the contribution -- looks good overall! Added a few comments

[vbarda]

Could you add some unittests as well?

[shroominic]

Could you add some unittests as well?

I added unit-tests before but removed them because I could not find a useful way to do them without requiring network connection since outlines requires local models and you initialize the chat model with a hugging face model id.
https://github.com/langchain-ai/langchain/blob/30071e3182459caa975c16a84b3e22ab7e2a94d3/libs/community/tests/unit_tests/llms/test_outlines.py

[vbarda]

Could you add some unittests as well?

I added unit-tests before but removed them because I could not find a useful way to do them without requiring network connection since outlines requires local models and you initialize the chat model with a hugging face model id. https://github.com/langchain-ai/langchain/blob/30071e3182459caa975c16a84b3e22ab7e2a94d3/libs/community/tests/unit_tests/llms/test_outlines.py

yea, you would probably need to do some mocking for the model interfaces to avoid network connections or using local models. primarily it would be nice to test different scenarios for constraints validation etc. you could also factor some of those out outside the classes and that would also make it easier to test

[shroominic]

Could you add some unittests as well?

I added unit-tests before but removed them because I could not find a useful way to do them without requiring network connection since outlines requires local models and you initialize the chat model with a hugging face model id. https://github.com/langchain-ai/langchain/blob/30071e3182459caa975c16a84b3e22ab7e2a94d3/libs/community/tests/unit_tests/llms/test_outlines.py

yea, you would probably need to do some mocking for the model interfaces to avoid network connections or using local models. primarily it would be nice to test different scenarios for constraints validation etc. you could also factor some of those out outside the classes and that would also make it easier to test

✅

[vbarda]

@shroominic thanks for adding the tests! i think it's almost ready to merge -- docs page for LLMs is failing (the chat model one is correct)

ValueError: Document /home/runner/work/langchain/langchain/docs/docs/integrations/llms/outlines.ipynb is missing headers

[shroominic]

everything passing 🙌