e5-mistral evaluation via vLLM

[robertgshaw2-neuralmagic]

• Launch server:

vllm serve intfloat/e5-mistral-7b-instruct

• Run eval script:

python3 run-e5.py

[Samoed]

You should move imports of external library inside the class to make it optional

[Samoed]

When this file will be imported it will try to evaluate model. I think this should be removed

[KennethEnevoldsen]

Very happy to see this PR! We already have an implementation of e5 mistral. However, I assume this is notably faster, but it would be nice we an actual comparison of both scores and speed.

We also implement models in mteb/models. I believe we could do this here as well.

I assume it produces different scores (given vLLM) so it might be worth changing the name (related to #1211), but we can take that after a test.

[robertgshaw2-neuralmagic]

Very happy to see this PR! We already have an implementation of e5 mistral. However, I assume this is notably faster, but it would be nice we an actual comparison of both scores and speed.

We also implement models in mteb/models. I believe we could do this here as well.

I assume it produces different scores (given vLLM) so it might be worth changing the name (related to #1211), but we can take that after a test.

Hey @KennethEnevoldsen - apologies, I did not meant to submit this PR onto upstream it was just a simple POC for myself to eval the embedding models running in vLLM. I managed to match the scores running through GritLM on a few STS tasks. I made this to automate correctness testing in vLLM

If this is something you would like to see in MTEB, I will clean it up

[KennethEnevoldsen]

If this is something you would like to see in MTEB, I will clean it up

If you can get it to match the performance at a faster speed I would love to have it added (cc @Muennighoff; it might be worth integrating this in GritLM)