Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added test case for online and offline embedding model, correspond to 天机-任务看板 No.7 #33

Merged
merged 8 commits into from
Mar 5, 2024

Conversation

tackhwa
Copy link
Contributor

@tackhwa tackhwa commented Mar 2, 2024

Assumption:

  • For online embedding model, all the api key had been set in the environment variable:

    • "BAIDU_API_KEY" correspond to api key for ernie embedding model
    • "OPENAI_BASE_URL" and "OPENAI_API_KEY" correspond to api setting for openai embedding model
    • "ZHIPUAI_API_KEY" correspond to api key for Zhipu embedding model
  • For offline embedding model, user had been required to login to huggingface or set the "HF_TOKEN" environment variable to access the gated jina ai repo

Output example:
image
image
image
image
image

Copy link
Contributor

@sanbuphy sanbuphy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

感谢贡献! 只需要在开头加上这个操作环境变量的加载即可;除此外,请记得在提交代码前运行下列程序:

pip install pre-commit 
pre-commit install 

pre-commit可以帮助规范python代码格式,更加优美同时做一些必要的检查,当你commit的时候他就会这么做


def load_model(self):
import torch
from transformers import AutoModel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import需要统一写在开头

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

涉及到transformers这种huggingface的下载,可以按照这个加入环境变量 (默认增加)
os.environ['HF_HOME'] = './cache/'
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
可以把这个配置到 .env 文件中

test/embedding/online/Ernie.py Show resolved Hide resolved
@sanbuphy
Copy link
Contributor

sanbuphy commented Mar 3, 2024

等待env修改后可合~

@tackhwa
Copy link
Contributor Author

tackhwa commented Mar 3, 2024

等待env修改后可合~

好了,pre commit我就不跑了,他把json里的prompt格式啥的都换了。

@sanbuphy
Copy link
Contributor

sanbuphy commented Mar 5, 2024

很棒的工作,非常感谢你的贡献!

@sanbuphy sanbuphy merged commit 03fcb00 into SocialAI-tianji:main Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants