[Tokenizer] Support reading Tiktoken tokenizer.model. #9215
Conversation
…d from pretrained, update method to get attr from a module
|
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9215 +/- ##
===========================================
+ Coverage 52.99% 53.04% +0.04%
===========================================
Files 671 667 -4
Lines 109835 106886 -2949
===========================================
- Hits 58212 56699 -1513
+ Misses 51623 50187 -1436 ☔ View full report in Codecov by Sentry. |
| @@ -176,7 +324,7 @@ def _get_tokenizer_class_from_config(cls, pretrained_model_name_or_path, config_ | |||
| return tokenizer_class | |||
|
|
|||
| @classmethod | |||
| def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs): | |||
| def from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs): | |||
There was a problem hiding this comment.
这个参数修改名称需要注意，需判断其他from_pretrained方法参数是否使用相同名称
There was a problem hiding this comment.
其他Tokenizer都没有override from_pretrained方法，所以应该不会有影响
There was a problem hiding this comment.
这里的问题是使用auto.from_pretrained()和Qwen2XXX.form_pretrained()的代码写法可能会发生变化，建议统一
paddlenlp/utils/download/__init__.py
Outdated
| @@ -272,7 +272,7 @@ def resolve_file_path( | |||
| f"'{log_endpoint}' for available revisions." | |||
| ) | |||
| except EntryNotFoundError: | |||
| return None | |||
| raise EnvironmentError(f"Does not appear one of the {filenames} in {repo_id}.") | |||
There was a problem hiding this comment.
这个Error类型是不是应该是EntryNotFoundError？
There was a problem hiding this comment.
如果要raise EntryNotFoundError，那前面就不需要用except捕获EntryNotFoundError了，之前这么做应该有这么做的道理（吧）。
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software |
There was a problem hiding this comment.
目前AutoTokenizer加载Tokenizer的方式是根据模型目录的名称进行匹配，举个例子，之前albert，albert_chinese，albert_english都在albert目录下，但是根据名称进行匹配(TOKENIZER_MAPPING_NAMES表)只允许有一个Tokenizer和一个TokenizerFast，如果不分开会导致albert_chinese和albert_english无法通过AutoTokenizer加载，因为他们三个加载时需要用不同的Tokenizer类。
PR types
New features
PR changes
APIs
Description
Support reading Tiktoken tokenizer.model.
Split
PretrainedTokenizerBase.from_pretrainedinto two separate methods:from_pretrainedand_from_pretrained.Prefer not to use FastTokenizer even it is available. (When you want to load TokenizerFast through AutoTokenizer, you should explicitly set
use_fast=True)Use
LazyMappingto load keys and values when it is accessed.Do not allow multiple
TokenizerClasses in atokenizer.pye.g.
Previously
paddlenlp.transformers.albert.tokenizerwas split intopaddlenlp.transformers.albert.tokenizer,paddlenlp.transformers.albert_chinese.tokenizer,paddlenlp.transformers.albert_english.tokenizerPreviously
paddlenlp.transformers.mbart.tokenizerwas split intopaddlenlp.transformers.mbart.tokenizer,paddlenlp.transformers.mbart50.tokenizerModify
tests/transformers/test_modeling_common.pyto supportLlamaTokenizerFastTOKENIZER_MAPPING_NAMES,MODEL_NAMES_MAPPING,CONFIG_MAPPING_NAMESshould be reviewed carefully