saving & loading config for AutoTokenizer
If trying to load the tokenizer from disc and config is not saved there, either:
- load it from huggingface:
tokenizer = AutoTokenizer.from_pretrained(path_to_tokenizer, config=AutoConfig.from_pretrained(model_name))
- throw an error
Save config (in load_tokenizer) if saving models is enabled
More information: