[Feature]: Use tiktoken in offline mode computer #886
Description
Feature Description
When using tiktoken in offline mode computer, I got the following error
SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken
Use Cases
Like the unitest in test_transform.py::TestTextSplitterBase::test_from_tiktoken_encoder_with_model_name
splitter = _TextSplitterBase(chunk_size=100, overlap=20)
splitter.from_tiktoken_encoder(encoding_name='gpt2')
Design Ideas / Technical Details
No response