ishangsf/workflows
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix tiktoken_util.py to set TIKTOKEN_CACHE_DIR dynamically

Browse Source
This commit is contained in:
bobo 2024-04-29 11:06:30 +08:00 committed by kagami
parent daea79e719
commit 801c146348
2 changed files with 100259 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100256
unit_tests/tools/tiktoken_cache/9b5ad71b2ce5302211f9c61530b329a4922fc6a4 Normal file
View File

File diff suppressed because it is too large Load Diff

3
unit_tests/tools/tiktoken_util.py
View File

@ -1,5 +1,8 @@
import os
import tiktoken
script_dir = os.path.dirname(os.path.realpath(__file__))
os.environ['TIKTOKEN_CACHE_DIR'] = os.path.join(script_dir, 'tiktoken_cache')
def get_encoding(encoding_name: str):
"""