Their tokenizer is open source: https://github.com/openai/tiktoken Data files th...

		dchest on Jan 22, 2024 \| parent \| context \| favorite \| on: GPT-3.5 crashes when it thinks about useRalativeIm... Their tokenizer is open source: https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...