HybridChunker with tiktoken tokenizer #1031
Unanswered
ruizguille
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
What would be the recommended way to use
HybridChunkerwith the tiktoken tokenizer for use with OpenAI's embedding models? Reading the code, it seems to accept only HuggingFace tokenizers.Thank you for the amazing library!
Beta Was this translation helpful? Give feedback.
All reactions