NLPExplorer
Papers
Venues
Authors
Authors Timeline
Field of Study
URLs
ACL N-gram Stats
TweeNLP
API
Team
Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
Sachin Pawar
|
Manoj Apte
|
Kshitij Jadhav
|
Girish Keshav Palshikar
|
Nitin Ramrakhiyani
|
Paper Details:
Month: December
Year: 2025
Location: Mumbai, India
Venue:
IJCNLP |
AACL |
Citations
URL
No Citations Yet
https://spacy.io/api/tokenizer/
https://huggingface.co/microsoft/Phi-3
https://huggingface.co/mistralai/
https://huggingface.co/Qwen/Qwen2
https://huggingface.co/meta-llama/
https://docs.scipy.org/doc/scipy/reference/
https://docs.scipy.org/doc/scipy/reference/
Field Of Study