[Python] clip_tokenize: unknown token ' '

[h3ndrik]

I've copy-pasted the example python code from examples/python_bindings/README.md

The tokenizer complains about the spaces in 'cat on a Turtle'.

I've tried both "mys/ggml_CLIP-ViT-B-32-laion2B-s34B-b79K/CLIP-ViT-B-32-laion2B-s34B-b79K_ggml-model-f16.gguf" and the q8_0 variant.

Full log:

(venv)$ python test_clip.py 
[File Info] models/CLIP-ViT-B-32-laion2B-s34B-b79K_ggml-model-q8_0.gguf
clip_model_load: description:  two-tower CLIP model
clip_model_load: GGUF version: 2
clip_model_load: alignment:    32
clip_model_load: n_tensors:    397
clip_model_load: n_kv:         25
clip_model_load: ftype:        q8_0

clip_model_load: text_encoder:   1
clip_model_load: vision_encoder: 1
clip_model_load: model size:     156.10 MB
clip_model_load: metadata size:  0.13 MB

clip_model_load: text model hparams
n_vocab            49408
num_positions      77
t_hidden_size      512
t_n_intermediate   2048
t_projection_dim   512
t_n_head           8
t_n_layer          12

clip_model_load: vision model hparams
image_size         224
patch_size         32
v_hidden_size      768
v_n_intermediate   3072
v_projection_dim   512
v_n_head           12
v_n_layer          12

clip_model_load: 24 MB of memory allocated
clip_tokenize: unknown token ' '
Similarity score: 0.1402660459280014

[monatis]

Hi @h3ndrik, thanks for raising the issue. Did you compile it from source or install from PyPi?

[h3ndrik]

PyPi

I did an pip install clip_cpp in a Python 3.11.9 venv

[monatis]

Weird. Cannot reproduce on Ubuntu 24.04. But look more into this later. Can you provide what OS and locale are you using?

[h3ndrik]

Alright, thanks for confirming this is a bug. Maybe I can find something. I mean it seems to run. Just the tokenization seems to fail.

LANG=en_IE.UTF-8 but I've set a few things to de_DE.UTF-8

OS is: NixOS 24.11 (which might be a problem, since NixOS is very different to other Linux distros and for example doesn't follow the FHS. I thought I set up the environment correctly to mitigate for that, but maybe I made a mistake.)

[h3ndrik]

Seems to be the capitalized "T" in "Turtle". If I do the following, it runs without errors:
text_2encode = 'cat on a turtle'

But I don't know why it doesn't show 'T' in the warning message then, but ' '

[monatis]

Oh missed that. CLIP tokenizers lowercase the input text. See the "do_lowercase": true setting here

[monatis]

But I should add lowercasing anyway.

[MotzWanted]

@monatis, I'm encountering a similar issue with the conversion of the ViT-B-16-SigLIP-i18n-256 model to gguf using the script models/convert_hf_to_gguf.py. Despite the conversion completing successfully, the model's performance does not align with the expected behavior I observe when running it via open-clip.

When executing the same Python code as @h3ndrik, including lowercase string, the tokenizer fails to recognize spaces, leading to unknown token errors. Could you provide insights into what might be going wrong with the conversion process?

clip_model_load: model name:   ViT-B-16-SigLIP-i18n-256-hf
clip_model_load: description:  two-tower CLIP model
clip_model_load: GGUF version: 3
clip_model_load: alignment:    32
clip_model_load: n_tensors:    397
clip_model_load: n_kv:         25
clip_model_load: ftype:        f32

clip_model_load: text_encoder:   1
clip_model_load: vision_encoder: 1
clip_model_load: model size:     1388.05 MB
clip_model_load: metadata size:  0.13 MB

clip_model_load: text model hparams
n_vocab            250200
num_positions      64
t_hidden_size      768
t_n_intermediate   3072
t_projection_dim   768
t_n_head           12
t_n_layer          12

clip_model_load: vision model hparams
image_size         256
patch_size         16
v_hidden_size      768
v_n_intermediate   3072
v_projection_dim   768
v_n_head           12
v_n_layer          12

clip_model_load: 24 MB of memory allocated
clip_tokenize: unknown token ' '
clip_tokenize: unknown token ' '
clip_tokenize: unknown token ' '

[LimitlessDonald]

@monatis, I'm encountering a similar issue with the conversion of the ViT-B-16-SigLIP-i18n-256 model to gguf using the script models/convert_hf_to_gguf.py. Despite the conversion completing successfully, the model's performance does not align with the expected behavior I observe when running it via open-clip.

When executing the same Python code as @h3ndrik, including lowercase string, the tokenizer fails to recognize spaces, leading to unknown token errors. Could you provide insights into what might be going wrong with the conversion process?

clip_model_load: model name:   ViT-B-16-SigLIP-i18n-256-hf
clip_model_load: description:  two-tower CLIP model
clip_model_load: GGUF version: 3
clip_model_load: alignment:    32
clip_model_load: n_tensors:    397
clip_model_load: n_kv:         25
clip_model_load: ftype:        f32

clip_model_load: text_encoder:   1
clip_model_load: vision_encoder: 1
clip_model_load: model size:     1388.05 MB
clip_model_load: metadata size:  0.13 MB

clip_model_load: text model hparams
n_vocab            250200
num_positions      64
t_hidden_size      768
t_n_intermediate   3072
t_projection_dim   768
t_n_head           12
t_n_layer          12

clip_model_load: vision model hparams
image_size         256
patch_size         16
v_hidden_size      768
v_n_intermediate   3072
v_projection_dim   768
v_n_head           12
v_n_layer          12

clip_model_load: 24 MB of memory allocated
clip_tokenize: unknown token ' '
clip_tokenize: unknown token ' '
clip_tokenize: unknown token ' '

Hi, were you able to resolve this ?

[monatis]

@LimitlessDonald When I have a quick look at this model, it uses T5Tokenizer as tokenizer_class in tokenizer_config.json. The original CLIP uses BERTTokenizer and this implementation is aligned for it. AFAIK, T5 uses a sentencepiece (BPE) tokenizer, which regards the space character as an ordinary byte, unlike the wordpiece tokenizer. So this requires to implement sentencepiece algorithm as a tokenizer implemented in.

[LimitlessDonald]

@monatis Thanks for your feedback and your software, its really helpful.

I just wanted to know if there is any plans to support multi-language models like this https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1

[monatis]

Hi @LimitlessDonald, I'm aware that this project remained a little bit outdated, but it is getting continuous attention from the community to my surprise and it's quite satisfying. I'mm 100% sure that it deserves a rework to support multilingual tokenizers, other varients and compute backends. The only reason why I cannot give an exact deadline is, I'm running a newly established startup of my own, but I'll do my best to do these feature developments very soon.

[LimitlessDonald]

Thanks @monatis . I am quite an experienced software engineer, I have implemented many models in my work, but not built something like your software, I would like to contribute to it if possible . Do you have any pointers to some crash course , that is condensed to get me up and running quickly to enable me contribute to this repo ?

[monatis]

hi @LimitlessDonald, the main tensor library and file format used in this repository is the same as in GGML / GGUF / llama.cpp. This repository was written at a very early stage of llama.cpp when we just coded the GGUF file format, then we made some modifications to the GGUF format itself and, more importantly, compute backends in GGML. So the current implementations in llama.cpp and GGML might differ from that in this repo, that's why I need a modernization in the codebase. If you are interestedin these work, This wiki page might be a good starting point. Sorry but I don't know a comprehensive documentation because GGML and llama.cpp are fast-moving projects, lacking detailed and difinitive documentation. You should embrace the necessity to read the source code from time to time to better understand how things are implemented and wired together etc.