faireborn/zipfer
Zipf toolkit for tokenizer evaluation
zig build --release=fast
zipfer --target=encoded.txt --output=output_directory