ToRunes aliases the rune array, and the editing actions append into
t.input in place when the cursor is not at the end, so the keystrokes
edit the item. Non-ASCII items only, ASCII gets a fresh slice.
printf '한글abcde\n' | fzf --bind 'ctrl-y:replace-query'
ctrl-y, Left, BSpace, ctrl-u -> 한글abcee
Runes and ToRunes are now documented read-only. A stale fold bit was the
other symptom, letting the prefilter reject an item the general path
matches.
asciiFuzzyIndex gave up on non-ASCII lines, so every item ran the full
score matrix. A []rune is a fixed 4-byte stride, so the SIMD byte
scanners can run over it directly: find the low byte, then confirm
4-byte alignment and three zero bytes.
Case folding and normalization can turn a non-ASCII rune into the ASCII
char being searched, which the scan cannot see. ToChars now flags lines
holding such a rune and those keep the old path. Normalization is
Latin-only, so Hangul, CJK, Cyrillic, Greek, Hebrew, Arabic, Thai, kana
and emoji never set the flag.
Chars had no spare padding, so inBytes moves into a flags byte.
Measured on 1.4M-line corpora:
- Mostly-ASCII paths behind a Hangul prefix: 'conf' 1.8x, 'binutils'
2.9x, 'ltversion' 4.0x, no-match 8.4x
- Every line CJK: 17x on both matching and non-matching queries
- ASCII input unchanged, non-ASCII patterns not covered yet
- Capacity was byte length, over-allocating by bytes-per-rune (2-4x)
- Count non-continuation bytes with SWAR before allocating
- Invalid bytes undercount, never overcount, so append covers the gap
- Query performance unchanged, this is a memory fix
- The gain tracks bytes-per-rune, the cost tracks how much of the line
follows the first non-ASCII byte, so the two move independently
Measured on 1.4M-line corpora:
- Every line CJK: RSS 362MB -> 255MB, ingestion -5%
- Mostly-ASCII paths behind a Hangul prefix: RSS 556MB -> 533MB,
ingestion +3.5%, the counting pass covering the whole line
- The same paths with the Hangul at the end: RSS 563MB -> 535MB,
ingestion +0.9%, the counting pass covering six bytes
When --with-nth is used, fzf used to preprocess each line and store the
result as rune array, which was wasteful if the line only contains ascii
characters.
In the best case (all ascii), this reduces the memory footprint by 60%
and the response time by 15% to 20%. In the worst case (every line has
non-ascii characters), 3 to 4% overhead is observed.
Junegunn Choi