The scan only ran for ASCII patterns, so searching CJK text with a CJK
query still built the full score matrix. Scan for one byte of the pattern
rune and verify all four.
Which byte matters. Every ASCII rune contributes three zero bytes, so
U+AE00 scanned by its zero low byte hits on nearly every character of an
ASCII-heavy line. Pick a byte that cannot occur in an ASCII rune, else
any non-zero one.
A non-ASCII pattern rune is safe only when no other rune lowercases onto
it. Uncased is not sufficient: U+00DF has no simple uppercase, yet U+1E9E
lowercases to it, so the foldable set is excluded too.
Measured on 1.4M-line corpora, with a non-ASCII query:
- Every line CJK: 5.1x to 10.2x
- Mostly-ASCII paths behind a Hangul prefix: 5.6x to 6.0x, and 1.2x
where every line matches so nothing can be rejected
- ASCII queries unchanged, kept off the non-inlinable guard
normalizeRune guarded with 0x00C0..0xFF61, which does not exclude Hangul,
CJK or Cyrillic, so every rune of those scripts hashed into the map only
to miss. Every key of the map folds to ASCII, so the bitmap added for the
rune prefilter rejects them without a lookup.
- Non-ASCII queries 1.21x where every line is CJK, 1.05x on mostly-ASCII
paths behind a Hangul prefix
- ASCII queries unchanged, the prefilter already skips Phase 2 for them
- Normalization share of query time for a non-ASCII query: 17.2% -> 0%
asciiFuzzyIndex gave up on non-ASCII lines, so every item ran the full
score matrix. A []rune is a fixed 4-byte stride, so the SIMD byte
scanners can run over it directly: find the low byte, then confirm
4-byte alignment and three zero bytes.
Case folding and normalization can turn a non-ASCII rune into the ASCII
char being searched, which the scan cannot see. ToChars now flags lines
holding such a rune and those keep the old path. Normalization is
Latin-only, so Hangul, CJK, Cyrillic, Greek, Hebrew, Arabic, Thai, kana
and emoji never set the flag.
Chars had no spare padding, so inBytes moves into a flags byte.
Measured on 1.4M-line corpora:
- Mostly-ASCII paths behind a Hangul prefix: 'conf' 1.8x, 'binutils'
2.9x, 'ltversion' 4.0x, no-match 8.4x
- Every line CJK: 17x on both matching and non-matching queries
- ASCII input unchanged, non-ASCII patterns not covered yet
- Vary normalize in the fuzz targets; document why the exhaustive sweep
keeps it fixed (no-op for ASCII input)
- Seed the fuzz targets with patterns of the right length; single-char
seeds were all two-char and got rejected
- Note that the disable* hook forbids parallel test execution
For two ASCII characters, rows 0 and 1 of the score matrix collapse to
scalar running state, so Phase 2 and Phase 3 fuse into one pass with no
score arrays. withPos stores the two rows for the backtrace. Up to 1.4x
on two-char queries, the most common multi-char length.
Verify both fast paths against the general algorithm with exhaustive
(every short string over a class-complete alphabet) and fuzz tests,
runnable via the new make fuzz target.
For single ASCII character, jump between occurrences with vectorized
search and score each directly from bonus matrix instead of running
full pipeline with prefilter and score arrays. 2x faster on 100K-line
benchmark corpus (5.4ms -> 2.5ms). Matters most on first keystroke
where candidate set is largest and result cache cannot help.
Phase 3 writes C rows only from column F[r], but tie-breaking in the
backtrace could read row i+1 at column j+1 left of F[i+1]. With reused
slab, that cell holds data from previously processed item, so highlight
positions in equal-score ties depended on processing order.
On 32-bit platforms (GOARCH=386, arm), N*M overflows int when N is
large and M approaches 1000, wrapping negative. The wrapped value
slips past both `N*M > cap(slab.I16)` and `M > 1000`, so the V1
fallback is skipped and alloc16 panics on a negative slice bound.
Cast to int64 before multiplying.
Affects shipped 32-bit ARM builds (linux_armv5/6/7, windows_armv5/6/7).
Reported with fix by Michal Majchrowicz and Marcin Wyczechowski
(AFINE Team).
A non-word character (e.g. '.') used to receive a flat bonusNonWord
regardless of context. Now it gets bonusBoundaryWhite at the start of
input and bonusBoundaryDelimiter right after a delimiter, matching the
treatment of word characters at the same boundaries.
Without this, '.completion' matching '.completion' lost to
'bash_completion.d/completions/X' because the consecutive chunk anchor
in the long path (the 'c' after '/') received bonusBoundaryDelimiter
while the exact match's '.' was capped at bonusNonWord.
Fix#4795
In Phase 3 of FuzzyMatchV2, when a cell's left neighbor score is <= 1
and the current character doesn't match the pattern character, the
cell's score is guaranteed to be 0 (since gap penalties are -1 and -3).
Skip the bonus/gap computation entirely and fast-forward through
consecutive non-matching characters in the dead zone.
This yields 6-11% faster fuzzy searches on typical workloads.
There is an edge-case in FuzzyMatchV1 during backward scan, related to
normalization: if string is initially denormalized (e.g. Unicode symbol),
backward scan will proceed further to the next char; however, when the
score is computed, the string is normalized first, then scanned based on
the pattern. This leads to accessing pattern index increment, which
itself leads to out-of-bound index access, resulting in a panic.
To illustrate the process, here's the sequence of operations when search
is perfored:
1. during backward scan by "minim" pattern
```
xxxxx Minímal example
^^^^^^^^^^^^
||||||||||||
miniiiiiiiim <- compute score for this substring
```
2. during compute score by "minim" pattern
```
Minímal exam
minimal exam <- normalize chars before computing the score
^^^^^^
||||||
minim <- at this point the pattern is already fully scanned and index
is out-of-the-bound
```
In this commit the char is normalized during backward scan, to detect
properly the boundaries for the pattern.
Find the last occurrence of the last character in the pattern and
perform the search algorithm only up to that point.
The effectiveness of this mechanism depends a lot on the shape of the
input and the pattern.
- Prefix matcher will trim leading whitespaces only when the pattern
doesn't start with a whitespace
- Suffix matcher will trim trailing whitespaces only when the pattern
doesn't end with a whitespace
- Equal matcher will trim leading whitespaces only when the pattern
doesn't start with a whitespace, and trim trailing whitespaces only
when the pattern doesn't end with a whitespace
Previously, only suffix matcher would trim whitespaces unconditionally.
Fix#1894
* Remove 1 unused field and 3 unused functions
unused elements fount by running
golangci-lint run --disable-all --enable unused
src/result.go:19:2: field `index` is unused (unused)
index int32
^
src/tui/light.go:716:23: func `(*LightWindow).stderr` is unused (unused)
func (w *LightWindow) stderr(str string) {
^
src/terminal.go:1015:6: func `numLinesMax` is unused (unused)
func numLinesMax(str string, max int) int {
^
src/tui/tui.go:167:20: func `ColorPair.is24` is unused (unused)
func (p ColorPair) is24() bool {
^
* Address warnings from "gosimple" linter
src/options.go:389:83: S1003: should use strings.Contains(str, ",,,") instead (gosimple)
if str == "," || strings.HasPrefix(str, ",,") || strings.HasSuffix(str, ",,") || strings.Index(str, ",,,") >= 0 {
^
src/options.go:630:18: S1007: should use raw string (`...`) with regexp.MustCompile to avoid having to escape twice (gosimple)
executeRegexp = regexp.MustCompile(
^
src/terminal.go:29:16: S1007: should use raw string (`...`) with regexp.MustCompile to avoid having to escape twice (gosimple)
placeholder = regexp.MustCompile("\\\\?(?:{[+sf]*[0-9,-.]*}|{q}|{\\+?f?nf?})")
^
src/terminal_test.go:92:10: S1007: should use raw string (`...`) with regexp.MustCompile to avoid having to escape twice (gosimple)
regex = regexp.MustCompile("\\w+")
^
* Address warnings from "staticcheck" linter
src/algo/algo.go:374:2: SA4006: this value of `offset32` is never used (staticcheck)
offset32, T := alloc32(offset32, slab, N)
^
src/algo/algo.go:456:2: SA4006: this value of `offset16` is never used (staticcheck)
offset16, C := alloc16(offset16, slab, width*M)
^
src/tui/tui.go:119:2: SA9004: only the first constant in this group has an explicit type (staticcheck)
colUndefined Color = -2
^
Junegunn Choi