Oct
22
1 month ago
push

IWANABETHATGUY push alsotang/is_chinese_rs

IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY

feat: 🎸 fix condition compile

IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY
IWANABETHATGUY

Merge pull request #6 from IWANABETHATGUY/condition-compile

Condition compile

commit sha: a175bf8e206a9efc713ac918f123bad8db64b0ef

push time in 1 month ago
Oct
21
1 month ago
Activity icon
issue

IWANABETHATGUY issue comment alsotang/is_chinese_rs

IWANABETHATGUY
IWANABETHATGUY

Condition compile

IWANABETHATGUY
IWANABETHATGUY

node版本就不动了。这个库一般是用来判断昵称或者单字符用的,不是用来判断长文章的。

这里的node addon 只是顺便做下实验,并不是作为一个替代品🤣

Activity icon
issue

alsotang issue comment alsotang/is_chinese_rs

alsotang
alsotang

Condition compile

alsotang
alsotang

node版本就不动了。这个库一般是用来判断昵称或者单字符用的,不是用来判断长文章的。

Activity icon
issue

IWANABETHATGUY issue comment alsotang/is_chinese_rs

IWANABETHATGUY
IWANABETHATGUY

Condition compile

IWANABETHATGUY
IWANABETHATGUY

除此之外添加了 node addon

is_chinese_napi string x 5,295,950 ops/sec ±1.39% (89 runs sampled)
is_chinese_napi buffer x 6,370,853 ops/sec ±1.26% (90 runs sampled)
is_chinese_js x 10,824,426 ops/sec ±1.73% (85 runs sampled)
isChinese("扁担宽,板凳长,扁担想绑在板凳上。") bench suite: Fastest is is_chinese_js
is_chinese_napi string x 6,539,979 ops/sec ±1.83% (87 runs sampled)
is_chinese_napi buffer x 8,684,546 ops/sec ±1.63% (87 runs sampled)
is_chinese_js x 43,698,976 ops/sec ±1.50% (87 runs sampled)
isChinese("ss扁担宽,板凳长,扁担想绑在板凳上。") bench suite: Fastest is is_chinese_js
is_chinese_napi string x 6,300,983 ops/sec ±1.25% (90 runs sampled)
is_chinese_napi buffer x 8,043,654 ops/sec ±1.34% (85 runs sampled)
is_chinese_js x 30,377,247 ops/sec ±2.38% (88 runs sampled)
isChinese("扁担宽,板凳长,扁担想绑在板凳上。ss") bench suite: Fastest is is_chinese_js
is_chinese_napi string x 180,143 ops/sec ±1.51% (91 runs sampled)
is_chinese_napi buffer x 442,211 ops/sec ±1.40% (92 runs sampled)
is_chinese_js  x 351,385 ops/sec ±1.22% (87 runs sampled)
isChinese(chars1000) true bench suite: Fastest is is_chinese_napi buffer
is_chinese_napi string x 289,675 ops/sec ±1.22% (90 runs sampled)
is_chinese_napi buffer x 5,796,385 ops/sec ±1.43% (89 runs sampled)
is_chinese_js x 1,771,844 ops/sec ±1.15% (93 runs sampled)
isChinese(chars1000WithS) false bench suite: Fastest is is_chinese_napi buffer

node addon buffer 传递的overhead 有点高,在很多次bench时,overhead最终比rust 带来的收益还要高,但是当数据量比较大时,我们可以看到 rust 版本依然可以做到node 版本快。

Activity icon
issue

IWANABETHATGUY issue comment alsotang/is_chinese_rs

IWANABETHATGUY
IWANABETHATGUY

Condition compile

IWANABETHATGUY
IWANABETHATGUY

main branch:

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [52.302 ns 52.574 ns 52.884 ns]
Found 18 outliers among 100 measurements (18.00%)
  7 (7.00%) high mild
  11 (11.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [4.1467 ns 4.1949 ns 4.2447 ns]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [4.8500 ns 4.8838 ns 4.9228 ns]
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.8905 us 2.9146 us 2.9424 us]
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                            
                        time:   [100.18 ns 100.53 ns 100.94 ns]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

current branch:

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [46.530 ns 49.090 ns 52.108 ns]
                        change: [-30.027% -28.092% -26.047%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [1.8472 ns 1.8525 ns 1.8582 ns]
                        change: [+16.794% +18.154% +19.542%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [9.1091 ns 9.1742 ns 9.2601 ns]
                        change: [+60.777% +63.702% +66.532%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) high mild
  5 (5.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.0190 us 2.0553 us 2.1013 us]
                        change: [-31.444% -30.385% -29.334%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                            
                        time:   [50.272 ns 50.850 ns 51.482 ns]
                        change: [-51.617% -50.891% -50.187%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

这里的基准值有问题,实际上除了 is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss") ,均有明显的提升

Oct
20
1 month ago
push

alsotang push alsotang/is_chinese_rs

alsotang
alsotang

Merge pull request #5 from IWANABETHATGUY/feature/simd

feat: 🎸 simd

commit sha: d976066efc15d82c5209a619bc2f71d7f360882a

push time in 1 month ago
pull request

alsotang pull request alsotang/is_chinese_rs

alsotang
alsotang

feat: 🎸 simd

main branch:

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                                                                              
                        time:   [38.533 ns 38.819 ns 39.152 ns]
                        change: [-54.813% -54.023% -53.260%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [4.8136 ns 4.8368 ns 4.8620 ns]
                        change: [+138.40% +140.61% +142.54%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [41.423 ns 41.788 ns 42.235 ns]
                        change: [-5.8544% -4.5882% -3.3184%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  3 (3.00%) high mild
  11 (11.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.1482 us 2.1632 us 2.1816 us]
                        change: [-54.895% -54.039% -53.332%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  6 (6.00%) high mild
  7 (7.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                             
                        time:   [2.1280 us 2.1559 us 2.1968 us]
                        change: [-17.167% -15.932% -14.731%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

simd branch

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [52.302 ns 52.574 ns 52.884 ns]
Found 18 outliers among 100 measurements (18.00%)
  7 (7.00%) high mild
  11 (11.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [4.1467 ns 4.1949 ns 4.2447 ns]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [4.8500 ns 4.8838 ns 4.9228 ns]
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.8905 us 2.9146 us 2.9424 us]
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                            
                        time:   [100.18 ns 100.53 ns 100.94 ns]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

这次修改使用simd 加速了扫描 可打印 ascii 的速度, 对于全是中文的情况有 30% 左右的回归,这是正常的。但是,对于有ascii 的情况下速度可以提升 10-20倍 这次实现没有像 之前的 ascii_check 一样使用 chars() 来做遍历,这是因为 chars() 由于 可打印 ascii 字符 均小于等于 0x7f, aka 01111111 同时 utf-8 中的单字节字符最大也是 0x7f, 也就是说如果我们遍历 u8 时某一个 u8 小于等于127 他一定不属于任意中文字符中,参考 image

The str type, also called a 'string slice', is the most primitive string type. It is usually seen in its borrowed form, &str. It is also the type of string literals, &'static str.

String slices are always valid UTF-8.

&str 一定是 合法的 UTF-8 字符串。

因此一种很方便的方法就是遍历 &[u8] 判断是否有任意的 u8 <= 127, 如果有任意就说明存在一个可打印 ascii 字符,这里的逻辑 与你 nodejs 的实现是保持一致的。当然这里,很自然的想到用simd 来加速了。

pull request

IWANABETHATGUY pull request alsotang/is_chinese_rs

IWANABETHATGUY
IWANABETHATGUY

feat: 🎸 simd

main branch:

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                                                                              
                        time:   [38.533 ns 38.819 ns 39.152 ns]
                        change: [-54.813% -54.023% -53.260%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [4.8136 ns 4.8368 ns 4.8620 ns]
                        change: [+138.40% +140.61% +142.54%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [41.423 ns 41.788 ns 42.235 ns]
                        change: [-5.8544% -4.5882% -3.3184%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  3 (3.00%) high mild
  11 (11.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.1482 us 2.1632 us 2.1816 us]
                        change: [-54.895% -54.039% -53.332%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  6 (6.00%) high mild
  7 (7.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                             
                        time:   [2.1280 us 2.1559 us 2.1968 us]
                        change: [-17.167% -15.932% -14.731%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

simd branch

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [52.302 ns 52.574 ns 52.884 ns]
Found 18 outliers among 100 measurements (18.00%)
  7 (7.00%) high mild
  11 (11.00%) high severe

is_chinese("ss扁担宽,板凳长,扁担想绑在板凳上。")                                         
                        time:   [4.1467 ns 4.1949 ns 4.2447 ns]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

is_chinese("扁担宽,板凳长,扁担想绑在板凳上。ss")                                         
                        time:   [4.8500 ns 4.8838 ns 4.9228 ns]
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

isChinese(chars1000) true")                                                                             
                        time:   [2.8905 us 2.9146 us 2.9424 us]
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe

is_chinese("isChinese(chars1001) false")                                                                            
                        time:   [100.18 ns 100.53 ns 100.94 ns]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

这次修改使用simd 加速了扫描 可打印 ascii 的速度, 对于全是中文的情况有 30% 左右的回归,这是正常的。但是,对于有ascii 的情况下速度可以提升 10-20倍 这次实现没有像 之前的 ascii_check 一样使用 chars() 来做遍历,这是因为 chars() 由于 可打印 ascii 字符 均小于等于 0x7f, aka 01111111 同时 utf-8 中的单字节字符最大也是 0x7f, 也就是说如果我们遍历 u8 时某一个 u8 小于等于127 他一定不属于任意中文字符中,参考 image

The str type, also called a 'string slice', is the most primitive string type. It is usually seen in its borrowed form, &str. It is also the type of string literals, &'static str.

String slices are always valid UTF-8.

&str 一定是 合法的 UTF-8 字符串。

因此一种很方便的方法就是遍历 &[u8] 判断是否有任意的 u8 <= 127, 如果有任意就说明存在一个可打印 ascii 字符,这里的逻辑 与你 nodejs 的实现是保持一致的。当然这里,很自然的想到用simd 来加速了。

Previous