src: move all 1-byte encodings to native #61118

ChALkeR · 2025-12-18T21:03:09Z

Tracking: #61041

This builds on top of #61093 and gives an additional ~2x improvement by moving the same logic to native

The old native fast path removed in #61093 had a bunch of nested ifs and converted data from utf16 to utf8 and then back to utf16
This instead constructs strings using direct maps as #61093 and returns them as raw buffers

windows-1252, main pre-#61093:

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	0.31 GiB/s	0.272 ms
Complex 1	79.771 KiB	0.06 GiB/s	1.292 ms

windows-1252, main with #61093:

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	33.41 GiB/s	0.003 ms
Complex 1	79.771 KiB	1.48 GiB/s	0.056 ms

windows-1252, this PR:

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	36.83 GiB/s	0.002 ms
Complex 1	79.771 KiB	3.11 GiB/s	0.027 ms

This also similarly improves all other 1-byte encodings compared to #61093

For comparison, Bun:

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	36.89 GiB/s	0.003 ms
Complex 1	79.771 KiB	0.23 GiB/s	0.329 ms

v8/jsc is not an issue, unoptimal code is

cc @nodejs/performance

nodejs-github-bot · 2025-12-18T21:03:14Z

Review requested:

@nodejs/startup

lib/internal/encoding.js

ChALkeR · 2026-01-17T11:19:05Z

#61093 landed, rebased.

src/encoding_binding.cc

mertcanaltin · 2026-01-17T11:50:46Z

src/encoding_singlebyte.h

+const uint16_t* const tSingleByteEncodings[28] = {
+  tIBM866, tKOI8_R, tKOI8_U, tMacintosh, tXMacCyrillic,
+  tISO_8859_2, tISO_8859_3, tISO_8859_4, tISO_8859_5, tISO_8859_6, tISO_8859_7,
+  tISO_8859_8, tISO_8859_8,  // second time for for 8-i
+  tISO_8859_10, tISO_8859_13, tISO_8859_14, tISO_8859_15, tISO_8859_16,
+  tWindows874, tWindows1250, tWindows1251, tWindows1252, tWindows1253,
+  tWindows1254, tWindows1255, tWindows1256, tWindows1257, tWindows1258
+};
+
+// Matches the list of encoding sin single-byte.js
+const bool fSingleByteEncodings[28] = {
+  fIBM866, fKOI8_R, fKOI8_U, fMacintosh, fXMacCyrillic,
+  fISO_8859_2, fISO_8859_3, fISO_8859_4, fISO_8859_5, fISO_8859_6, fISO_8859_7,
+  fISO_8859_8, fISO_8859_8,  // second time for for 8-i
+  fISO_8859_10, fISO_8859_13, fISO_8859_14, fISO_8859_15, fISO_8859_16,
+  fWindows874, fWindows1250, fWindows1251, fWindows1252, fWindows1253,
+  fWindows1254, fWindows1255, fWindows1256, fWindows1257, fWindows1258
+};


If the header is included in more than one translation unit, it may cause ODR violations. We can either mark them inline or move their definitions to a .cc file.

Should be fixed now

src/encoding_singlebyte.h

mertcanaltin · 2026-01-17T11:54:00Z

For the cpp & other lint errors, this might help: make lint or CLANG_FORMAT_START=$(git merge-base HEAD main) make format-cpp

ChALkeR · 2026-01-17T13:05:55Z

@mertcanaltin lint was just not ready yet (as was noted in the PR description), fixed.
Thanks for the reviews!

codecov · 2026-01-17T14:22:25Z

Codecov Report

❌ Patch coverage is 83.67347% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.95%. Comparing base (77e8d44) to head (bec0adb).

Files with missing lines	Patch %	Lines
src/encoding_binding.cc	69.38%	5 Missing and 10 partials ⚠️
lib/internal/encoding.js	97.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #61118      +/-   ##
==========================================
- Coverage   89.87%   88.95%   -0.92%     
==========================================
  Files         671      670       -1     
  Lines      203178   203117      -61     
  Branches    39062    38967      -95     
==========================================
- Hits       182599   180681    -1918     
- Misses      12926    14717    +1791     
- Partials     7653     7719      +66

Files with missing lines	Coverage Δ
src/encoding_binding.h	`100.00% <ø> (ø)`
src/string_bytes.cc	`69.20% <100.00%> (+0.18%)`	⬆️
src/string_bytes.h	`77.77% <ø> (ø)`
lib/internal/encoding.js	`98.95% <97.87%> (-1.05%)`	⬇️
src/encoding_binding.cc	`55.40% <69.38%> (+2.67%)`	⬆️

... and 109 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gurgunday

nice!

lgtm other than one comment

gurgunday · 2026-01-17T15:44:31Z

src/encoding_binding.cc

+    return;
+  }
+
+  uint16_t* dst = node::UncheckedMalloc<uint16_t>(length);


Here we can allocate a huge uint16_t[length] before any V8 string-length limit check, and if it ends up being higher than what V8 accepts, we would allocate for no reason

Maybe it would be better to guard this with sth like:

Suggested change

uint16_t* dst = node::UncheckedMalloc<uint16_t>(length);

if (length > static_cast<size_t>(v8::String::kMaxLength)) {

// throw

}

uint16_t* dst = node::UncheckedMalloc<uint16_t>(length);

Let me know if I'm missing something

ChALkeR · 2026-01-21T20:08:51Z

src/string_bytes.h

+  static v8::MaybeLocal<v8::Value> Raw(v8::Isolate* isolate,
+                                       uint16_t* buf,
+                                       size_t buflen);
+


I'm not sure about adding this
cc @addaleax @joyeecheung perhaps, wdyt?

Co-authored-by: Mert Can Altin <mertgold60@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Dec 18, 2025

ChALkeR changed the title ~~perf: move all 1-byte encodings to native~~ src: move all 1-byte encodings to native Dec 18, 2025

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch 7 times, most recently from 6130c12 to 4be8283 Compare December 19, 2025 00:16

ChALkeR mentioned this pull request Dec 19, 2025

src: improve windows1252 decoding speed #61120

Draft

mcollina reviewed Dec 19, 2025

View reviewed changes

lib/internal/encoding.js Outdated Show resolved Hide resolved

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from 4be8283 to 6a46ae3 Compare December 19, 2025 16:58

ChALkeR mentioned this pull request Dec 27, 2025

lib: implement all 1-byte encodings in js #61093

Merged

This was referenced Jan 17, 2026

lib: correct windows-1252 decoding in TextDecoder #60737

Open

lib: map Latin1 labels to iso-8859-1 instead of Windows-1252 #58890

Closed

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from 6a46ae3 to 109aa87 Compare January 17, 2026 11:16

ChALkeR marked this pull request as ready for review January 17, 2026 11:18

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch 3 times, most recently from 51adfc3 to 82a6ecc Compare January 17, 2026 11:47

mertcanaltin reviewed Jan 17, 2026

View reviewed changes

src/encoding_binding.cc Show resolved Hide resolved

src/encoding_binding.cc Outdated Show resolved Hide resolved

mertcanaltin reviewed Jan 17, 2026

View reviewed changes

src/encoding_singlebyte.h Outdated Show resolved Hide resolved

mertcanaltin reviewed Jan 17, 2026

View reviewed changes

src/encoding_singlebyte.h Outdated Show resolved Hide resolved

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from 82a6ecc to 35b7960 Compare January 17, 2026 11:54

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch 7 times, most recently from b5d51ae to ee8b680 Compare January 17, 2026 13:03

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from ee8b680 to 0847010 Compare January 17, 2026 13:27

ChALkeR mentioned this pull request Jan 17, 2026

lib: unify ICU and no-ICU TextDecoder #61409

Open

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from 0847010 to 1d96511 Compare January 17, 2026 14:50

gurgunday reviewed Jan 17, 2026

View reviewed changes

ChALkeR mentioned this pull request Jan 18, 2026

src: improve StringBytes::Encode perf on UTF8 #61131

Merged

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch 2 times, most recently from 0ff7b97 to 7e40472 Compare January 21, 2026 20:00

ChALkeR requested review from gurgunday and mertcanaltin January 21, 2026 20:02

lemire approved these changes Jan 21, 2026

View reviewed changes

ChALkeR commented Jan 21, 2026

View reviewed changes

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from 7e40472 to fee12a7 Compare January 21, 2026 20:13

src: move all 1-byte encodings to native

bec0adb

Co-authored-by: Mert Can Altin <mertgold60@gmail.com>

ChALkeR force-pushed the chalker/decoder/single-byte/1 branch from fee12a7 to bec0adb Compare January 21, 2026 20:21

Uh oh!

src: move all 1-byte encodings to native #61118

Are you sure you want to change the base?

src: move all 1-byte encodings to native #61118

Conversation

ChALkeR commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Dec 18, 2025

Uh oh!

Uh oh!

ChALkeR commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mertcanaltin Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

ChALkeR Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mertcanaltin commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChALkeR commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gurgunday left a comment

Choose a reason for hiding this comment

Uh oh!

gurgunday Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

ChALkeR Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

ChALkeR Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ChALkeR commented Dec 18, 2025 •

edited

Loading

ChALkeR commented Jan 17, 2026 •

edited

Loading

mertcanaltin commented Jan 17, 2026 •

edited

Loading

ChALkeR commented Jan 17, 2026 •

edited

Loading

codecov bot commented Jan 17, 2026 •

edited

Loading