Backend
btoa('café') gives you the wrong bytes, and nothing tells you
Vishnu Shankar DEV Community
1 views
Run this in your browser console:
btoa('café');
// 'Y2Fm6Q=='
Now encode the same word as UTF-8 first:
btoa(String.fromCharCode(...new TextEncoder().encode('café')));
// 'Y2Fmw6k='
Same word, two different answers. Both are valid base64. Neither one throws. Only the second is what everyone else means by "café in base64".
Why this happens
btoa and atob are old. They work on "binary strings", where every character has to fit into one byte.
Anything above U+00FF has no byte to go into, so btoa throws. Anything at or below U+00FF gets used as its own byte value. That is latin-1, not UTF-8.
Take é. It is U+00E9. In latin-1 that is one byte, 0xE9. In UTF-8 it is two bytes, 0xC3 0xA9. btoa picks the latin-1 one. Everything else expects the UTF-8 one.
See it yourself
Paste this in once and you get all four cases:
const show = (text) => {
let naive;
try {
naive = btoa(text);
} catch (e) {
naive = 'THROWS ' + e.name;
}
const correct = btoa(String.fromCharCode(...new TextEncoder().encode(text)));
console.log({ text, naive, correct, back: atob(correct) });
};
['José 🎉', '日本語テキスト', 'Grüße aus München', 'café'].forEach(show);
Here is what comes back. The console prints the broken text with the control characters escaped, like \x9f, which is a hint at the real problem. More on that below.
Input
btoa(text)
atob(correct base64)
José 🎉
throws InvalidCharacterError
José ð\x9f\x8e\x89
日本語テキスト
throws InvalidCharacterError
æ\x97¥æ\x9c¬èª\x9eã\x83\x86...
Grüße aus München
R3L832UgYXVzIE38bmNoZW4=
GrüÃ\x9fe aus München
café
Y2Fm6Q==
café
Three things happen, not one
Encoding is loud. Anything above U+00FF throws. Emoji and CJK break straight away. That is the good case. You find out.
Decoding is quiet. Run atob on proper UTF-8 base64 and you get mojibake back, with no error at all. This is the one that hurts, and it is the common one, because the base64 usually came from somewhere else. A backend, a JWT, another tool.
Plain accented text survives. btoa('café') then atob gives you back café, exactly. é fits in one byte, so a broken tool talking to itself looks completely fine.
That last one is why this sits in code for years. Encode and decode in the same place and the test passes. So does clicking around by hand. You need two implementations before anything looks wrong, and by then the bytes are already in a database.
What the mojibake hides
The broken output has C1 control characters in it, between 0x80 and 0x9f. They do not render at all.
José 🎉 comes back as ten characters. You see seven.
So if you copy that into a bug report to show what went wrong, the invisible ones get dropped along the way, and it looks less broken than it is. It hides from the exact thing you would use to find it.
The fix
Stop handing text to btoa. Turn it into bytes yourself, and be strict on the way back.
const UTF8_ENCODE = new TextEncoder();
const UTF8_DECODE = new TextDecoder('utf-8', { fatal: true });
function encodeBase64(text) {
const bytes = UTF8_ENCODE.encode(text);
const CHUNK = 0x8000;
let binary = '';
for (let i = 0; i < bytes.length; i += CHUNK) {
binary += String.fromCharCode.apply(null, bytes.subarray(i, i + CHUNK));
}
return btoa(binary);
}
function decodeBase64(b64) {
const bytes = Uint8Array.from(atob(b64), (ch) => ch.charCodeAt(0));
return UTF8_DECODE.decode(bytes);
}
Three things in there matter.
fatal: true. Without it, TextDecoder swaps bad bytes for U+FFFD and hands you a string anyway. That puts you back where you started: a wrong answer and no error. With it, bad input throws, so you can tell "this is not text" from "this is text and I broke it".
The chunked loop. Do not write String.fromCharCode(...bytes). Spreading a big array into arguments blows the stack. I tested it: 100,000 was fine, 125,000 threw Maximum call stack size exceeded. The limit moves between engines, so do not tune to that number. Just do not spread.
Lone surrogates. TextEncoder swaps an unpaired surrogate for U+FFFD instead of failing, so your input changes before you have encoded anything. Check first if you care:
const LONE_SURROGATE = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
Two more things
There are two alphabets. RFC 4648 §4 uses + and /. §5 uses - and _, usually with the padding stripped. JWTs and most APIs use §5. Just accept both when you decode. You can tell them apart, and asking the user which one they have is worse than working it out.
Node does not check either. Buffer.from(b64, 'base64').toString('utf8') hands you replacement characters instead of throwing, same as a non-fatal TextDecoder. Use TextDecoder with fatal: true if the difference matters.
In short
atob and btoa are latin1-only and always have been. Encoding fails loudly, which is fine. Decoding fails quietly, which is not. And plain accented text round-trips fine through a broken tool, so nobody notices until the data goes somewhere else.
I ran into all of this building the base64 tool on Express Lint, which is why it goes through TextEncoder and comes back through a fatal TextDecoder instead of handing your text to btoa. It does both directions if you want to try any of this without writing code. Free, no account, and what you paste stays in the browser.
Read original: https://dev.to/vish045/btoacafe-gives-you-the-wrong-bytes-and-nothing-tells-you-4i20
← Previous
Build with Gemini Event Review: Developing AI Agents with ADK and Agents CLI
Next →
I replaced a paid pomodoro app with a 300-line Python script
Related
I replaced a paid pomodoro app with a 300-line Python script
Backend
1
DEV Community
Three Playwright/Apify bugs that took me way too long to find (and the fixes)
Backend
1
DEV Community
SOLID in the WordPress Ecosystem: What 4wp.dev's Plugins Actually Prove
Backend
2
DEV Community
What does 20x more OCR model size actually buy you?
Backend
3
Dev.to (EN Zone)
Comments0
No comments yet — be the first