[core] Harden switch_b64_encode output bound (#3111)

switch_b64_encode wrote each base64 character before checking the
output bound, and computed that bound as `bytes >= (int)olen - 1`.
Both are unsafe:

- Writing before the check meant a buffer with no room for a data
  byte still received one: `olen == 1` wrote a character at index 0
  and the NUL at index 1, and `olen == 0` (where `(int)olen - 1` is
  -1) wrote two bytes, both past the end.
- Casting the unsigned `olen` to `int` truncated it above INT_MAX,
  producing a wrong bound for very large buffers.

Reject `olen == 0`, make the output index `bytes` a `size_t`, and test
`bytes + 1 >= olen` before each write, so the bound never casts or
underflows and always leaves the last byte for the terminator. Apply
the same `bytes + 1 < olen` guard to the trailing partial-group
character and the `=` padding, so no write can pass the end.

Remove the dead line-wrap counter `y`: it only gated a commented-out
newline emit, so it counted and reset with no effect on the output.

Add unit tests covering the output-bound edges for every write site.
This commit is contained in:
Dmitry Verenitsin
2026-08-08 22:05:20 +03:00
committed by GitHub
parent 394b48b9f5
commit 3630e3dd85
2 changed files with 82 additions and 11 deletions
+9 -11
View File
@@ -1022,32 +1022,30 @@ static const char switch_b64_table[65] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijkl
#define B64BUFFLEN 1024
SWITCH_DECLARE(switch_status_t) switch_b64_encode(unsigned char *in, switch_size_t ilen, unsigned char *out, switch_size_t olen)
{
int y = 0, bytes = 0;
size_t x = 0;
size_t x = 0, bytes = 0;
unsigned int b = 0, l = 0;
if (olen == 0) { /* no room even for the trailing NUL */
return SWITCH_STATUS_FALSE;
}
for (x = 0; x < ilen; x++) {
b = (b << 8) + in[x];
l += 8;
while (l >= 6) {
out[bytes++] = switch_b64_table[(b >> (l -= 6)) % 64];
if (bytes >= (int)olen - 1) {
if (bytes + 1 >= olen) { /* reserve the last byte for the NUL */
goto end;
}
if (++y != 72) {
continue;
}
/* out[bytes++] = '\n'; */
y = 0;
out[bytes++] = switch_b64_table[(b >> (l -= 6)) % 64];
}
}
if (l > 0) {
if (l > 0 && bytes + 1 < olen) {
out[bytes++] = switch_b64_table[((b % 16) << (6 - l)) % 64];
}
if (l != 0) {
while (l < 6 && bytes < (int)olen - 1) {
while (l < 6 && bytes + 1 < olen) {
out[bytes++] = '=', l += 2;
}
}