The crash started apparently out-of-the-blue, hitting thousands of Argentinian users on a Debian-based distro called Huayra, and specifically on version 5 which was based on Debian 10.
https://bugzilla.mozilla.org/show_bug.cgi?id=1839139
Everybody seemed to crash while searching for images on Google. All versions of Firefox - even very old ones - were affected suggesting that the change didn't happen on our side, but on Google's. 2/6
A colleague analyzed Firefox' behavior at the point of crash and realized that it happened during stack probing. The JIT touched the area that would hold the variables for the next JavaScript call and somehow hit an overflow.
https://bugzilla.mozilla.org/show_bug.cgi?id=1839139#c8
This is where things got weird, Google's code was allocating 20000 variables in a single frame. Ouch, that's probably some machine-generated code which went out of hand. Think twice before using ChatGPT to write code. 3/6
But why was it crashing? Linux automatically extends the stack and we had reserved more than enough space, something that we confirmed by looking at the memory map of the affected processes.
Well it turns out that the Linux kernel used to have a check that prevented stack accesses that were too far from the stack pointer. Specifically accesses 64KiB + 256 bytes away would crash instead of extending the stack.
This was fixed in kernel 4.20 so users of more recent distros are unaffected, and we'll see if we can deploy a workaround to help users of older systems.
It is interesting though that we find ourselves working around a bug we did not introduce triggered by code we do not control. 5/6
And since we're at it let's shame Google for putting 20 thousand variables in a single function. Bad Google, no cookie.
Well no cookie anyway since Firefox has total cookie protection!
https://blog.mozilla.org/security/2021/02/23/total-cookie-protection/ 6/6
