I like ETags to reduce page loads, and Nginx does not support them. To get it support ETags, there is a module in GitHub, which I am using.
After a nice couple of hours of debugging it turned out that all other places in Nginx-code call ngx_memzero() before calling ngx_open_cached_file(). It turned out to be the crucial mistake in the module. I filed an issue to the original author to notify everybody else.
My production boxes did actually return weird errors now and then, which I didn't think much of. Weird things happen sometimes. However, this fix seems to help, there are no worker processes dying and page load erros seem to gone away. Hopefully the box does not crash again.