Fixing a crash when using the H.264 encoder in Chrome Remote Desktop
TL;DR: calculate row size and offsets for |input_sample_| w/o using the stride of the frame provided in ProcessInput(). I am working on getting our H.264 HW encoding solution running again after ~3 year hiatus (we did some prototyping in 2017 but haven't used it since). This solution used to work but when I tried it this week, I hit a crash in libyuv::I420ToNV12() which was called from MediaFoundationVideoEncodeAccelerator::ProcessInput(). After some debugging, the crash appears to be due to the params passed to libyuv::I420ToNV12(). The dst_* params get initialized using the stride of the planes in the VideoFrame instead of the width (in pixels) of the screen. |input_sample_| is initialized using a size which does not take stride into account so the libyuv conversion function eventually runs off the edge of the input_buffer and AVs. It looks like this change was introduced in: https://chromium-review.googlesource.com/c/chromium/src/+/2124213 I'm not sure why other usages of this class have not hit this crash. I suspect that either the usage in the Chrome browser use a VideoFrame with no additional padding (stride - width == 0) or there is something specific to how we are are using this class. Change-Id: Id16415c93cc7f1180d1964e88be4b9db2a6ab36c Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2572751 Commit-Queue: Joe Downing <joedow@chromium.org> Reviewed-by:Dale Curtis <dalecurtis@chromium.org> Cr-Commit-Position: refs/heads/master@{#834976}
Showing
Please register or sign in to comment