Have you modified the recording quality option, encoder & format in your tinkering? Assuming it is the same as windows you should have different levels of quality with corresponding increasing file sizes, if u raise this by a tick or two I imagine it could fix the problem. Otherwise, I imagine there is a conflict with game resolution / your actual computer's resolution, try to make the canvas and output the same as your monitor. This is as far as my help can go as a windows user but best of luck! Also I apologize for not informing of this but I assisted the OP of this thread in discord and the issue is now resolved.
