That’s ok if we are talking about malware publicly shown in the published source code… but there’s also the possibility of a private source-code patch with malware that it’s secretly being applied when building the binaries for distribution.

This is why it’s important for builds to be reproducible, any third party should be able to build their own binary from clean source code and be able to obtain the exact same binary with the same hash. If the hashes match, then you have a proof of the binary being clean.