You always loose quality when you reencode a video (or audio). Something as a lossless encoding way that will still reduce the size doesn't exists. Also it's not HandBrake itself that does the work, it's x264, x265, vpx or any other encoder you select in it. And to get back to the original topic, x264 and x265 are both made by VideoLAN, the organisation that also made VLC. :)