In my opinion setting up postfix on its own isn't that bad, but setting up postix + dovecot can be very challenging.
This article is a very good read for anyone who has never set up a mail server from scratch: https://blog.codinghorror.com/so-youd-like-to-send-some-email-through-code/

Btw everyone use letsencrypt so you can use ssl with your setup.