I couldn't help but notice you're having the same problem i am with switching to a home server.
Everything works great apart from the sendmail function (well tell a lie this does work but it's the isp's config which is the prob).
I've tried everything from consulting my IT manager at work to ringing my isp direct. Non of which have helped.
I did however make a break through when i downloaded a program called 1st Mail Server which simplified everything and was really easy to configure.
After setting this program up and getting it to talk to my website perfectly i managed to narrow it down to 1 thing. This was that my ISP would not allow the distribution of third party mail due to spam regulations or something. I found this out after days of testing, registering with my website using different email addresses and i found that the only emails it would pass on was emails to accounts with my isp's domain after it.
So i found my next step was to try and get a complete web mail server which does everything and not just a forwarder.
So if anyone knows of a remedy for this problem please feel free to contribute your knowledge.
I feel i am at whits end and am thinking of resorting back to paid webhosting.