Hello,
As a newbie myself with H living at home since BD 4 months ago H has said exactly the same to me. I have to witness him disappearing off daily to see OW and yet as he also works from home - I see him here most of the remaining time. However, legally if I leave the home and leave my S here, I could end up losing it even though I pay the mortgage and 85% of the bills. H is in the same legal position technically but if I leave it will be even harder for me to return as women leaving the home are stated to have abandoned the family. I have two friends who did leave and lost their case.
It depends on your legal rights where you live but realistically why should you leave when he is the one causing all the grief and pain?
Read and re-read everything you can on here, I know that one of the wise mentors will point you in a good direction for a link but reading one of RCRs blog articles on "should my MLCer stay or go" is a good start.
However know that you are not alone in having that conversation or feelings - there are quite a few of us exactly in that position.