If the factory reset does not work, you should assume that the ROM itself got hacked. He probably simply installed a backdoored ROM, cooked to order (I've seen such ROMs made available for children's phones - whatever you do, Mom will still be able to track you and read your chats and emails). So, to all intents and purposes, that is his phone, no longer yours.
The only thing you can reasonably do is to get the phone to the tech representative and ask for a full ROM restore, mentioning the fact that there's probably an unreliable/hacked loader. This is known as bare-metal restore, or sometimes "unbricking".
While the phone is being serviced, change all your passwords. All at the same time. Possibly late at night when he's likely to be asleep. Then turn on two-factor authentication wherever possible, and be sure never to leave the phone in easy reach of anyone.