I'd like to say that I also tried to find similar questions and found yours to be the best worded question demonstrating the correct basic ideas/worries but still enough knowledge to get it done if you decide to.
I was and am all for the idea of home server, but it should be done because you want to learn EVERYTHING about networking from top to bottom. If you just want it to work with minimal work, you'd save money and time by paying a service to host for you.
To start, I'd like to say I have been running a home server for probably 5 years (man time flies). My main site is https://www.freesoftwareservers.com/wiki. I run everything on a VMWare ESXi tower which I also passthrough my GPU to use as my main workstation. (To much $ to get a powerful server and workstation for me)
I have had hacking attempts back when I used to host SSH on port 22. But eventually I just closed SSH to myself from public since its the most bot-attacked server surface in existence.
I'll start by addressing your highlighted portion of the question then get onto a few things I "learned" along the way (in regards to hosting a home server, I learned TONS about linux, but you can do that without hosting a home server).
What are some "safe" ways to run a public server from your home?
First off is your idea of a whitelist without VPN.
The most ideal situation in my mind, would be to have some way to give only my customers access to my home public server. Some kind of set up like a white list, so all they had to do was give me their IP, and I could add it to a config which would allow them to see my server -- ideally without a host entry. I like the idea of VPNs, but I don't want to require that level of tech-savvy from my customers.
Personally, I think that now-a-days VPNs are not as uncommon as you might think and are very easy to configure for the end client. My 60+Yr old mother had it for her corporate network. (Worst case you could TeamViewer in and setup for them). VPN's allow the most concrete firewall experience from your perspective, just forward the VPN ports and thats it on the Gateway
I did want to address the term "public". All IP's that are connected to the WAN (Internet) are "public". I assume you mean more like 'broadcast my home IP' as in anybody can ping freesoftwareservers.com and get my home IP. If you don't host public websites then only your clients should have your domain name and it won't show up in google etc so it shouldn't be considered "public".
If security is your main concern, I'd go that route and setup a domain name with "Dynamic DNS" updating for your home network. I doubt you can get a static IP from your ISP for a home ISP subscription. (I won't talk about the legalities of running a server at home, but lets say that I used to be with a major US ISP and never had a complaint). How I have DNS setup for my VPN is vpn.freesoftwareservers.com points to my home gateway. I then have a "Dynamic DNS" client that pings my domain name hoster every X Minutes and updates it if changed. I generally never have my home IP change unless I reboot the gateway. In this way you can setup your VPN certs to use that DNS name and know it should resolve.
- My Personal Thoughts I'd like to pass on
I started messing with servers while gaining my degree in computer networking and cyber security, so it seemed logical to do it "the hard way". The beauty of a home server is you have to learn EVERY STEP. Nobody does it for you. Also, you will likely want to learn about Hypervisors/VM's and that's a years worth of learning right there!
When I first started, I had 1 MB Upload, but this was mostly about me having access to my home network. Once I got 20MB Upload I was able to host moderate traffic on a simple txt based site without issues, but this can be hard to find. Also, I moved a few times since I setup my server which was inevitable downtime.
My Biggest Downfalls were
- Downtime when moving or rebooting
- NOT LEARNING CLOUD SERVICES
I want to be a sysadmin and I am way more knowledgeable on some matters, but lack familiarity with the GUI's of Major Cloud Vendors. But, once you understand what the GUI is doing, navigating a GUI is much easier vs click and test.
--Clearly I need to wrap this up, but this topic is dear to me!
In short, I think security wise you would be fine with a VPN if you are a SMB and could save money by doing things yourself. You also get cool hardware to play with, but you have to want to learn really bad. For me, it was instant love, you will either start and have 0 patience for learning/fixing problem after problem or find its fun and challenging. You can "Whitelist IP's" but your clients can't have DHCP IP's then, otherwise I'd use passwords + VPNs and re-issue every X Days.
There may be legality concerns with running a server on a home ISP. You could call and record a conversation where you ask point blank if it is ok. My site is non-profit so while its public, its not really money generating.
Can you handle a bit of downtime, because it will happen unless you do home + cloud.
If you really want to learn + have the best/cheapest setup I would have home server + cloud backup. The cloud servers can be charged based on bandwidth and usage. So if they are not being used because your home server is "up" then you won't get charged "much" for the cloud servers. This takes you into High Availability which is super fun to learn and crazy difficult! But I'm going to guess not needed, if you can plan your downtime as well, it helps clients.
That's it, If you do decide, I hope that you will take a look at my site, I think it has a lot of resources a home server enthusiasts could utilize, but know my site doesn't do much "teaching". Its more copy and paste-able code snippets that work for me.