You are very right about the limitations and possible increased risks of anti virus software. Not only is it the single biggest software component on a computer that contributes to slow I/O causing over all sluggishness, but the concept of building an increasingly huge database of all malware is flawed since it is always at best playing catch up after malware is released and discovered.
It should also be mentioned that antivirus software primarily scans I/O to prevent any malicious software from being transferred to or from permanent storage. Antivirus software does not scan network connections or other non file I/O sources of software exploits. An exploit can come in over a network connection, exploit some running software, and now have code executing on the system. This code can then disable the antivirus software and then proceed to install itself on the system.
Your web browser is an immensely large and complicated piece of software, and given its extensive use it is probably the biggest attack vector for users who don't download random software, even if plugins are not being used. Unfortunately there are really only four choices of code bases for a current standards compliant browsers. Microsoft's Internet Explorer, Microsoft's Edge, open source Mozilla or Gecko based browsers (Firefox and many others), and open source WebKit / Blink browsers (Konqueror, Safari, Chromium). Opera couldn't keep up with Javascript and switched to using Blink as their rendering engine.
If Javascript wasn't a big enough attack surface and security issue already, a new kind of Javascript called WebAssembly is being added to FireFox now. Consider disabling it in FireFox like this: go to about:config then set javascript.options.wasm = false.
Others have mentioned using a firewall or not enabling unnecessary system services, so I'll mention other things.
edit: There are different types of attacks. Some attacks are directed against a target, and others are generic attacks meant to target a large number of systems. Some security measures are excellent for protecting against generic attacks but do little to thwart a directed attack. Other methods protect against generic attacks and protect against typical directed attacks, but fail to protect against a direct attack from an attacker who is willing to analyze your specific configuration and spend the time to figure out how to attack it. As a general rule, smaller the code base and the greater the emphasis on security during application development will reduce the number of security vulnerabilities in a program. If a program is open source, and if it is popular, then a larger number of vulnerabilities will be discovered and eventually reported or made public. This reduces the over all number of security vulnerabilities in the program, but in most cases increases the risk of using the program since vulnerabilities are discovered and known by the public so much more frequently. On the other hand, the decreased number of vulnerabilities in the program due to its popularity and patching means that someone who deliberately searches the code to discover new vulnerabilities will have less success. In summary, if a vulnerability exists but isn't known by anyone, it remains harmless until it is discovered. So there are pros and cons to increased popularity and bug discovery in software.
Lying about application versions: This one is rarely talked about. Often, in order to successfully exploit a vulnerability, the exact version of the program and the operating system must be known. Unfortunately your web browser reports its exact version as well as the operating system that it is on every time you connect to a site. Consider changing the UserAgent in your browser and any other application that reveals too much information about itself.
Use your operating system's user level security. Run your browser in a limited user account that is not an administrator account and preferably different than your regular user account. This alone provides more security than any antivirus software. Keep in mind that how well this works depends on your operating system. On Windows, even a restricted program running with other windows on your session can monitor all keyboard input except for specialized full screen system password entry windows. Clients on *nix using the X Window system can monitor keyboard input as well.
Make sure DEP and ASLR are enabled. Windows may not enable DEP (non executable memory) for non Windows programs in order to prevent crashes due to compatibility problems. Enable DEP for everything and exempt crashing programs as needed. WehnTrust can be used to add ASLR to Windows NT5 versions (https://archive.codeplex.com/?p=wehntrust the installer buried in there).
Use an obscure operating system. Windows, OS X, and Linux have gotten rather popular. There are still alternatives like BSD and Solaris. If you configure your browser or other applications to lie about their operating system, an attacker may attempt to exploit your application and have it result in a crash instead of working. Edit: As is wrote above, it depends on the situation. Around 2004, the number of security vulnerabilities discovered in Linux vastly increased compared to BSD, and prior to this time the number of vulnerabilities discovered was similar. I believe this is due to the growing increase in popularity of Linux compared to BSD. Both BSD and Linux probably contain a large number of vulnerabilities, but BSD appears to be much more secure due to its lack of popularity resulting in far fewer vulnerabilities being publicly discovered. According to the DEF CON 25 - Ilja van Sprundel presentation, analyzing the BSD kernel source revealed a number of vulnerabilities. I still stand by what I said, that running an obscure operating system is more secure. However, if you are the target of a directed attack where someone is willing to spend a lot of their time analyzing your obscure configuration then you are less secure!
Don't overlook embedded systems! Your wifi chip has a CPU and firmware! Your antivirus software can do nothing to prevent embedded systems from being attacked by malware. Wifi chipsets have their own CPUs and firmware and they can be attacked remotely. In the summer of 2017 at Defcon they demonstrated a remote buffer overflow exploit in Broadcom wifi chipsets! The demonstration didn't go beyond changing a function call in the Broadcom firmware and making it send an "owned" packet out, but such an exploit allows complete takeover of the wifi chipset firmware. Broadcom is used in many smart phones and Apple products! Someone can DMA your system memory and send the data back using a channel outside of the normal wifi spectrum. They can also write to your RAM and install a root kit while bypassing everything. They could also write directly to your computer's system management mode memory. Since the SMM memory can be locked off by the chipset (except for when the CPU is in SMM) no memory scan can even detect it. It doesn't matter how many security programs you have or how many VMs your computer has. This is a direct attack to ring 0/-1/-2!!! There is no antivirus or other software out there that can detect such an attack. It is not much harder for someone who is familiar with development for an embedded system to write malware for it than it is on a regular computer operating system.
edit: I see a lot of negative response to my answer, but remember that the original question is about an alternative to anti-virus, and any knowledgeable IT security person knows how ineffective down right useless anti-virus can be at times. So some of my suggestions like using an obscure operating system are really doing the same thing that anti-virus does, which is making virus writers modify their malware to to get around the anti-virus software. I'm offering a somewhat ineffective security solutions in place of database based anti-virus software, which even less effective!