I am oddly qualified to answer your questions, because I wrote a toy AV as my final year project in university. Mine used a kernel driver for process and thread creation notifications, but the majority of the work was done by a userspace service running as SYSTEM.
The first conclusion is that the AV must use some form of Inter-Process Communication (IPC) between its hook (in the remote process app.exe) and its own process/service.
How does it happen in "real world" products? [There are many IPC options, each with its own pros & cons... of course there are many "clients" --- many processes to monitor at the same time in the system]
There are a few approaches here. The two main ones are named pipes, or sections and events. In the case of pipes, your AV service can open a named pipe server and each process opens a handle to it so that it can send messages to it. This allows all messages to come in over a pipe and the server can process them all in one place. The one downside is that pipes can be a bit finnicky and stream-based message parsing can get a bit complicated.
The other way to go is to have a shared memory section (using CreateFileMapping, MapViewOfFile and associated APIs). These APIs are often used to map a filesystem file into virtual memory, but you can also created a named section that is backed by physical memory rather than a file on disk. You can then use a named event (e.g. with the CreateEvent API) and use that to signal (via SetEvent) that the service should read the section because a new message is ready. The service can then use ResetEvent to clear the event. Both processes can use WaitForSingleObject to handle synchronisation. The benefit here is that you get multi-threading support, and the message parsing is easier. The more complicated aspect is that you need a section and an event for each process that the hook is injected into, and those names need to be unique. The way I approached this was to have each named object to be something along the lines of AV_sharemem_proc%d formatted with the process ID. The service could then enumerate processes (or, in my case, use the process creation events from the driver) in order to try to open these sections and events.
How does it affect the performance of a system with real-time protection enabled?
I mean, once a hook is triggered, app.exe's code must be suspended, and the results from the AV's "brain" must be received, dictating if a normal execution of app.exe should continue or not.
Yes, it affects performance. You get a context switch to the service process every time an API call has to be monitored. This can introduce latency while the thread scheduler goes through and gives each process its timeslice, although the scheduler is usually smart enough to switch context directly to a waiting process when a synchronisation event occurs. You can improve latency by setting the service process priority to high so that it more frequently gets given execution time. Don't use realtime priority though, as it is very liable to cause system instability.
Is the communication safe? (encrypted? Can a virus 'fake' AV's internal messages? etc.)
All AVs have to assume IPC may be compromised in some limited way, but a usermode hooking model is inherently not secure. Malware that expects your particular AV to be installed can simply break your injected hook, or take over it and send back "everything's ok". This is why most AVs have a driver; that way they can keep the hooking and monitoring stuff out of the process, where malware has complete control. The driver can also process some of the hooking stuff in the kernel without having to bounce each event back to the usermode service, which improves performance a lot.
Of course a sophisticated AV should maintain a 'State' of execution during run-time of app.exe, and make actions based on the State. E.g. ransomware being 'caught' by the use of these APIs in a loop: file access, crypto, write. So each API is benign on its own, but the combination (maintained State) is malicious.
How a real-time AV can stop a running program like this using simple user-space hooks? (Since it got its running "time share", in the remote app.exe only during API hooks... there isn't a different thread in the context of the remote process for the purpose of communication with the other AV processes, isn't it?)
This is mostly an implementation detail, but you need thread-safe hooks in the first place because the process might be making multiple calls across different threads at once. The way I'd approach it is to have a queue that contains each of the API calls that need to be sent via IPC to the service, then use an SRW lock so that the queue can be peeked by multiple threads but only written to (or dequeued) by one thread at once. Then you can have a dedicated thread for monitoring the queue and dequeuing items to send via IPC to the service. Since the scheduler operates on threads not processes it leads to less blocking than trying to do everything in a single thread.
Please answer my question and perhaps elaborate more on the subjects. Thank you! :-)
Make sure you set an appropriate DACL on system objects like pipes, sections, events, mutexes, SRWs, etc. - for global objects in your model (e.g. a pipe server on your service) you'll want to set it to allow read/write/synchronise for users, and full control by SYSTEM. For per-process stuff you should set the DACL to only allow access from the process' user context (i.e. the user that the process is running as) and SYSTEM. If you fail to do this, you'll end up with potential cross-user interference. If you end up adding a driver, you should definitely set a restrictive DACL that only allows SYSTEM to open handles to the driver object and any IPC objects used.
Using a driver is very similar to the IPC scenario I already described, except you generally send messages to the driver using IOCTLs or by performing other operations on the driver object.
While it's a bit outdated (focused on Windows Vista) I do strongly recommend having a read of Bill Blunden's book "The Rootkit Arsenal", as it covers a lot of the techniques you would use when writing usermode and kernel hooks.
