Nmap's -D option stands for decoy, which means that the attacker can simulate that the attack is coming from multiple IPs, including attacker's IP.
From the point of view of the victim, is it possible to identify the real IP and then trace back the attacker?
In a nmap -D scan, for scanning to return any result, your real IP address must be used amongst the pool of decoys. Without your real IP being used, you will not be able to receive any response from your target server, and nmap scanning would not work.
The -D option creates confusion by introducing decoy IP addresses. So, if the target server is logging incoming connections, it will see an assortment of spoofed IP addresses plus your real IP address.
If you are not careful to ensure that the decoys are up (no response after initial SYN on active port that returns SYN-ACK) or if your ISP filter spoofed IP addresses (only your IP will appear on the server log), it would not be difficult to spot the actual IP address doing the scan.
Here are the relevant parts from the man page of nmap,
-D decoy1[,decoy2][,ME][,...] (Cloak a scan with decoys) .
Causes a decoy scan to be performed, which makes it appear to the remote host that the host(s) you specify as decoys are scanning the target network too. Thus their IDS might report 5-10 port scans from unique IP addresses, but they won't know which IP was scanning them and which were innocent decoys. While this can be defeated through router path tracing, response-dropping, and other active mechanisms, it is generally an effective technique for hiding your IP address....Note that the hosts you use as decoys should be up or you might accidentally SYN flood your targets. Also it will be pretty easy to determine which host is scanning if only one is actually up on the network....
Decoys are used both in the initial ping scan (using ICMP, SYN, ACK, or whatever) and during the actual port scanning phase. Decoys are also used during remote OS detection (-O). Decoys do not work with version detection or TCP connect scan....
... Also, some ISPs will filter out your spoofed packets, but many do not restrict spoofed IP packets at all.
One approach is to analyse the TTL field on the packets.
Time to live is an IP feature to mitigate routing loops. Every packet starts with a certain TTL value, usually 64, and each routing hop reduces the TTL by one. If the TTL gets to zero then the packet is dropped, and an ICMP "TTL expired in transit" message is returned. The traceroute utility uses the TTL field. It first sends a packet with TTL=1 then TTL=2 and so on. Then it tracks the ICMP responses to determine the trace to the target IP address.
To assess if a particular packet is from a decoy, you can use traceroute to measure the routing distance to that IP address. If you add that to the TTL you received, you get the initial TTL of the packet. It turns out that there are only certain initial TTL values that network stacks normally produce. If your calculated TTL is something else, it implies the packet actually travelled a different route, and it is a decoy. This technique is not perfect, because routes can change, and any initial TTL is technically legal. But it is a reasonable rough-and-ready way to detect decoys.
Of course, you can only do this if you are logging the received TTLs, which does not happen by default. You can configure iptables to do this.
