There is a utility called tcptraceroute, and this enhancement called intrace that is used just like a standard traceroute, but it works over TCP.
How is the syn flag in TCP used to achieve traceroute like functionality (when ICMP is off)
What information can be disclosed (or other risks)?
How can this be mitigated? (routers, hosts, ...both?)
This has been described as similar to the nmap command when passed the -sS flag. If this is accurate, what does it actually mean?
All the tracerouting tools rely on the following principle: they send packets with a short life, and wait for ICMP packets reporting the death of these packets. An IP packet has a field called "TTL" (as "Time To Live") which is decremented at each hop; when it reaches 0, the packet dies, and the router on which this happens is supposed to send back a "Time Exceeded" ICMP message. That ICMP message contains the IP address of the said router, thus revealing it.
None of the tools you link to can do anything if some firewall blocks the "Time Exceeded" ICMP packets. However, blocking such packets tend to break the Internet (because hosts adaptively change the TTL in the packets they send in order to cope with long network paths, and they need these ICMP for this process), so, on a general basis, the "Time Exceeded" ICMP packets are not blocked.
What is often blocked, however, is the kind of short-lived packets that traceroute sends. These are the packets with the artificially low TTL. If they are blocked by a firewall, they never get to die "of old age", and thus no Time Exceeded ICMP. For TTL-processing and the "Time Exceeded" ICMP, the type of packet does not matter; this occurs at the IP level. But firewalls also look at packet contents. The goal is to fool firewalls so that they allow the short-lived packet to flow (and then die).
Plain traceroute uses either UDP packets, or ICMP "Echo" packets, both kinds being routinely blocked by (over)zealous sysadmins. tcptraceroute instead uses a TCP "SYN" packet, i.e. the kind of packet that would occur as first step in the TCP "three-way handshake". That kind of packet is not usually blocked by firewall, at least as long as the destination port is "allowed". tcptraceroute will not complete any TCP handshake; it just relies on the ideas that SYN packets are not shot on sight by firewalls.
intrace goes one step further in that it waits for an existing TCP connection (it does so by inspecting all packets, à la tcpdump). When it sees a connection, and the user presses ENTER, intrace will send short-live packets which appear as being part of the observed connection. intrace can do that because it has seen the packets, and so knows the IP addresses, ports and sequence numbers. All relevant firewalls will let these packets pass, since they (obviously) allow the observed TCP connection to proceed. The short-lived packets are adjusted so that they will not disrupt the TCP connection (i.e. they are simple "ACK" packets with no data by themselves, so the destination OS will simply ignore them).
Edit: I notice that I did not answer part of the question. Here it goes: there is no risk. There is nothing to mitigate.
traceroute reveals IP addresses of routers involved in routing packets. IP addresses are not meant to be secret and are rather easy to obtain for attackers through various means (mass scanning comes to mind, but also searching garbage bags for printouts of network maps -- the modern fashion of recycling makes dumpster diving a much easier and cleaner activity than what it used to be). However, a relatively widespread myth is that keeping your addresses secret somehow ensures security. Correspondingly, many sysadmins consider traceroute as a serious breach, to be fixed and blocked as soon as possible. In practice, though, this is all baloney. If revealing a few internal IP addresses is a major issue, then this means that your network is doomed.
Worrying about secrecy of IP addresses is like triggering a major incident response plan because an outsider learned the menu at the company's cafeteria. It is disproportionate. Granted, having precise and extensive knowledge of the network infrastructure can only help attackers; but not in really significant amounts. Keeping IP addresses secret is not worth breaking connectivity through excessive filtering (for instance, blocking the "fragmentation required" ICMP is deadly for any client behind an ADSL+PPPoE link).
Both traceroute and tcptraceroute work on the same basic principle:
The only difference is what type of packet is sent out (default is UDP on Linux, ICMP on Windows, and TCP is becoming a more popular option). All will generate ICMP error messages from helpful devices in between when the TTL expires.
SYN flag is used because intervening devices like firewalls will often permit SYN but block other TCP packets that aren't part of an established connection.nmap -sS will perform a SYN scan, where only SYN packets are sent. However, it won't perform a traceroute in and of itself; you need to combine it with the --traceroute option (and then I think you're still scanning, not just performing a traceroute, so if you want to trace I recommend tcptraceroute or traceroute -T if your OS supports it).
- How is the syn flag in TCP used to achieve traceroute like functionality (when ICMP is off)
Any IP packet will trigger an ICMP "Time Exceeded" response from an intermediate router when the TTL value reaches zero. This could be TCP, UDP, ICMP, or any other. As long as intervening routers will pass the packet, the trace will work. Traditional traceroute uses ICMP echo requests, but this is just by convention.
Note that whether or not ICMP is allowed on the destination server is entirely irrelevant as it is the intermediate routers that are returning the Time Exceeded responses. The only thing you might miss is a response from the final destination host, but with traceroute that is the least important response, since what traceroute provides you is the list of intermediate routers.
Using a SYN flag on the TCP packet will potentially get you a SYN/ACK from the destination server, or perhaps not, if the port isn't open or a policy blocks your packet. But again, the destination server is the least important. You could just as easily send a UDP or a TCP continuation if all you were interested in was the hop list.
Some routers are configured to treat PING packets differently; different route, different priority, etc. Using TCP packets instead may give you different results which should be more indicative of the route TCP traffic would take.
- What information can be disclosed (or other risks)?
Nothing that isn't public. If your security depends on keeping this sort of information secret, then you should sit down and reconsider the decisions that got you here.
- How can this be mitigated? (routers, hosts, ...both?)
There is nothing to mitigate. Of course, you could always unplug your server, that'll mitigate all sorts of stuff. Otherwise, before you employ a mitigation strategy, you should consider what you're gaining and what you're giving up. In this case, it doesn't appear you have anything to gain.
