ISAKMP isn't a protocol as much as a framework for key exchanges (I know it has protocol in the name). Implementations of the framework include the Internet Key Exchange (IKE) and Kerberized Internet Negotiation of Keys (KINK).
If you read the ISAKMP RFC (RFC2408) it has a nice diagram for where ISAKMP sits in the network stack.
RFC2408 Section 2.2 ISAKMP Placement
+------------+ +--------+ +--------------+
! DOI ! ! ! ! Application !
! Definition ! <----> ! ISAKMP ! ! Process !
+------------+ --> ! ! !--------------!
+--------------+ ! +--------+ ! Appl Protocol!
! Key Exchange ! ! ^ ^ +--------------+
! Definition !<-- ! ! ^
+--------------+ ! ! !
! ! !
!----------------! ! !
v ! !
+-------+ v v
! API ! +---------------------------------------------+
+-------+ ! Socket Layer !
! !---------------------------------------------!
v ! Transport Protocol (TCP / UDP) !
+----------+ !---------------------------------------------!
! Security ! <----> ! IP !
! Protocol ! !---------------------------------------------!
+----------+ ! Link Layer Protocol !
+---------------------------------------------+
Figure 1: ISAKMP Relationships
This is a little bit ambiguous because it wants to protect the IP layer, but it must also perform negotiations between two IPs prior to encapsulation. However, in section 2.5.1 it states the following:
ISAKMP can be implemented over any transport protocol or over IP
itself. Implementations MUST include send and receive capability for
ISAKMP using the User Datagram Protocol (UDP) on port 500. UDP Port
500 has been assigned to ISAKMP by the Internet Assigned Numbers
Authority (IANA). Implementations MAY additionally support ISAKMP over
other transport protocols or over IP itself.
ISAKMP can be implemented at either Layer 3 (IP) or Layer 4 (Transport). Implementations must have support for transport layers, and optionally support an IP implementation.
IKE is one of the most common protocols used to negotiate IPSec keys. However both KINK (RFC4430 Section 9) and IKE use UDP packets to perform their exchange of keys between two peers.
While ISAKMP can occur at either IP or Transport it's most commonly implemented over Transport which would place it at Layer 5 (Session), but it could sit at Layer 4 (Transport) if it's implemented over Layer 2.