Here is the wording from official source:
The following steps present an outline of NTLM noninteractive
authentication. The first step provides the user's NTLM credentials
and occurs only as part of the interactive authentication (logon)
process.
- (Interactive authentication only) A user accesses a client computer and provides a domain name, user name, and password. The
client computes a cryptographic hash of the password and discards the
actual password.
- The client sends the user name to the server (in plaintext).
- The server generates a 16-byte random number, called a challenge or nonce, and sends it to the client.
- The client encrypts this challenge with the hash of the user's password and returns the result to the server. This is called the
response.
- The server sends the following three items to the domain controller:
- User name
- Challenge sent to the client
- Response received from the client
- The domain controller uses the user name to retrieve the hash of the user's password from the Security Account Manager database. It
uses this password hash to encrypt the challenge.
- The domain controller compares the encrypted challenge it computed (in step 6) to the response computed by the client (in step 4). If
they are identical, authentication is successful.
So the challenge is a server generated message that is encrypted with the hash of the account password by the client and by the DC and compared on DC.
Encryption methods are variable between versions of NTLM and different server settings.
Here's a bit from Wikipedia:
Both LMv2 and NTv2 hash the client and server challenge with the NT
hash of the user's password and other identifying information. The
exact formula is to begin with the NT Hash, which is stored in the SAM
or AD, and continue to hash in, using HMAC-MD5, the username and
domain name.