TACACS+ requires that a pre-shared key (length of the key is restricted to 63 characters) is configured. This is how the device authenticates to the server. This is not 'challenge-response'.
This pre-shared key is used to set up encryption that encrypts the whole packet, which means that usernames and passwords are protected from the start.
How it is encrypted is explained in the RFC in the section "Body Encryption". It's a version of the One Time Pad.
In this case, the packet body is encrypted by XOR-ing it byte-wise
with a pseudo random pad.
ENCRYPTED {data} == data ^ pseudo_pad
The pad is generated by concatenating a series of MD5 hashes (each
16 bytes long) and truncating it to the length of the input data.
Whenever used in this document, MD5 refers to the "RSA Data
Security, Inc. MD5 Message-Digest Algorithm" as specified in [3].
pseudo_pad = {MD5_1 [,MD5_2 [ ... ,MD5_n]]} truncated to len(data)
The first MD5 hash is generated by concatenating the session_id,
the secret key, the version number and the sequence number and then
run- ning MD5 over that stream. All of those input values are
available in the packet header, except for the secret key which is
a shared secret between the TACACS+ client and daemon.
The version number is the one byte combination of the major and
minor version numbers.
The session id is used in the byte order in which it appears in the
TACACS+ header. (i.e. in network byte order, not host byte order).
Subsequent hashes are generated by using the same input stream, but
concatenating the previous hash value at the end of the input stream.
MD5_1 = MD5{session_id, key, version, seq_no}
MD5_2 = MD5{session_id, key, version, seq_no, MD5_1}
....
MD5_n = MD5{session_id, key, version, seq_no, MD5_n-1}
User authentication process is described here:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/CLIConfigurationGuide/sec_tacacsplus.html
Update 01.03.2018:
It seems that Cisco has updated TACACS+ and users are now able to encrypt passwords with AES128 when using newer high performance switches. So far this only encrypts passwords. It does not change the encryption of the payload of the TACACS+ protocol. But it's something. Note though that not all products can make use of this.