I am working on setting up authentication into an Acme Packet Net-Net 3820 (SBC) via RADIUS. The accounting side of things is working just fine with no issues. The authentication side of things is another matter. I can see from a packet capture that the access-request messages are in fact getting to the RADIUS server at which point the RADIUS server starts communicating with the domain controllers. I then see the chain of communication going back to the RADIUS and then finally back to the SBC. The problem is the response I get back is always an access-reject message with a reason code of 16 (Authentication failed due to a user credentials mismatch. Either the user name provided does not match an existing user account or the password was incorrect). This is confirmed by looking at the security event logs where I can see events 4625 and 6273. See the events below (Note: The names and IPs have been changed to protect the innocent):
Event ID: 6273
Network Policy Server denied access to a user.
Contact the Network Policy Server administrator for more information.
User:
Security ID: NULL SID
Account Name: real_username
Account Domain: real_domain
Fully Qualified Account Name: real_domain\real_username
Client Machine:
Security ID: NULL SID
Account Name: -
Fully Qualified Account Name: -
OS-Version: -
Called Station Identifier: -
Calling Station Identifier: -
NAS:
NAS IPv4 Address: 10.0.0.10
NAS IPv6 Address: -
NAS Identifier: radius1.real_domain
NAS Port-Type: -
NAS Port: 101451540
RADIUS Client:
Client Friendly Name: sbc1mgmt
Client IP Address: 10.0.0.10
Authentication Details:
Connection Request Policy Name: SBC Authentication
Network Policy Name: -
Authentication Provider: Windows
Authentication Server: RADIUS1.real_domain
Authentication Type: MS-CHAPv2
EAP Type: -
Account Session Identifier: -
Logging Results: Accounting information was written to the SQL data store and the local log file.
Reason Code: 16
Reason: Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect.
Event ID: 4625
An account failed to log on.
Subject:
Security ID: SYSTEM
Account Name: RADIUS1$
Account Domain: REAL_DOMAIN
Logon ID: 0x3E7
Logon Type: 3
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: real_username
Account Domain: REAL_DOMAIN
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006D
Sub Status: 0xC000006A
Process Information:
Caller Process ID: 0x2cc
Caller Process Name: C:\Windows\System32\svchost.exe
Network Information:
Workstation Name:
Source Network Address: -
Source Port: -
Detailed Authentication Information:
Logon Process: IAS
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
This event is generated when a logon request fails. It is generated on the computer where access was attempted.
The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).
The Process Information fields indicate which account and process on the system requested the logon.
The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.
The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
So at first glance it would seem that the issue is merely a case of an invalid username or mismatched password. This is further confirmed in the packet capture where I can see the MSCHAPv2 response has an error code of 691 (Access denied because username or password, or both, are not valid on the domain). The thing is I know I am using a valid username and I have tried many usernames including new ones I created just for troubleshooting. I don't know how many times I have reset the password in an attempt to ensure it is not a mismatch password. I have even made sure to use passwords that are fairly short and contain only letters to ensure there was no terminal encoding issues (we connect to the SBC via SSH clients). I have also done this same thing with the shared secret used during communication between the SBC and the RADIUS server. I have tried prefixing the username with the domain name at login (though I don't think that should be necessary). I have also tried using the full UPN of the user to login. I have tried several RADIUS testing clients (NTRadPing, RadiusTest, etc.), but they either don't support MSCHAPv2 or only support EAP-MSCHAPv2. I have even created my own client using PHP's PECL RADIUS module. Still it always seems to fail with the MSCHAPv2 authentication with an error code of 691. Does anyone have any ideas as to why I always get an invalid username or bad password response when I have done everything possible to ensure that is not the case?
Here are the specs for our RADIUS configuration:
- Windows Server 2012 R2
- SQL Server 2012 Back End Database for accounting.
- The server has been authorized on the domain and is a member of the "RAS and IAS Servers" group. For which that group does have access to the accounts we are testing with.
- The accounts we are testing with do have the "Control access through NPS Network Policy" option checked under their "Dial-in" property tab.
- RADIUS clients configured to simply match on the IP address which you can see from the events above that it is applying the client friendly name.
- Connection Request Policy: The "SBC Authenication" policy is being applied as seen above. The only condition is a regex expression that does successfully match the friendly name.
- Network Policy: As seen in events above, none are getting applied. For troubleshooting purposes I have created a Network Policy that is set to "1" for the processing order and its only condition is a Day and Time Restriction currently set to any time, any day.
- The authentication method is set to only MSCHAPv2 or MSCHAPv2 (User can change password after it has expired). I have tried adding this to just the Network Policy and I have also tried adding this to the Connection Request Policy and setting it to override the authentication method of the Network Policy.
- We do have other RADIUS servers in our domain that use PEAP to authenticate wireless clients and they all work fine. However, we need this to work with MSCHAPv2 only (No EAP).
- All other configurations are set to the defaults.
The only other things of note to consider is the fact that in the events above you can see that the Security ID is "NULL SID". Now I know this is common especially among failed logons but given that this issue is stating an invalid username or bad password, perhaps it matters in this case. Also, this server has been rebuilt using the same computer account in Active Directory. I do not know if it would have worked before the rebuild. Essentially we built this server and only got as far as authorizing the server to the domain and adding SQL when we decided to separate out the SQL role onto another server. Rather than uninstalling SQL we just rebuilt the machine. However, before reinstalling Windows I did do a reset on the computer account. I don't think this should matter but thought I would point it out if there is some weird quirk where reusing the same SID of a previously authorized NPS server would cause an issue.
All in all it is a fairly basic setup and hopefully I have provided enough information for someone to get an idea of what might be going on. Apologies if my understanding of this seems a bit basic, after all, when it comes to RADIUS servers I guess you could say I'm the new guy here.
Edit 1:
In an attempt to further troubleshoot this issue I have tried bringing up additional servers for testing. Here are the additional tests I have performed.
Multiple Domains
I have now tried this in 3 different isolated domains. Both our test and production domains as well as my private home domain which has very little in the way of customizations aside from the modifications made for Exchange and ConfigMgr. All have the same results described above.
VPN Service
Using Windows Server 2012 R2 we brought up a separate server to run a standard VPN setup. The intent was to see if we could use RADIUS authentication with the VPN and if that worked we would know the issue is with the SBCs. However, before we could even configure it to use RADIUS we just attempted to make sure it worked with standard Windows Authentication on the local VPN server. Interestingly, it too fails with the same events getting logged as the RADIUS servers. The client machine being a Windows 8.1 workstation. Again I point out that we have working RADIUS servers used specifically for our wireless environment. The only difference between those RADIUS servers and the ones I am having problems with is that the working wireless servers are using PEAP instead of MSCHAPv2.
FreeRADIUS
Now I'm no Linux guru but I believe I have it up and running. I am able to use ntlm_auth to authenticate users when logged on to the console. However, when the radiusd service tries to use ntlm_auth to do essentially the same thing it fails and returns the same message I've been getting with the Windows server (E=691). I have the radiusd service running in debug mode so I can see more of what is going on. I can post the debug info I am getting if requested. The lines I am seeing of particular interest however is as follows:
(1) ERROR: mschap : Program returned code (1) and output 'Logon failure (0xc000006d)'
(1) mschap : External script failed.
(1) ERROR: mschap : External script says: Logon Failure (0xc000006d)
(1) ERROR: mschap : MS-CHAP2-Response is incorrect
The thing to note here is that while we are essentially still getting a "wrong password" message, the actual status code (0xc000006d) is slightly different than what I was getting on the Windows Servers which was (0xc000006a). From this document you can see what these codes mean: NTSTATUS values . The good thing about this FreeRADIUS server is that I can see all of the challenge responses when it is in debug mode. So if I can wrap my head around how a MSCHAPv2 response is computed I can compare it to see if this is simply a miscomputed challenge response. Update Was just noticing that the 6a code is just the sub-status code for the 6d code. So nothing different from the Windows Servers, I still wonder if there is a computation error with the challenge responses though.
Currently, I am working on bringing up a Windows Server 2008 R2 instance of a RADIUS server to see if that helps at all. However, I would be surprised if something with the service broke between W2K8 R2 and W2K12 R2 without anyone noticing until now. If this doesn't work I may have to open a case with Microsoft. Update: Same results with W2K8 R2.
Update 2
I opened a case with Microsoft and they worked on it for a couple of weeks. The first week I spent my time just trying to get them to understand this has nothing to with wireless and that the device we are trying to connect to does not support authentication against a RADIUS server using any form of EAP. Once they finally understood that we are trying to setup the authentication method as just MSCHAPv2 only, his initial reaction was simply "you can't do that". He said, that no matter what you always need some form of EAP or PEAP setup in the top box (even for PAP and other "Less secure methods" in that authentication method window). This seemed highly unlikely to me so I asked for some documentation stating this for which he then change the subject and never provided any documentation. It became clear I was getting nowhere with Microsoft when they told me that the issue seems to be with the username and password. So it took them 2 weeks to come to that conclusion when the subject line of my premiere ticket had the E691 error code in it which means mismatched username and password and despite the very lengthy paragraph of me explaining that I have done everything possible to ensure it is not a bad username and password.
Unfortunately my project lead does not want to waste anymore premiere support hours on this so we have closed the case. We will likely just deal with the SBC local shared login and setup some other way to enforce accountability (there is only 4 of us that will have access).
I will point out that before I was told to drop the project I did get the FreeRADIUS server to work using MSCHAPv2 only but was only able to do this using accounts local to the FreeRADIUS server. That is plain text passwords stored in a FreeRADIUS config file somewhere. So obviously not a solution but it at least shows that the SBC is correctly communicating to a RADIUS server via MSCHAPv2. This and the other things I have mentioned above lead me to believe that the issue lies between the RADIUS server and the Domain Controllers. While NTLM authentication works fine on both the Windows RADIUS and FreeRADIUS servers while logged into the servers locally (Can login to the Windows RADIUS via the test account and can get successful authentication on the FreeRADIUS server when using ntlm_auth command with just a username and password), neither RADIUS server seems to authenticate via users when coming in as a RADIUS access request (FreeRADIUS uses the same ntlm_auth command I use when logged in locally but instead of providing a username and password to the command it provides a username and challenge response).
So I will end this thread here but I will keep an eye on this and it will notify me if anyone posts something. If someone posts a solution or has comments I will respond.
