Here's what worked for me solving this login issue on OS X 10.11.6 with Server 5.1.7. These steps also resolved several other problems like not being able to install profiles due to trust issues, user permissions issues, or the inability to contact the server, etc.
From https://support.apple.com/en-us/HT200018 :
Check DNS:
Open Terminal and type hostname to verify the name of your server.
Open Server.app and select your server in the sidebar.
Select the Overview tab. Your server's fully qualified domain name should be displayed in the Host Name field.
If the qualified domain name in the Server window doesn't match the results of the hostname command, you need to correct this before
proceeding with the next steps in this article. If you are certain the
Fully Qualified Domain Name assigned to your server's IP address is
correct, use the change hostname assistant in Server to set the
correct hostname for your server.
Open System Preferences and click the Network icon.
Select the network interface that has been configured for your server. Your server's IP address is listed here.
Note the DNS Servers listed.
If you have configured the DNS server on your server and DNS
records have been created for this server, your server should be
listed as 127.0.0.1 (not the server's IP address).
If another server on your network is hosting DNS records for your
server, that server's IP Address should be listed in the DNS Servers
field.
If this information is not correct, click the “Advanced…” button
and then the DNS tab. The correct DNS server address(es) can be set
here.
Testing your DNS settings
You can use the Terminal to test your name to IP address resolution. Open Terminal (/Applications/Utilities/Terminal.app) and use the following commands:
Use the “host” command to test name to ip address resolution:
host <your server's fully qualified domain name>
The expected output is has
address You can also use the "host" command to test
IP address to name resolution:
host <your ip address>
The expected output is .in-addr.arpa domain name pointer
In addition to this, Default Network Access should be set to All Networks and I am not sure if it helps, but turning on Apple Push Notifications in Settings was another step I took (MDM relies on Push to a degree).
Further, if you are hosting at a .local domain name, it is also important that the certificate assigned to your server be signed by the Open Directory Certificate Authority on your computer (otherwise of course you'd want to use one signed by an actual certificate authority). OS X server version 5 has some annoying bugs when it comes to how it automatically assigns certificates to various services including websites. What I had to do was manually create a new certificate like this, prior to the rekerberize step below, I did the following steps to reset certificates for the system:
(Optionally) After freshly backing up your system, start fresh by turning off all server services and deleting the Master from Open Directory. Then go to Certificates and delete any certificates except for special ones for any custom websites. Also go to /etc/certificates and make sure there are not any rogue certificates there that do not show up in Server app (if there are, move to a temp folder or delete).
Next, in OS X Server app, if you did not already, then TURN OFF ALL SERVICES. Then go to OS X Server app and do Certificates > + > Create a Certificate Identity...
For the name, I put my hostname, but you can put anything here. Identity type: Leaf. Certificate type: SSL Server. Check "let me override defaults." Click "Continue."
For Serial Number, pick something you haven't already used. Click continue.
For Name (Common Name) make sure it matches your hostname. The rest does not matter. Press Continue.
For Certificate Authority, select the one that includes the text, "Open Directory Certificate Authority", which should have been created when you first set up Open Directory. Press Continue.
Set the bit size of the cert (2048 RSA is fine).
In Key Usage Extension, make sure to select Key Encipherment and Key Agreement. Press Continue.
In Key Usage Extension, make sure to select SSL Client Authentication and PKINIT Client Authentication (this may not matter but I did see an error in the iPad log that indicated the client needed a cert and the time I reset everything and checked this, it started working.) Press continue.
Go past Basic Constraint Extension. Press continue there.
In Subject Alternate Name Extension, make sure the dDNSName is set to your hostname verified in the Check DNS step from Apple above. Make sure the IP Address includes the actual IP of your server. Press continue.
Authenticate and press OK. Now the new cert should show up in OS X server's Certificates area. Select Secure services using: > Custom. In the Server Certificates box that pops up, set the certificate for ALL of your services (except any custom websites that need their own cert) to your new cert.
(Optional.) If you deleted your Open Directory master in step 0, then make a new one here. At least make sure Open Directory has a locale with your local IP range, e.g. 10.1.10.0/24 etc.
Next do the "rekerberize" step below, from Apple:
Rekerberize your server:
sudo mkdir /var/db/openldap/migration
sudo touch /var/db/openldap/migration/.rekerberize
sudo slapconfig -firstboot
The last terminal command above will have turned on the Open Directory server. Go back to server app and make sure Open Directory is on. Then do the following steps (first few steps I believe are just to ensure it can work with a .local server):
Make sure your server's DNS server is configured with an entry for your machine's hostname, and set to forward all other DNS requests to a normal DNS server. (This is especially important if you're using a .local domain for your machine.) Add the entry for your hostname and your IP, then turn on the DNS server.
Now turn on Websites. If you're still in your testing phase and your server is still on a .local domain name, then Double-click on Server Website (SSL) and verify its SSL certificate is set up as the new one that we created above.
Turn on Profile Manager. Set it to Sign configuration profiles. The cert shown ought to say it's been signed by an intermediary authority.
In Users, create a user to use with Profile Manager. All users used with Profile Manager MUST be in the "Local Directory" and NOT in the "Local Network Directory"... this is VERY important... ! The reason is because when you log out from OS X Server (even the Profile Manager) it may cause the un-mounting of the volume containing that user's data, preventing further logins by them. See this Apple support site: https://support.apple.com/en-us/HT203325
In Safari, login to your server's Profile Manager. If you are intending to have devices auto-setup from the Welcome screen, you'll need to check the box in the Everyone group for "Allow enrollment during Setup Assistant for devices configured using Apple Configurator". (See this Spiceworks post & replies.). If this is turned off then the user's login will fail without explanation during the Welcome screen process on a "prepared" device.
Now, if you're using a .local server, then to get an iPad to download and install profiles you'll need to do the following steps:
Turn off the iPad's Cellular Data and go to WiFi and make sure it's on the same subnet as your server.
If the iPad is already set up past the Welcome screen, set the iPad's DNS server to exactly match the hostname determined in the Check DNS steps above (e.g., mycomputer.local).
OR
If the iPad is on the Welcome screen and it is a Supervised device set up with Configurator to download the profile at setup time from your .local MDM server, then you'll need to turn on Internet Sharing on the server to create a hotspot, and use that for the initial WIFI connection. Note that this requires the server to have its DNS server setup properly as described in the Check DNS section quoted from Apple's support site above.
Please note:
• Technically speaking, a .local address is a fully qualified domain name. If your clients are on the same LAN and subnet as the server then things will work if the server has a .local address as long as Apple Push Notification service is activated on the server. If you set up Profile Manager and then later turn off APN in OS X Server, then you won't be prompted to re-enable it and things will start failing.
• In the device logs if you get the following type of error:
Desc : The server certificate for “https://my-macbook-pro.local/devicemanagement/api/device/mdm_checkin” is invalid.
Then it's because your iPad's not trusting the certificate. This could mean one of several things:
It could mean a certificate issue. Make sure profile signing is turned on in Profile Manager in OS X Server app. If it's not, and you don't want it on, then you may need to install and trust the root certificate onto your device by either e-mailing it to the Device, or putting it up on a website and then downloading it from that website onto the device. BTW, "the root certificate" means the one that signed the intermediary root that signed the code signing profile used to sign the Remote Management Profile; you can see which cert signed which by going into Keychain Access and looking at the details of the certificates (roots are not accessible from within OS X Server app itself as far as I'm aware). (You can install a certificate onto an iOS device by exporting it from Keychain Access as a .cer file, then placing it in /Library/Server/Web/Data/Sites/Default/, then sudo chown _www:_www CertName.cer, then on the iPad, navigate to myserver.local/CertName.cer.) However I did not need to do this if I just signed the cert. If all that fails then repeat all the above steps because you probably have the wrong IP or hostname or settings on the certs in question, or Kerberos needs a reset, etc.
It could mean your client does not have the server's hostname as its DNS server. Either set it to have it, or set it to use the server's Internet Sharing hotspot, both described above.
This Apple Discussions thread, particularly Linc Davis's post, was also very helpful: https://discussions.apple.com/message/30689429#30689429
