Use custom SSH authentication mechanism (OpenSSH)

Question

I have a collection of servers (Debian/Ubuntu) for which I need to create a custom ssh authentication mechanism.

I found a couple of 'same-ish' examples for reference with Duo and Authy. They are using their own API to do the authentication but the idea of delegating the ssh authentication process to a separate application is what I'm trying to achieve.

Ideally I would like to write the authentication program in C++ but I'm happy to script it if thats what is needed.

Answer 1

tl;dr version: If you want to implement your own way of checking the entered password, write a PAM module that has a pam_sm_authenticate() function, compile it as , and add it to the server's /etc/pam.d/sshd.

See the Linux-PAM Module Writer's Guide as well as various other sources. Start with something like this:

#define PAM_SM_AUTH

#include <security/pam_modules.h>
#include <security/pam_appl.h>

int pam_sm_authenticate(pam_handle_t *pamh, int flags, int argc, const char *argv[]) {
    return PAM_SUCCESS;
}

struct pam_module pam_foobar_modstruct = {
    "pam_foobar",
    pam_sm_authenticate,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
};

[You can use pamtester to verify the module locally, before messing with sshd's configuration.]

---

"trying to answer the 'real' question" version: That depends on what you really mean by a "custom authentication mechanism".

Usually the word "mechanism" is used by protocols like SASL or GSSAPI to describe the entire 'dialog' between a client and a server – whether it uses a password, or a keypair; whether the password is hashed, or encrypted, or sent in plain; whether the mechanism only proves the client's identity, or also the server's; and so on. In SSH, the same is called an "authentication method".

The SSH authentication layer has several "top-level" authentication methods. The most common ones are password (which only sends the raw password to the server), public-key (which uses your SSH id_rsa keypair to sign a challenge from the server), and gssapi-with-mic (which calls into a second authentication layer – GSSAPI – and almost always sends a Kerberos ticket).

These authentication methods are implemented directly within OpenSSH and you'd have to edit its source code (both the client and server) to implement one from scratch. However, some of them are possible to extend, as they already call into external libraries:

• With the password method, the client sends the raw password, and the server calls the PAM library to validate it. PAM then uses modules such as pam_unix (check against /etc/passwd + /etc/shadow), pam_radius (check against a RADIUS server), and so on. These are configured through /etc/pam.d.

• With the gssapi-with-mic method, both the client and server call the GSSAPI library, which usually has Kerberos as a built-in GSSAPI mechanism and allows adding custom ones; on Linux, it's configurable through either /etc/gss/mech or /etc/gssapi_mech.conf files.

The hierarchy is, basically:

• SSH connection (sshd)
  • password method – one plain password prompt
    • check received password against /etc/shadow
    • check received password against a Kerberos KDC ("fake Kerberos")
    • give received password to PAM (libpam), which then calls the configured module stack:
      • pam_unix for checking against /etc/shadow
      • pam_radius for RADIUS
      • pam_krb5 for "fake" Kerberos
      • pam_opie for one-time passwords
      • etc.
  • publickey method – send a challenge, have the client sign it with the user's key
  • keyboard-interactive method – any number of question-answer prompts
    • interact with PAM
      • same PAM modules as above, but can have multiple prompts
    • interact with bsdauth
    • etc.
  • gssapi-with-mic method – call the GSSAPI library; shuffle messages between library and client
    • Kerberos (99.9% of the time) – verify the ticket sent by client
    • NTLM, SPKM3, GSS-BrowserID, etc. (rare but possible)
  • hostbased method – send a challenge, have the client sign it with the host key