There are following problems with this approach:
The passwords with formulas seem to be stored in plaintext. So this would be better if it's split into two parts - first part known, and second part being formula. This way the first part can be stored as hash.
Similar mechanism is used by crypto tokens, where you enter password followed by number from crypto token (like "password" + "123456" which is "password123456"). Crypto token is small hardware device which shows number after pressing button. Every crypto token hardware has different key so everyone needs to use their own token. Now the thing is, that the problem is that formula is very easy to reverse if it's very simple. In fact, it should be some sort of strong encryption which person cannot perform in memory.
Anyway, you seem to be on good track and what you are trying to do can be achieved in following way for example:
Create software crypto token application, so this app will simply agree the secret key with your server during installation and will be able to generate crypto tokens in offline (or even online) mode.
Your website can read the password and remove last 6 numbers, then check the password with stored hash. (the 6 numbers change very 10 secs or minute or so, so the time synchronization is important, but if it's none, it's possible to synchronize by checking past / future tokens and do the adjustments).
Then your website can contact API and check if the crypto token for given username matches.
Now the thing to know is that crypto tokens work on time and secret key. So that token generated in one minute isn't valid in different time.
However the main challenge would be to protect the security of your crypto tokens. You could provision dedicated cloud server for each customer using it for example, permitting access only from his servers and maintain extreme update schedule for security patches, multi-layered defense and good software architecture where all libraries are actively maintained and the source code is audited.
Such solution would be eventually good not only for websites but for any corporate system including Active Directory. However to sell it to customers you'd need to comply with official security policies once and secondly have it reviewed by educational / government institutions which I am sure will be helpful.
The overall proposed scenario is good because on your side you do not risk much really - if crypto tokens are stolen, then the passwords are not known anyway, and you can issue update to the soft crypto token or force reactivation of these.
The downside is that it requires dedicated soft crypto token to get onto some single service which would be using it. So the hardware crypto token is a lot more handy, but the soft one is still good for demoing and development.
Regarding "enter letter 1, 3 and 5 from your second password" is somewhat weaker but a lot cheaper compared to tokens. However to produce token is not hard today given it's long life and a lot stronger security.
The business model of this is based on maintaining both secure method of communication and proper secure storage of crypto tokens, so the whole business on your side has to properly focus on it, so it's job for 3rd party companies and not banks themselves. This way attack surface is reduced the way that the tokens are stored and verified by 3rd party company.