I have a Manjaro system running gitea server. Now when I tried to do something with the repos on the server, eg. cloning or pushing, using TortoiseGit or SourceTree, the server will ask for a password for the account gitea.

When I do the same operation using the command line client, it would work just fine.

Also I have tried SSH into the system using gitea account and it looks good I think ...

In the client machine, I have a private key loaded using pageant agent. TortoiseGit and SourceTree also have the key setup.
So what else I should do to get the git over ssh working?