Some password managers expose SSH agent socket, so you can do that instead of fumbling with files.