Do you mind if somebody takes notes of the conversation you're having with them?

It would certainly change the way most people will interact with them.

At what point do the notes become too detailed?

It doesn't have to be a binary and can vary for different people and situations.

Do you mind the fact that all modern smartphones technically record all calls?

You cannot retrieve the "recording" of a call, so this is not really meaningful in any relevant way.

How's an AI "recording" your conversation in memory, transcribing it and writing a summary of that transcript, different from a phone doing the recording to RAM, and a human brain doing the transcribing and writing?

Because it is doing it all the time. "Quantity has a quality of its own"