Realistically, how much added security does having a passphrase on a ssh key have?

Let's assume ssh key brute force is unrealistic.

It seems to me your greatest vulnerability would be someone gaining access to a client filesystem. If that's the case then key loggers and a host of other nasties are possible, making a passphrase a minor obstacle.

Is key bruteforce actually unrealistic? If key bruteforce is realistic, isn't it reasonable to assume a passphrase is also bruteforcable?