Is PGP or S/MIME useful when both servers are self-controlled?

With regards to encryption and preventing a 3rd party from viewing my emails (and ignoring validity/signing), is encrypting an email via PGP or S/MIME useful when I control my server and my recipient controls their server?

I'm assuming the servers are not compromised and do not have other untrustworthy admins. And, for the sake of this argument, they are both physical servers, so there shouldn't be a problem of virtual machine vulnerabilities.

From my understanding, email between servers is encrypted, as long as they are both set up to support that. Are there other security concerns that are still valid that would be negated by email encryption?