SSL with client certificate authentication

can somebody explain to me if for example i use Server certificate, with 2048bit keys and client certificate with only 512bit strength, does this use in any way affect the security of the established SSL?

Why i'm asking is because using client certificate requires RSA decryption in ssl handshake and on embedded ARM5 devices using 2048 client cert negotiation takes 7 sec :S

So using a lesser client cert 512 is significantly faster...

Thank you and best regards, Lonko