OpenID Connect: Why use authorization code flow?

I have a question about implementing OpenID Connect I was hoping I could get some help on. I understand the different flows and get that the authorization code flow is good because it allow client credentials and server-to-server communication is more secure than communication through the user agent. But even so using the intermediary code seems unnecessary if the client and OP were to use authenticated encryption on their correspondence through the user agent. Groundhog Day attacks could either be restricted through the use of a very short expiration time specified within the message or eliminated with a preliminary request to the OP for a nonce. Assuming a strong enough encryption, i.e. AES-256, I would think that even refresh tokens could be sent through the user agent with negligible decrease in security. Are there any other considerations or reasons why this could/should not be done that I may have overlooked?