SMTP error 5.7.11 Encryption required for requested authentication mechanism

What SMTP error 5.7.11 means

A narrower and now largely historical cousin of 5.7.10: the specific mechanism chosen may only be used once the connection is encrypted. The defining RFC records it for historical purposes, because a modern implementation simply does not advertise a mechanism it is not willing to accept, so a client never selects one and never receives this reply. Meeting 524 5.7.11 or 538 5.7.11 in the wild means you are almost certainly talking to an ageing on-premise server or an appliance with a dated SMTP stack.

Security or policy status
Sent by A submission server still advertising a mechanism it will not accept unencrypted
Rare

What SMTP error 5.7.11 Encryption required for requested authentication mechanism actually means

Two behaviours are possible when a server offers a mechanism it will not honour without encryption. The old behaviour was to list it anyway and refuse when it was chosen, which is what this code reports. The current behaviour is to omit it from the capability list entirely until STARTTLS has completed, so the situation cannot arise. Because the modern approach won, the appearance of this code is mostly interesting as evidence about the software at the far end rather than as a problem in its own right. The overlap with 5.7.10 is real and the two get used interchangeably by some implementations; if you are troubleshooting, the practical remedy is identical and lies in getting TLS established before authentication is attempted.

This response to the AUTH command indicates that the selected authentication mechanism may only be used when the underlying SMTP connection is encrypted. Note that this response code is documented here for historical purposes only. Modern implementations SHOULD NOT advertise mechanisms that are not permitted due to lack of encryption, unless an encryption layer of sufficient strength is currently being employed.
— RFC 4954

How 5.7.11 appears in a bounce

A server reporting this condition sends it alongside a three-digit reply code, and a bounce prints the two together. The registry lists these pairings for this status: 524 5.7.11, 538 5.7.11. Which one you get depends on the point in the conversation at which the server refused.

The leading 5 is the server's verdict rather than part of the code's identity: it marks this as a permanent failure, which means the sending server has given up and returned the message.

What causes SMTP error 5.7.11 on a business phone system

An on-premise server well past its support dateLegacy Exchange installations kept alive for one application, and Unix mail transfer agents that nobody has touched since the person who built them left, both behave this way. The code is a reliable indicator that the host has not been reviewed in a long time.
An appliance with an embedded mail stackFirewalls, backup servers, storage arrays and monitoring boxes carry their own SMTP implementations, often derived from code that predates the current guidance. They interoperate fine until they are asked to talk to something modern.
A client that always requests the same mechanismAn application choosing its method up front rather than reading the capability list will keep asking for the one it knows, encrypted or not. The server then has to refuse it, and this is the refusal.
A test or laboratory relay reached by accidentDevelopment environments frequently include a permissive relay built years ago. A configuration copied from test into production, or a hostname that never got changed, sends real traffic at it and produces unusual codes like this one.

How to fix SMTP error 5.7.11

Look at the capability list before and after STARTTLSConnect and issue EHLO, start TLS, then issue EHLO again. Comparing the two responses tells you exactly what the server is willing to offer in each state and removes all the guesswork from this class of fault.
Let the client pick from what is advertisedRemove any hard-coded mechanism from the application or device configuration. Nearly every mail library negotiates sensibly when it is allowed to, and the ones that do not are usually configurable.
Establish encryption first and the objection disappearsWhether the server meant this code or 5.7.10, getting TLS up before AUTH resolves both. Configure the client for explicit TLS on the submission port and retest.
Question whether the legacy relay should still existAn unmanaged host quietly accepting mail on your network is a liability as much as an inconvenience. Migrate the application onto authenticated submission through your tenant, or onto a managed relay you can patch, and decommission the old one properly.

Fixing the underlying problem

This page explains the code. These guides walk through the fix in detail.

Questions about SMTP error 5.7.11

Why would we see a code the RFC describes as historical?

Because the software sending it is historical. The guidance changed but a great deal of deployed code did not, particularly inside appliances where the mail stack was written once and never revisited. Its appearance tells you something useful about the age of the far end.

How do we tell this apart from 5.7.10 in practice?

For troubleshooting purposes you largely do not need to. Both say encryption must come before authentication, and both are fixed by establishing TLS first. The distinction is that this one is tied to a specific mechanism being selected, whilst 5.7.10 is the broader statement that a privacy layer is needed.

Did our credentials go across the network unprotected?

No. The server refused the mechanism at the point it was selected, which is before any password is exchanged. The connection was unencrypted, but nothing sensitive had been put on it.

Related SMTP status codes

Source. SMTP error 5.7.11 Encryption required for requested authentication mechanism is defined in RFC 4954 and registered in the IANA SMTP Enhanced Status Codes registry. The causes and fixes above are drawn from our own experience supporting UK business email systems.

All Email delivery & SMTP codes