Connection handshake failed: Unexpected or badly formatted message.

Hello,

I am unable to send a message to a target service on a different SQL Server instance. Using the SQL Profiler, I get the following error from the target machine:

Broker::Connection: Connection handshake failed: An OS call failed: (80090326) 0x80090326(The message received was unexpected or badly formatted.). State 106.

On the initiator side, the transmission_status shows the following error:

An error occurred while receiving data: '64(The specified network name is no longer available.)'.

Any ideas on why the target thinks the message is "unexpected or badly formatted"? Btw, I am specifying the default message type and contract for the services.

Thanks.

The problem is in the NTLM/Kerberos authentification sequence and I reckon I never saw it before. If you only want to get this to work, you can change the endpoint authentication from WINDOWS to CERTIFICATE. If you want to see why is this failing in the first place, I suggest you follow up using the MS Connect site at https://connect.microsoft.com/SQLServer/Feedback?wa=wsignin1.0. A netmon capture of the handhsake sequence would be needed to investigate this.