This is one of the issues I have faced lately where in users used to get error (mentioned in subject) when they tried to browse the SharePoint site hosted over https. In my case we had a two server farm (app and WFE). upon initial investigation we found that whenever the load balancer was forwarding request to WFE, user used to get TSL error, Pages displayed properly when the request was forwarded to app server. Strange things was the TSL protocols were already enabled in user's internet explorer.
IIS settings were same on both server (when I have checked/compared with IISCrypto tool). user request did not even reached the IIS or SharePoint layer as I could not see any log entries in either IIS or SharePoint logs. from IIS console prospective its had same SSL certificate and all settings just looked fine.
Decided to use WireShark Tool to see the network traffic and to check is the request were reaching server or not. I had modified host file of my laptop for site URL to point it directly to server (which was having issues), bypassing the load balancer.
Important thing which I noticed with Wireshark packet capture was that Client's PC sent request (client hello packet) and server sends ACK packet (acknowledging the the web request). but next packet just after that ACK is a FIN and ACK packet, which indicated the server was gracefully closing/finishing the session. And post that client also acknowledges the FIN flag.
In order to understand why server was setting FIN flag was time since one has to understand SSL negotiation architecture and understand how it internally works. You can follow this link to understand SSL architecture.
So in my case first phase of SSL negotiation (client hello) went successfully. but did not see server hello in wireshark logs. instead saw FIN flag.
So my investigation initiated with 2nd phase where http.sys does reading before server sends out a server hello packet.
Normally if you have a site with SSL binding and an IP assigned to it. there will be a corresponding registry key as well in the registry which Http.sys read to know what SSL cert is assigned to the IIS site. What I noticed was in my case was this registry specific to my site was missing from registry even though I have tried rebinding the SSL certificate to site in IIS. IIS was just not creating the key.
Registry key that HTTP.sys check looks like below format
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo\<IP address:SSLport number>
Finally I had to manually add this key to registry and site started working after this.
IIS settings were same on both server (when I have checked/compared with IISCrypto tool). user request did not even reached the IIS or SharePoint layer as I could not see any log entries in either IIS or SharePoint logs. from IIS console prospective its had same SSL certificate and all settings just looked fine.
Decided to use WireShark Tool to see the network traffic and to check is the request were reaching server or not. I had modified host file of my laptop for site URL to point it directly to server (which was having issues), bypassing the load balancer.
Important thing which I noticed with Wireshark packet capture was that Client's PC sent request (client hello packet) and server sends ACK packet (acknowledging the the web request). but next packet just after that ACK is a FIN and ACK packet, which indicated the server was gracefully closing/finishing the session. And post that client also acknowledges the FIN flag.
In order to understand why server was setting FIN flag was time since one has to understand SSL negotiation architecture and understand how it internally works. You can follow this link to understand SSL architecture.
So in my case first phase of SSL negotiation (client hello) went successfully. but did not see server hello in wireshark logs. instead saw FIN flag.
So my investigation initiated with 2nd phase where http.sys does reading before server sends out a server hello packet.
Normally if you have a site with SSL binding and an IP assigned to it. there will be a corresponding registry key as well in the registry which Http.sys read to know what SSL cert is assigned to the IIS site. What I noticed was in my case was this registry specific to my site was missing from registry even though I have tried rebinding the SSL certificate to site in IIS. IIS was just not creating the key.
Registry key that HTTP.sys check looks like below format
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo\<IP address:SSLport number>
Finally I had to manually add this key to registry and site started working after this.
