To understand how to configure Hybrid Search topologies see Part 1 and Part 2 of this series.Things are not always straightforward though and sometimes errors occur or mistakes are made when following this guidance.
The information in this post is generated while troubleshooting a test hybrid environment with SharePoint 2013 and Office 365. Some of the guidance below involves retrieving ULS logs for which you would have to engage Microsoft Support.
In this article we will describe how to troubleshoot certificate issues that occur in a hybrid search deployment of Microsoft SharePoint Online in Office 365 and of on-premises Microsoft SharePoint 2013 Server. For example when you try to submit a search query from Sharepoint Online to on premises SharePoint 2013 Server the search query fails with below error.
1¾System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure
Examination of the Office 365 SharePoint Online ULS logs shows the following entries.
NodeRunner.exe (0x1194) 0x1B34 SharePoint Foundation Topology 8311 Critical An operation failed because the following certificate has validation errors: Subject Name: CN=spweb.mbiswas.com, OU= Online , O=Hybrid Corp, L=Bangalore, S=Blr, C=IN Issuer Name: CN=Test CA, DC=hybrid, DC=test, DC=dc, DC=mbcloud, DC=com Thumbprint: <certificate thumbprint> Errors: PartialChain: A certificate chain could not be built to a trusted root authority. RevocationStatusUnknown: The revocation function was unable to check revocation for the certificate. OfflineRevocation: The revocation function was unable to check revocation because the revocation server was offline.
NodeRunner.exe (0x1194) 0x0C94 SharePoint Foundation Topology 8311 Critical An operation failed because the following certificate has validation errors: Subject Name: CN=spweb.mbiswas.com, OU= Online , O=Hybrid Corp, L=Bangalore, S=Blr, C=IN Issuer Name: CN=Test CA, DC=hybrid, DC=test, DC=dc, DC=mbcloud, DC=com Thumbprint: <certificate thumbprint> Errors: PartialChain: A certificate chain could not be built to a trusted root authority. RevocationStatusUnknown: The revocation function was unable to check revocation for the certificate. OfflineRevocation: The revocation function was unable to check revocation because the revocation server was offline. .
NodeRunner.exe (0x1194) 0x1B34 SharePoint Server Search Query ajhxa High RemoteSharepointProducerSystem.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure. at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
During the configuration of inbound Hybrid search you will have setup the Reverse Proxy publishing rule for client certificate pre-authentication and uploaded a copy of the SAN or wildcard certificate to the Secure Store in SharePoint Online, as well as to the local computer account personal store on the Web Application Proxy server. The above error is thrown due to an invalid or unreachable AIA (Authority Information Access) location specified on this certificate. An AIA location is a url indicating the publication source of the certificate used for verification purposes.
You can inspect the certificate yourself by downloading a public copy of the certificate by browsing to the published SharePoint On prem URL. If you verify the certificate using the certutil tool you should see an error similar to the example below
<certificate thumbprint>
Missing Issuer: CN=Test CA, DC=hybrid, DC=test, DC=dc, DC=mbcloud, DC=com
Issuer: CN=Test CA, DC=hybrid, DC=test, DC=dc, DC=mbcloud, DC=com
NotBefore: 05/04/2014 00:45
NotAfter: 05/04/2015 00:45
Subject: CN=spweb.mbiswas.com, OU= Online , O=Hybrid Corp, L=Bangalore, S=Blr, C=IN
Serial: 64813e6d99970000066a
SubjectAltName: DNS Name=spweb.mbclould.com, DNS Name= spwebmail.mbclould.com, DNS Name= sharepoint.mbclould.com, DNS Name=App.mbcloud.com
A certificate chain could not be built to a trusted root authority. 0x800b010a (-2146762486 CERT_E_CHAINING)
Incomplete certificate chain
Cannot find certificate:
CN=Test CA, DC=hybrid, DC=test, DC=dc, DC=mbcloud, DC=com is an End Entity certificate
ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATIO
N_OFFLINE)
CertUtil: The revocation function was unable to check revocation because the revocation server was offline.
If your analysis shows a similar output, you need to validate why the AIA location(s) specified on the certificate are not accessible. You need to work with the certificate issuer to determine why these AIA location are unavailable publically. Alternatively, if you are able to obtain a certificate with a valid publically accessible AIA location, this should fix the issue.
The current list of Microsoft Windows supported root certificate authorities can be found here.
This article is the first of the series of Hybrid troubleshooting posts . Watch this space for more troubleshooting tips and Tricks around Hybrid.