Microsoft Exchange 2010 - Search does not return all the results

We recently had a problem with our Exchange 2010 search indexer where it would only return the results from today forward but none of the older results. When we checked the event log event ID 107 was reported:

Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database Database1 (GUID = 32299692-d5af-4415-8ca0-36f9d485ddb6) due to an error (Microsoft.Mapi.MapiExceptionNotFound: MapiExceptionNotFound: Unable to update CI Retry table  (hr=0x80004005, ec=-1601)

Diagnostic context:

    Lid: 1494    ---- Remote Context Beg ----

    Lid: 28242   StoreEc: 0xFFFFF9BF

    Lid: 59457 

    Lid: 22610   StoreEc: 0xFFFFF9BF

    Lid: 52115   dwParam: 0x1148628

    Lid: 4000    StoreEc: 0xFFFFF9BF

    Lid: 16793   StoreEc: 0xFFFFF9BF

    Lid: 14126   StoreEc: 0xFFFFF9BF

    Lid: 1750    ---- Remote Context End ----

    Lid: 10098   StoreEc: 0xFFFFF9BF

    Lid: 20226 

    Lid: 15346   StoreEc: 0xFFFFF9BF

   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)

   at Microsoft.Mapi.ExRpcAdmin.CiUpdateRetryTable(Guid mdbGuid, Guid instanceGuid, UInt32[] documentIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates)

   at Microsoft.Exchange.Search.PropertyStore.UpdateRetryTable(PropertyStore ps, UInt32[] docIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates, Object& unusedOut1, Object& unusedOut2)

   at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type& outParameter1, OutParameter2Type& outParameter2, OperationDelegate`7 operationDelegate)

   at Microsoft.Exchange.Search.PropertyStore.UpdateDocumentStatesOnBatchCompletion(BatchInformation batchInformation)

After this error, it was almost that the server reached a checkpoint and would not return any older items in the search result. We restarted the Microsoft Exchange Search Indexer and the Microsoft Search (exchange) service but it did not resolve the problem. To test whether there are any corruption problems, we ran the script for testing exchange search

C:\Windows\system32>test-exchangesearch -identity Username -verbose

Everything seemed to be normal, except for the search for older items.

Exchange provides a range of scripts that are present in the \Scripts directory.  By Default this is C:\Program Files\Microsoft\Exchange Server\V14\Scripts for Exchange 2010.  Exchange 2010 SP1 introduced the Content Index Troubleshooter Script – Troubleshoot-CI.ps1.  This script can monitor and perform troubleshooting on a server’s CI catalogs and can detect and resolve symptoms such as

• Deadlock: Exchange Search deadlocks waiting on threads from MSSearch
• Corruption: One or more search indices are corrupted
• Stall: Similar to deadlock in that the indices are not getting updated
• Backlog: The Search catalog is backlogged resulting in missing index searches

 To fix this problem we had to rebuild the search indexer catalog. The easy way is to use the script under the scripts folder

ResetSearchIndex.ps1 -force -all

This will delete all the catalog folders and rebuild the catalog. The search will not be accessible until the catalog crawling is complete. We can monitor the search index to see how many mailboxes are left to crawl (I will write a blog about that soon). I do recommend running the rebuild after business hours if the database is anything bigger than 500GB as it does take a long time (>5 hours) to rebuild the catalog and may cause outlook to be lag if you are running in online mode.