Microsoft Exchange 2010 - Monitoring Search indexer Crawl status

Search indexer full crawl can take a long time depending on the size of the database. At the mailbox level, the powershell command can be used in verbose mode.

C:\Windows\system32>test-exchangesearch -identity testuser1 -verbose

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Active Directory session settings for 'Test-ExchangeSearch' are: View

Entire Forest: 'False', Default Scope: 'domain.com', Configuration Domain Controller: 'dc.domain.com',

Preferred Global Catalog: 'dc.domain.com', Preferred Domain Controllers: '{ dc.domain.com }'

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Runspace context: Executing user:

Domain.com/Admin Accounts/Individual/admin, Executing user organization: ,

Current organization: , RBAC-enabled: Enabled.

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Beginning processing &

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Instantiating handler with index 0 for cmdlet extension agent "Admin

Audit Log Agent".

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write

Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s):

{}, Exclusive Configuration Scope(s): {} }

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Searching objects "testuser1" of type "ADUser" under the root "$null".

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Processing object "domain.com/Users/testuser1".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Searching objects "MBX06_DB01" of type "MailboxDatabase" under the

root "$null".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Admin Audit Log: Entered Handler:Validate.

VERBOSE: Testing Exchange Search.

VERBOSE: [18:34:15.331 GMT] Test-ExchangeSearch : Resolved current organization: .

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting catalog state for Mailbox Database "MBX_DB01"...

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Catalog state: Crawling.

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting the indexing state for mailbox "testuser1" ...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Mailbox state: NotStarted, Crawl start time: 0001-01-01T00:00:00.

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Getting the message indexing state...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Document state: Queued, state = 0.

VERBOSE: [18:36:16.903 GMT] Test-ExchangeSearch : TimeSinceLastNotificationWasIndexed for MDB MBX06_DB01 is 0 seconds.

The mailbox state will tell if this mailbox has been crawled or not, in this case, the crawl has not yet started for this mailbox.

At the mailbox database level or server level, we can use the performance monitor to monitor the status of the crawl.

Most of the counters can be found under MSExchange Search Indices

Some of the common ones I used are

Full Crawl Mode Status - The Value is 0 when the crawl is completed

Number of backlogged items added to the retry table - if the value is more than 30000 after the crawl is complete, restart the Microsoft Exchange Search indexer service

Number of failed mailboxes - this will tell how many mailboxes failed to be crawled

Number of mailboxes left to crawl - if the crawl is still running, how many mailboxes are left to crawl

Microsoft Exchange 2010 - Search does not return all the results

We recently had a problem with our Exchange 2010 search indexer where it would only return the results from today forward but none of the older results. When we checked the event log event ID 107 was reported:

Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database Database1 (GUID = 32299692-d5af-4415-8ca0-36f9d485ddb6) due to an error (Microsoft.Mapi.MapiExceptionNotFound: MapiExceptionNotFound: Unable to update CI Retry table  (hr=0x80004005, ec=-1601)

Diagnostic context:

    Lid: 1494    ---- Remote Context Beg ----

    Lid: 28242   StoreEc: 0xFFFFF9BF

    Lid: 59457 

    Lid: 22610   StoreEc: 0xFFFFF9BF

    Lid: 52115   dwParam: 0x1148628

    Lid: 4000    StoreEc: 0xFFFFF9BF

    Lid: 16793   StoreEc: 0xFFFFF9BF

    Lid: 14126   StoreEc: 0xFFFFF9BF

    Lid: 1750    ---- Remote Context End ----

    Lid: 10098   StoreEc: 0xFFFFF9BF

    Lid: 20226 

    Lid: 15346   StoreEc: 0xFFFFF9BF

   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)

   at Microsoft.Mapi.ExRpcAdmin.CiUpdateRetryTable(Guid mdbGuid, Guid instanceGuid, UInt32[] documentIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates)

   at Microsoft.Exchange.Search.PropertyStore.UpdateRetryTable(PropertyStore ps, UInt32[] docIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates, Object& unusedOut1, Object& unusedOut2)

   at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type& outParameter1, OutParameter2Type& outParameter2, OperationDelegate`7 operationDelegate)

   at Microsoft.Exchange.Search.PropertyStore.UpdateDocumentStatesOnBatchCompletion(BatchInformation batchInformation)

After this error, it was almost that the server reached a checkpoint and would not return any older items in the search result. We restarted the Microsoft Exchange Search Indexer and the Microsoft Search (exchange) service but it did not resolve the problem. To test whether there are any corruption problems, we ran the script for testing exchange search

C:\Windows\system32>test-exchangesearch -identity Username -verbose

Everything seemed to be normal, except for the search for older items.

Exchange provides a range of scripts that are present in the \Scripts directory.  By Default this is C:\Program Files\Microsoft\Exchange Server\V14\Scripts for Exchange 2010.  Exchange 2010 SP1 introduced the Content Index Troubleshooter Script – Troubleshoot-CI.ps1.  This script can monitor and perform troubleshooting on a server’s CI catalogs and can detect and resolve symptoms such as

• Deadlock: Exchange Search deadlocks waiting on threads from MSSearch
• Corruption: One or more search indices are corrupted
• Stall: Similar to deadlock in that the indices are not getting updated
• Backlog: The Search catalog is backlogged resulting in missing index searches

 To fix this problem we had to rebuild the search indexer catalog. The easy way is to use the script under the scripts folder

ResetSearchIndex.ps1 -force -all

This will delete all the catalog folders and rebuild the catalog. The search will not be accessible until the catalog crawling is complete. We can monitor the search index to see how many mailboxes are left to crawl (I will write a blog about that soon). I do recommend running the rebuild after business hours if the database is anything bigger than 500GB as it does take a long time (>5 hours) to rebuild the catalog and may cause outlook to be lag if you are running in online mode.

Netapp Snapmanager for SQL - setting different snapshot retention settings for different backup jobs

Netapp Snapmanager for SQL is an amazing product. I personally love it. Snapmanager allows you to SQL aware snapshots for the databases residing on the Netapp LUNs. It goes without saying that you have to follow the best practices for setting up the LUNs with databases and log files on separate LUNs for each instance to have this work correctly. The Jobs can be scheduled as SQL jobs or windows tasks. We can set retention on these backups according to the number of backups and number of days.

Now when the retention is simple, something like 7 days or 10 backups, it is fairly easy to specify that. However, the retention for our backups was a little more complicated than that. We required backups to occur every four hours at the business hours and retain those backups for 7 days and than have a once a day backup with a retention of 42 days. So in other words a four hourly backup for 7 days and a daily backup for 42 days needed to be the retention period. This is where the management groups come into place. Snapmanager for SQL has whats called backup management groups that allow you to set up different retention period for different backup jobs.

So for the 4-hourly backups we used the 'standard' management group. Under 'Backup' > 'backup and verify' > 'backup management group' - the option 'standard' is selected from the drop down.

For the daily backups we change this option to 'daily'. So when Snapmanager deletes the backups older than 7 days for the 4 hourly backups, it understands the daily management group and does not delete the backups under the 'daily' management group. That way we can have separate retention for different backup jobs.