How to modify the replication interval between Domain controllers in different sites to less than 15 minutes?

I recently came across a problem where DCs in our two data centers (Each datacenter is a different AD site) were not replicating fast enough. Because the DCs in question are in different sites, the minimum interval for replication between sites can only be 15 minutes, which is not good enough for us. So the workaround is to set the ADSI attribute “options” for the site link to 1 which means USE_NOTIFY attribute.

USE_NOTIFY forces the replication to occur when a change is made. But this does not work on manually created connections like we have. It only works on connections managed and created by KCC (these connections will have the value “automatically generated” shown under the first column. (details can be found at http://blogs.msdn.com/b/canberrapfe/archive/2012/03/26/active-directory-replication-change-notification-amp-you.aspx)

So what worked in our case to force the servers to override the default values by using integer value 12 for each server’s NTDS settings in ADSI. Details are in http://www.ryanjadams.com/2010/05/turbo-charge-active-directory/#axzz2PaGkqbUu. I have modified the attribute “options” for each server

a.       Open ADSI edit

b.      Connect to and expand the Configuration container

c. Navigate to Sites-> Sitename-> Servers-> servername-> NTDS Settings

d. Right click and select properties for each manual connection object in this folder.

e. Select the “Options” property.  Note that if the value is 1 then it is an intrasite connection object and owned by the KCC.  If the value is 5 then it is an intersite connection object and owned by the KCC.  If it is one of these values and owned by the KCC then do NOT change it.  It should be changed at the site level instead, otherwise if you change the value on a connection object that is owned by the KCC you force it out of KCC control and the KCC will no longer manage it.

f. In the edit attribute box, change the value to 12.

How to find out which device is the domain account getting locked out from?

Great tool from Microsoft to use is LockOutStatus [Account Lockout and Management Tools]
http://www.microsoft.com/en-us/download/details.aspx?id=18465
These tools are mainly for Windows XP and Server 2003, but LockOutStatus.exe works on Server 2008 R2 Active Directories too.

Here you can see the list of you AD servers and last time that particular account was locked out.
View the servers Security Event Viewer on the Active Directory computer where the account was last locked out and filter for Event ID: 644 [2003] or 4740 [2008]

The Caller Machine Name is will indicate where the incorrect credentials are originating from

2) Find out what process/application is using the incorrect credentials.

If the Caller Machine Name is coming from a known server such as Exchange, to check is if the user has additional devices such as iphone, ipad which perhaps do not have the latest users credentials.

Microsoft Exchange 2010 - Monitoring Search indexer Crawl status

Search indexer full crawl can take a long time depending on the size of the database. At the mailbox level, the powershell command can be used in verbose mode.

C:\Windows\system32>test-exchangesearch -identity testuser1 -verbose

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Active Directory session settings for 'Test-ExchangeSearch' are: View

Entire Forest: 'False', Default Scope: 'domain.com', Configuration Domain Controller: 'dc.domain.com',

Preferred Global Catalog: 'dc.domain.com', Preferred Domain Controllers: '{ dc.domain.com }'

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Runspace context: Executing user:

Domain.com/Admin Accounts/Individual/admin, Executing user organization: ,

Current organization: , RBAC-enabled: Enabled.

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Beginning processing &

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Instantiating handler with index 0 for cmdlet extension agent "Admin

Audit Log Agent".

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write

Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s):

{}, Exclusive Configuration Scope(s): {} }

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Searching objects "testuser1" of type "ADUser" under the root "$null".

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Processing object "domain.com/Users/testuser1".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Searching objects "MBX06_DB01" of type "MailboxDatabase" under the

root "$null".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Admin Audit Log: Entered Handler:Validate.

VERBOSE: Testing Exchange Search.

VERBOSE: [18:34:15.331 GMT] Test-ExchangeSearch : Resolved current organization: .

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting catalog state for Mailbox Database "MBX_DB01"...

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Catalog state: Crawling.

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting the indexing state for mailbox "testuser1" ...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Mailbox state: NotStarted, Crawl start time: 0001-01-01T00:00:00.

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Getting the message indexing state...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Document state: Queued, state = 0.

VERBOSE: [18:36:16.903 GMT] Test-ExchangeSearch : TimeSinceLastNotificationWasIndexed for MDB MBX06_DB01 is 0 seconds.

The mailbox state will tell if this mailbox has been crawled or not, in this case, the crawl has not yet started for this mailbox.

At the mailbox database level or server level, we can use the performance monitor to monitor the status of the crawl.

Most of the counters can be found under MSExchange Search Indices

Some of the common ones I used are

Full Crawl Mode Status - The Value is 0 when the crawl is completed

Number of backlogged items added to the retry table - if the value is more than 30000 after the crawl is complete, restart the Microsoft Exchange Search indexer service

Number of failed mailboxes - this will tell how many mailboxes failed to be crawled

Number of mailboxes left to crawl - if the crawl is still running, how many mailboxes are left to crawl

Microsoft Exchange 2010 - Search does not return all the results

We recently had a problem with our Exchange 2010 search indexer where it would only return the results from today forward but none of the older results. When we checked the event log event ID 107 was reported:

Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database Database1 (GUID = 32299692-d5af-4415-8ca0-36f9d485ddb6) due to an error (Microsoft.Mapi.MapiExceptionNotFound: MapiExceptionNotFound: Unable to update CI Retry table  (hr=0x80004005, ec=-1601)

Diagnostic context:

    Lid: 1494    ---- Remote Context Beg ----

    Lid: 28242   StoreEc: 0xFFFFF9BF

    Lid: 59457 

    Lid: 22610   StoreEc: 0xFFFFF9BF

    Lid: 52115   dwParam: 0x1148628

    Lid: 4000    StoreEc: 0xFFFFF9BF

    Lid: 16793   StoreEc: 0xFFFFF9BF

    Lid: 14126   StoreEc: 0xFFFFF9BF

    Lid: 1750    ---- Remote Context End ----

    Lid: 10098   StoreEc: 0xFFFFF9BF

    Lid: 20226 

    Lid: 15346   StoreEc: 0xFFFFF9BF

   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)

   at Microsoft.Mapi.ExRpcAdmin.CiUpdateRetryTable(Guid mdbGuid, Guid instanceGuid, UInt32[] documentIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates)

   at Microsoft.Exchange.Search.PropertyStore.UpdateRetryTable(PropertyStore ps, UInt32[] docIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates, Object& unusedOut1, Object& unusedOut2)

   at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type& outParameter1, OutParameter2Type& outParameter2, OperationDelegate`7 operationDelegate)

   at Microsoft.Exchange.Search.PropertyStore.UpdateDocumentStatesOnBatchCompletion(BatchInformation batchInformation)

After this error, it was almost that the server reached a checkpoint and would not return any older items in the search result. We restarted the Microsoft Exchange Search Indexer and the Microsoft Search (exchange) service but it did not resolve the problem. To test whether there are any corruption problems, we ran the script for testing exchange search

C:\Windows\system32>test-exchangesearch -identity Username -verbose

Everything seemed to be normal, except for the search for older items.

Exchange provides a range of scripts that are present in the \Scripts directory.  By Default this is C:\Program Files\Microsoft\Exchange Server\V14\Scripts for Exchange 2010.  Exchange 2010 SP1 introduced the Content Index Troubleshooter Script – Troubleshoot-CI.ps1.  This script can monitor and perform troubleshooting on a server’s CI catalogs and can detect and resolve symptoms such as

• Deadlock: Exchange Search deadlocks waiting on threads from MSSearch
• Corruption: One or more search indices are corrupted
• Stall: Similar to deadlock in that the indices are not getting updated
• Backlog: The Search catalog is backlogged resulting in missing index searches

 To fix this problem we had to rebuild the search indexer catalog. The easy way is to use the script under the scripts folder

ResetSearchIndex.ps1 -force -all

This will delete all the catalog folders and rebuild the catalog. The search will not be accessible until the catalog crawling is complete. We can monitor the search index to see how many mailboxes are left to crawl (I will write a blog about that soon). I do recommend running the rebuild after business hours if the database is anything bigger than 500GB as it does take a long time (>5 hours) to rebuild the catalog and may cause outlook to be lag if you are running in online mode.

Netapp Snapmanager for SQL - setting different snapshot retention settings for different backup jobs

Netapp Snapmanager for SQL is an amazing product. I personally love it. Snapmanager allows you to SQL aware snapshots for the databases residing on the Netapp LUNs. It goes without saying that you have to follow the best practices for setting up the LUNs with databases and log files on separate LUNs for each instance to have this work correctly. The Jobs can be scheduled as SQL jobs or windows tasks. We can set retention on these backups according to the number of backups and number of days.

Now when the retention is simple, something like 7 days or 10 backups, it is fairly easy to specify that. However, the retention for our backups was a little more complicated than that. We required backups to occur every four hours at the business hours and retain those backups for 7 days and than have a once a day backup with a retention of 42 days. So in other words a four hourly backup for 7 days and a daily backup for 42 days needed to be the retention period. This is where the management groups come into place. Snapmanager for SQL has whats called backup management groups that allow you to set up different retention period for different backup jobs.

So for the 4-hourly backups we used the 'standard' management group. Under 'Backup' > 'backup and verify' > 'backup management group' - the option 'standard' is selected from the drop down.

For the daily backups we change this option to 'daily'. So when Snapmanager deletes the backups older than 7 days for the 4 hourly backups, it understands the daily management group and does not delete the backups under the 'daily' management group. That way we can have separate retention for different backup jobs.