Remove Duplicate Exchange Objects on Office 365

Remove Duplicate Exchange Objects on Office 365

If you are doing a migration to Office 365 or an upgrade between Exchange versions, then you may end up with duplicates in some or all of your Exchange accounts. This happened to us after moving from Exchange 2013 to Office 365 when some things went wrong (future blog post coming I promise) but I wanted to get this down before I forgot it.

So, I've got about 100 users that now are on Office 365 (Exchange Online) and have duplicates and I don't want to manually go to every computer and try to fix it. Also, I don't want my staff to have to jump through a bunch of hoops to try and fix it (which will probably cause more work for me later). If only there was a way I could script all this...

Turns out, someone has! Michel de Rooij has created an AWESOME PowerShell script called Remove-DuplicateItems.ps1 and you can read more about it here. There are a lot of options and switches that you can use with this script and some are documented better than others.

In my initial testing I found that this wasn't removing duplicates even when used in "Full" mode and not "Quick" mode. It turns out that the way the 3rd party tool that we used to sync our accounts in our Office 365 migration changed the ID's of the items and even items that looked to match perfectly were measuring larger in the system (two attributes that are used when comparing). So, after some head scratching, I found that I needed to comment out a few lines of code to get it to work in our situation.

On line 357, I commented out
#if ($Item.Size) { $key+= ","+$Item.Size.ToString()} 
by putting a # in front of the line.

And then on "IPM.Contact" section starting at 359 I commented out a few lines as seen below:

This was done because the way our duplicates were made the Size comparison was failing because the two records weren't the same. Also I found that for whatever reason CompanyName, and the phone number fields caused some issues for me. Make sure you test this script on a small set of users because there can be some false positives when you comment these out. The good news is that you can use the -DeleteMode MoveToDeletedItems option to have them got to the Deleted Items folder and could be easily restored. So, awesome stuff but how does it work?

• First you need a computer with PowerShell installed (it doesn't have to be the Exchange server) as well as Exchange Web Services (EWS) Managed API 1.2 (or later) installed. 
• You need a .csv list of the users you want to remove duplicates from (I recommend several batches of less than 20 because it does take a while and only does one at a time and will stop the script of a box fails)
• If you're running this against Office 365 you need to have setup a user with Impersonation rights on the mailboxes (outlined in the source blog)

Now, run the script. (My comments are in green)

# Set credentials for the next scripts - If using Office 365, make sure you 
# use the username@domain.com

$UserCredential = Get-Credential 

#The next command references a script called Remove-DuplcateItems.ps1 which can be
# downloaded at http://bit.ly/1IRgTnc 

PS C:\Users\Administrator> Import-CSV Users.csv | .\Remove-DuplicateItems.ps1 -Type All -Credentials ($Credentials) -Impersonation -DeleteMode MoveToDeletedItems -Mode Full

When it's running, you will see something like this:

A helpful option when you are testing this out is -Verbose which will outline the task that it is doing and ask you what you want to do with each delete. When you are doing a batch it's not very helpful. You can also use the -Debug option and get in-depth look at how the script is comparing things (helpful in determining why some duplicates are getting deleted and some aren't)

I spent many hours trying different scripts and figuring out why it wasn't working but I did get it to work. I hope this blogpost helps someone else out there and I'm sure I'll look back the next time I have to run this :-)