Friday, April 11, 2014

Synology tip: How to find duplicate files or large files automatically

I love my Synology, as I use it for many things and it just works. From media server, to file archive, to backup, to photo server, to surveillance... It has many possibilities. If you add high capacity drives (I have 2 x 3 TB), it can hold quite a lot of information.

Lately I've been centralizing all digital information on the Synology, mainly because I used to have info on multiple computers, external hard drives, backups of backups, etc... and wanted to have everything in one place. After copying and organizing a bit, I found out that I had multiple copies of same files, pictures, videos, etc...

But how to check and erase only duplicates?

 

I started by using Gemini, a duplicate finding software for MacOS. However the search needs to be done via network (executed on the Mac) and takes forever to generate a report. Wouldn't it be great if Synology could do this without a computer involved?

Luckily, Synology thought of a way to do this automatically by the DSM. Not only that but it can automatically generate duplicate reports on a regular basis and email them to you. The email contains a link which takes you to a webpage where you can delete the duplicate file. EASY!

How to set it up:

Enter your Synology via the desktop DSM ( or simply put login to your Synology ). If you have DSM 5, you should see something like this:

Synology DSM 5 main screen
Charlietechlife.blogspot.com

Open the control panel and under "System" go to "Info center"

Information on your Synology product will appear. On the top there is a tab named "Report". Click on it

Press "Create" to start the report wizard

In the next screen give a identifiable name to your report. I called mine "Duplicates". Also add an email so that your Synology can email you a direct link when the report is ready.
 

After pressing next, the screen will contain several options. Check Only "Duplicate File Candidates" and press next.

 
 

The next screen is about scheduling, when do you want Synology to generate the report. I have it once per week every Monday at 00:00. To note this is when it starts generating the report, not when it finishes or sends you the email. In the case you want to verify duplicated (next screen), and depending on the quantity of files you have it make take a while.

After pressing next, you can check "Find confirmed duplicates", which makes 100% sure the files shown are duplicates. I definitely recommend this option. Here you can also choose to check every files in the Synology or only specific folders.

In the next screen verify your options and press "Apply". And there it is! After a some time you will get your first duplicate report.

 

And within it you can erase the files you don't want to keep!

 

50 comments :

  1. Brilliant, thank you very much!

    ReplyDelete
  2. Thanx! Just what I've been looking for

    ReplyDelete
  3. Thanks for the instructions!

    ReplyDelete
  4. That is exactly what I am looking for. THX!

    ReplyDelete
  5. Thank you sooooo much!

    ReplyDelete
  6. In DSM 4.3 there seems to be no 'Info Center' (nor an 'I'-icon - I have a Dutch version) in the System line-up. Is there a way to execute this great sounding procedure in DSM 4.3?

    ReplyDelete
  7. In DSM 4.3 there seems to be no 'Info Center' (nor an 'I'-icon - I have a Dutch version) in the System line-up. Is there a way to execute this great sounding procedure in DSM 4.3?

    ReplyDelete
  8. In DSM 4.3 there seems to be no 'Info Center' (nor an 'I'-icon - I have a Dutch version) in the System line-up. Is there a way to execute this great sounding procedure in DSM 4.3?

    ReplyDelete
    Replies
    1. Hello! Not sure on DSM 4.3, but would think it is under main menu -> system information -> reports

      Delete
  9. In DSM 4.3 there seems to be no 'Info Center' (nor an 'I'-icon - I have a Dutch version) in the System line-up. Is there a way to execute this great sounding procedure in DSM 4.3?

    ReplyDelete
  10. Correct but prefer the inhouse method! And that it emails me every week with this!

    ReplyDelete
  11. For removing all your junk files on your pc use DuplivateFilesDeleter.

    ReplyDelete
  12. Charlie, thanks for this article. Under DSM 5.1 this functionality has moved from a tab on the Info Centre into the Storage Analyzer application. The new report wizard works much the same way. Currently waiting for my first duplicate song report, tidy up my messy music collection. Cheers, Chris.

    ReplyDelete
    Replies
    1. Thanks! I'll update with a new post with the latest DSM way of doing it.

      Delete
  13. Nice post. Hope it will be helpful for those people. I am using DuplicateFilesDeleter for this problem. Its easy to use and very reliable. Hope everyone can use it.

    ReplyDelete
  14. my version of DSM 5.1 did not have "Reports" under the Info Center. Executed a search for "reports" and managed to locate the feature.

    ReplyDelete
    Replies
    1. Hi Clark, I couldn't find the "Report" using the search. Can you describe more? On the latest DSM 5.2 now. Thanks in advance.

      Delete
    2. it's now in an application called Storage Analyzer

      Delete
  15. Can't find this on my dsm latest version. Any advice please.

    ReplyDelete
  16. Thanks for this! Helpful. I got confused with finding duplicate files and deduplication.

    ReplyDelete
  17. How can I delete the duplicates now that I have found them? I am hoping not to have to go through and manually delete thousands of duplicates.

    ReplyDelete
    Replies
    1. I haven't found a way yet, just clicking file by file on the ones you want to delete ....

      Delete
    2. Many thanks Charlie, for right now I am trying to run a 3rd party duplicate remover on my audio files. It has been running for 3 days and is about 60% done, so I hope this will work. over 26,000 duplicate files thus far...

      Delete
    3. I tried the Gemini duplicate finder software from Macpaw. It has an auto mode to eliminate duplicates (i.e. no clicking) but you can't set up any rules which is frustrating.

      Delete
    4. I shall try that if this one fails (Leawo Tunes Cleaner), one problem with it deleting all but one duplicate is that some tracks may indeed be duplicates but be part of different albums (such as compilations/greatest hits releases, etc.) which does mess up the albums a bit.

      Delete
  18. it seems the tab report isnt there anymore in dsm 5.2

    ReplyDelete
  19. the reports have been moved to the "Storage Analyzer" module you can find in control panel.

    ReplyDelete
    Replies
    1. Thanks for clarification, I was first ecstatic, and then depressed. Synology keeps moving stuff around under the hood...

      Delete
  20. Great thanks! Just set it up in DSM 6.

    ReplyDelete
  21. When I delete a duplicate through Storage Analyzer it never actually deletes. Any ideas?

    ReplyDelete
  22. Thank you so much! I've been doing this with Gemini as well and it's such a pain! Great tip!

    ReplyDelete
  23. And...they moved it/removed it again in DSM 6.0

    ReplyDelete
  24. Super article. Tks for sharing

    ReplyDelete
  25. Super article. Tks for sharing

    ReplyDelete
  26. When I click 'delete' they show as 'removed' but are still there and haven't increased my available disk space ?!?

    ReplyDelete
    Replies
    1. Does the module have all rights to remove the duplicates if the user wishes so?

      Delete
  27. Storage analyzer is now an additional module which can be found in the package center for download

    ReplyDelete
    Replies
    1. Yes this article though was useful is now deprecated

      Delete
    2. ah - is this package not available for a DS410? I can't find it anywhere in the package center. I guess the packages are also product version specific??

      Delete
  28. Thank you! It really works. I use to remove duplicate files Duplicate Finder Free http://orderprog.com/remove-duplicates.php also.

    ReplyDelete
  29. Hello Goodday, I use a software called Duplicate Files Deleter, it's very easy to use and after it finds the duplicate files it lets you chose what you want to do with them (copy/delete/move). You can even check network files and you can check multiple paths in the same scan. This helps me alot. I hope you too.

    ReplyDelete
  30. Hello Goodday, I use a software called Duplicate Files Deleter, it's very easy to use and after it finds the duplicate files it lets you chose what you want to do with them (copy/delete/move). You can even check network files and you can check multiple paths in the same scan. This helps me alot. I hope you too.

    ReplyDelete
  31. For whatever reason, there is no option to delete from the duplicate report as stated. The report generated shows the groups and all other requested data but no link, button or method to actually delete the found duplicates via the reports HTML page. Running the very latest DSM 6.0

    ReplyDelete
  32. Still a shity way to delete duplicate files for those who have 1000s of duplicates

    ReplyDelete
  33. Must be a better method to remove the files than manually selecting. For example pattern match on a directory or path. I've got over 33,000 duplicates in images due to consolidation of previous backup locations.

    ReplyDelete