4
Nov

Powershell Translets Find Duplicates

by Chris

Here is a simple translet for finding duplicates.

function T-FindDuplicates{
    param ($inxml)
    begin{
        . PSlib:\xml\invoke-transform.ps1
        [xml]$xslt = @"

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="xml" indent="yes" />
    <xsl:key name="file-checksums" match="file" use="@Checksum" />
    <xsl:template match="file">
        <xsl:copy>
            <xsl:attribute name="Duplicate">
                <xsl:value-of select="count(key('file-checksums', @Checksum)) > 1" />
            </xsl:attribute>
            <xsl:apply-templates select="@*|node()" />
        </xsl:copy>
    </xsl:template>
    <xsl:template match="@* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

"@
    }
    process{
        if ($_ -is [xml]){
            [xml](invoke-transform -inxml $_ -inxsl $xslt)
        }
    }
    end{
        if ($inxml -is [xml]){
           [xml](invoke-transform -inxml $inxml -inxsl $xslt)
        }
    }
}

As you can see in the code it adds a Duplicate attribute with a value of true or false depending on whether there is a file with a duplicate @Checksum.

It can be used like this

PS> . .\T-GetDirAsXml.ps1
PS> . .\T-AddChecksum.ps1
PS> . .\T-FindDuplicates.ps1
PS> Get-DirAsXml | T-AddChecksum | T-FindDuplicates

It might produce

Here is the code

T-FindDuplicates.zip (745 b)

All of the files in folder test2 are copies of the files in test except for test.txt and as you can see only having an @Duplicate indicator doesn't tell you which file the file is a duplicate of so this translet is only useful if you have very few duplicate files.

What you do when you find a duplicate is up to you and depends very much on the downstream application. One thing you could do is put a list of the duplicate files into an attribute like this

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="xml" indent="yes" />
    <xsl:key name="file-checksums" match="file" use="@Checksum" />
    <xsl:template match="file">
        <xsl:copy>
            <xsl:if test="count(key('file-checksums', @Checksum)) > 1">
                <xsl:attribute name="Duplicate">true</xsl:attribute>
                <xsl:attribute name="Duplicates">
                    <xsl:for-each select="key('file-checksums', @Checksum)">
                        <xsl:call-template name="get-path" />
                        <xsl:value-of select="'
'" />
                    </xsl:for-each>
                </xsl:attribute>
            </xsl:if>
            <xsl:apply-templates select="@*|node()" />
        </xsl:copy>
    </xsl:template>
    <xsl:template name="get-path">
        <xsl:for-each select="ancestor-or-self::*[not(@Root)]">
            <xsl:value-of select="@Parent" />
            <xsl:text>\</xsl:text>
            <xsl:value-of select="@Name" />
        </xsl:for-each>
    </xsl:template>
    <xsl:template match="@* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

Which will produce