Convert text files recursively to UTF-8 in PowerShell

Question

I have a folder with text files which includes other folders in it, and these also contain some text files. I need to recursively convert all these files to UTF-8 encoding in PowerShell and preserve the folder structure during this process. I have tried this:

foreach( $i in get-childitem -recurse -name ) {
    get-content $i | out-file -encoding utf8 -filepath some_folder/$i
}

But it doesn't work, it can't reproduce the hierarchy of folders. How do I cope with this problem?

Answer 1

Try this one.

foreach($i in Get-ChildItem -Recurse) {
    if ($i.PSIsContainer) {
        continue
    }

    $dest = $i.Fullname.Replace($PWD, "some_folder")
    if (!(Test-Path $(Split-Path $dest -Parent))) {
        New-Item $(Split-Path $dest -Parent) -type Directory
    }

    get-content $i | out-file -encoding utf8 -filepath $dest
}

It grabs the full path of the file and replaces the current directory with the one you want. For example, you run this command in the directory C:\1\ ($PWD = C:\1\). If it finds the file C:\1\2\file.txt, it'll give you a $dest of some_folder\2\file.txt.

The first if block is there so you don't try to convert a directory.

The directories have to be created if they don't already exist - I originally forgot that.

---

If you want UTF8 without BOM, replace the get-content $i | out-file -encoding utf8 -filepath $dest line with the following (source):

$filecontents = Get-Content $i
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
[System.IO.File]::WriteAllLines($i, $filecontents, $Utf8NoBomEncoding)

Note that this may not be very performant for larger files, since it reads entire files into memory before writing them again. If efficiency is needed, it is possible to read line by line, or even a specific number of bytes at a time. However, I'd rather just write a quick program in C# by that point (since you'd be using .NET fuctions in PS anyway).

Answer 2

• Allows for files and folders
• File extension agnostic
• Overwrites original file if destination equals the path
• Encoding as a parameter

Usage: &"TextEncoding.ps1" -path "c:\windows\temps\folder1" -encoding "UTF8"

Here is the script I created:

[CmdletBinding()]
param(  
    [Parameter(Mandatory=$true)]
    [string]$path,
    [Parameter(Mandatory=$false)]
    [string]$dest = $path,
    [Parameter(Mandatory=$true)]
    [string]$encoding
)

function Set-Encoding(){

    #ensure it is a valid path
    if(-not(Test-Path -Path $path)){

        throw "File or directory not found at {0}" -f $path
    }

    #if the path is a file, else a directory
    if(Test-Path $path -PathType Leaf){

        #if the provided path equals the destination
        if($path -eq $dest){

            #get file extension
            $ext = [System.IO.Path]::GetExtension($path)

            #create destination
            $dest = $path.Replace([System.IO.Path]::GetFileName($path), ("temp_encoded{0}" -f $ext))

            #output to file with encoding
            Get-Content $path | Out-File -FilePath $dest -Encoding $encoding -Force

            #copy item to original path to overwrite (note move-item loses encoding)
            Copy-Item -Path $dest -Destination $path -Force -PassThru | ForEach-Object { Write-Output -inputobject ("{0} encoded {1}" -f $encoding, $_) }

            #remove the extra file
            Remove-Item $dest   

        }else{

            #output to file with encoding
            Get-Content $path | Out-File -FilePath $dest -Encoding $encoding -Force     

        }

    }else{

        #get all the files recursively
        foreach($i in Get-ChildItem -Path $path -Recurse) {


            if ($i.PSIsContainer) {
                continue
            }

            #get file extension
            $ext = [System.IO.Path]::GetExtension($i)

            #create destination
            $dest = "$path\temp_encoded{0}" -f $ext

            #output to file with encoding
            Get-Content $i.FullName | Out-File -FilePath $dest -Encoding $encoding -Force

            #copy item to original path to overwrite (note move-item loses encoding)
            Copy-Item -Path $dest -Destination $i.FullName -Force -PassThru | ForEach-Object { Write-Output -inputobject ("{0} encoded {1}" -f $encoding, $_) }

            #remove the extra file
            Remove-Item $dest

        }

    }

}

Set-Encoding