powershell/textutility

Microsoft.PowerShell.TextUtility module

Microsoft.PowerShell.TextUtility

This module is a collection of cmdlets for working with text content.

Compare-Text

This cmdlet compares two text strings using diff-match-patch.

diff-match-patch is under Apache 2.0 License

ConvertFrom-Base64

Return a string decoded from base64.

ConvertTo-Base64

Return a base64 encoded representation of a string.

ConvertFrom-TextTable

This will convert tabular data and convert it to an object.

Code of Conduct

Please see our Code of Conduct before participating in this project.

Security Policy

For any security issues, please see our Security Policy.

textutility's Issues

[question] Is this module only for Powershell Core ?

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

A dependency is missing under Powershell v5.1.

Expected behavior

?
[edit]
Works with this file : https://www.nuget.org/packages/System.Text.Json/6.0.0

.\system.text.json.6.0.0\lib\net461

Actual behavior

convertfrom-texttable : Impossible de charger le fichier ou l'assembly 'System.Text.Json, Version=6.0.0.0,
Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' ou une de ses dépendances. Le fichier spécifié est introuvable.
Au caractère Ligne:1 : 66
+ ... 0123456789'|convertfrom-texttable -noheader -columnoffset 0,15,23,40, ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [ConvertFrom-TextTable], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,TextTableParser.ConvertTextTableCommand

CompatiblePSEditions = @('Desktop', 'Core')

Error details

>$error[0]|select *


PSMessageDetails      :
Exception             : System.IO.FileNotFoundException: Impossible de charger le fichier ou l'assembly
                        'System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' ou une
                        de ses dépendances. Le fichier spécifié est introuvable.
                        Nom de fichier: 'System.Text.Json, Version=6.0.0.0, Culture=neutral,
                        PublicKeyToken=cc7b13ffcd2ddd51'
                           à TextTableParser.ConvertTextTableCommand.Emit(String line)
                           à TextTableParser.ConvertTextTableCommand.EndProcessing()
                           à System.Management.Automation.CommandProcessorBase.Complete()

                        AVT : la journalisation de liaison d'assembly est désactivée.
                        Pour activer la journalisation des échecs de liaison d'assembly, attribuez la valeur 1 à la
                        valeur de Registre [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD).
                        Remarque: une certaine perte de performance est associée à l'enregistrement dans le journal
                        des échecs de liaison d'assembly.
                        Pour désactiver cette fonctionnalité, supprimez la valeur de Registre
                        [HKLM\Software\Microsoft\Fusion!EnableLog].

TargetObject          :
CategoryInfo          : NotSpecified: (:) [ConvertFrom-TextTable], FileNotFoundException
FullyQualifiedErrorId : System.IO.FileNotFoundException,TextTableParser.ConvertTextTableCommand
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : à <ScriptBlock>, <Aucun fichier> : ligne 1
PipelineIterationInfo : {}

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.19041.3930
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.19041.3930}
BuildVersion                   10.0.19041.3930
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

Powershell v5.1

Visuals

Add multibase support

This one is more of a nice-to-have feature, but it could be very beneficial to add support for multibase encoding.

Multibase is a relatively simple encoding that uses a single prefix character to specify the base encoding of the rest of the string. This makes the base-encoded string self-descriptive, without having to specify separately the encoding that was used. Compatibility with non-multibase can be achieved by manually checking the letter prefix to make sure it matches the expected encoding and removing it to get the "clean" string.

A ConvertFrom-Multibase / ConvertTo-Multibase function could be made to support all the base encodings supported within this PowerShell module, starting with Base64 and Base64Url. From there it should be easy to add other common ones like base16 (hexadecimal).

ConvertFrom-TextTable -Skip parameter

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

There is some confusing behavior where Skip does not skip the number of lines but rather the number of non-empty lines.

TextUtility/src/code/TextTableParser.cs

Lines 102 to 111 in ad78ef8

	if(string.IsNullOrEmpty(_line))
	{
	continue;
	}
	// add the line to the list if we've skipped enough lines
	if (SkippedLines++ >= Skip)
	{
	Lines.Add(_line);
	}

Expected behavior

Skip should match the number of input object lines.

Actual behavior

Skip only matches on non-blank lines creating a disconnect between input and what is being processed.

Error details

No response

Environment data

7.4

Version

0.5.0

Visuals

No response

Add ConvertFrom-TextList

Summary of the new feature / enhancement

As a user I want to be able to easily parse text lists into PowerShell objects.

Here is an example from apt-cache.

Package: powershell
Version: 7.2.5-1.deb
Architecture: amd64
Section: shells
Priority: extra
Installed-Size: 186998
Maintainer: PowerShell Team <[email protected]>
Description: PowerShell is an automation and configuration management platform.
 It consists of a cross-platform command-line shell and associated scripting language.
Description-md5: 6ee533ea0564ca22e8caa699e30ba649
Homepage: https://microsoft.com/powershell
Depends: libc6, libgcc1, libgssapi-krb5-2, libstdc++6, zlib1g, libicu72|libicu71|libicu70|libicu69|libicu68|libicu67|libicu66|libicu65|libicu63|libicu60|libicu57|libicu55|libicu52, libssl3|libssl1.1|libssl1.0.2|libssl1.0.0
Vendor: Microsoft Corporation
License: MIT License
SHA256: 9132d97fea77d5a8de56b5801b7579bdc1ce669dae084295ddae6b43cf5a3fdb
Size: 69382904
Filename: pool/main/p/powershell/powershell_7.2.5-1.deb_amd64.deb

Proposed technical implementation details (optional)

No response

Implement Select-Text cmdlet

Unlike Select-String, this cmdlet would select off the Out-String output of an object. For tables, it should allow retaining the table header.

Side-By-Side comparison adds extraneous escape

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

$rightText = @("  This is other", "example text used!") -join [Environment]::NewLine
$leftText = @("This is some", "example text.") -join [Environment]::NewLine
compare-text $leftText $rightText -view side | out-string -str | %{$_.replace("`e","``e")}

Expected behavior

the escape sequence should be correct

Actual behavior

The output looks as follows:

`e[0m1 | `e[0mThis is `e[1;9;31msome`e[0m`e[0m `e[0m | `e[0;1;32m  `e[0mThis is `e[0;1;32mother`e…`e[0m
`e[0m`e[0m`e[1;9;31m`e[0m`e[0m`e[0m`e[0;1;32m`e[0m`e[0;1;32m`e[0m2 | `e[0mexample text`e[1;9;31m.`e[0m`e[0m | `e[0mexample text`e[0;1;32m…`e[0m

before the ellipsis of the first line of output there is an extraneous "`e" (or perhaps an escape sequence which is incomplete)

Error details

there is no error

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.0
PSEdition                      Core
GitCommitId                    7.3.0
OS                             Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Version

0.4.0 preview2

Visuals

No response

ConvertFrom-TextTable yum repolist --all fails

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

If you try to parse yum repolist --all | ConvertFrom-TextTable -Skip 4 it will error.

Here is the output from yum.

Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

repo id                               repo name                                                           status
packages-microsoft-com-prod           packages-microsoft-com-prod                                         enabled
ubi-8-appstream-debug-rpms            Red Hat Universal Base Image 8 (Debug RPMs) - AppStream             disabled
ubi-8-appstream-rpms                  Red Hat Universal Base Image 8 (RPMs) - AppStream                   enabled
ubi-8-appstream-source                Red Hat Universal Base Image 8 (Source RPMs) - AppStream            disabled
ubi-8-baseos-debug-rpms               Red Hat Universal Base Image 8 (Debug RPMs) - BaseOS                disabled
ubi-8-baseos-rpms                     Red Hat Universal Base Image 8 (RPMs) - BaseOS                      enabled
ubi-8-baseos-source                   Red Hat Universal Base Image 8 (Source RPMs) - BaseOS               disabled
ubi-8-codeready-builder               Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder           disabled
ubi-8-codeready-builder-debug-rpms    Red Hat Universal Base Image 8 (Debug RPMs) - CodeReady Builder     disabled
ubi-8-codeready-builder-rpms          Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder           enabled
ubi-8-codeready-builder-source        Red Hat Universal Base Image 8 (Source RPMs) - CodeReady Builder    disabled

Expected behavior

parses

Actual behavior

ConvertFrom-TextTable: Cannot process argument because the value of argument "name" is not valid. Change the value of the "name" argument and run the operation again.

Error details

Exception             :
    Type        : System.Management.Automation.PSArgumentException
    ErrorRecord :
        Exception             :
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Cannot process argument because the value of argument "name" is not valid. Change the value
 of the "name" argument and run the operation again.
            HResult : -2146233087
        CategoryInfo          : InvalidArgument: (:) [], ParentContainsErrorRecordException
        FullyQualifiedErrorId : Argument
    Message     : Cannot process argument because the value of argument "name" is not valid. Change the value of
the "name" argument and run the operation again.
    ParamName   : name
    TargetSite  : Void .ctor(System.String, System.Object)
    Source      : System.Management.Automation
    HResult     : -2147024809
    StackTrace  :
   at System.Management.Automation.PSNoteProperty..ctor(String name, Object value)
   at TextTableParser.ConvertTextTableCommand.GetPsObject(List`1 cInfos, String line, String[] columnHeaders) in
D:\a\_work\1\s\TextUtility\src\code\TextTableParser.cs:line 242
   at TextTableParser.ConvertTextTableCommand.Emit(String line) in D:\a\_work\1\s\TextUtility\src\code\TextTableP
arser.cs:line 167
   at TextTableParser.ConvertTextTableCommand.EndProcessing() in D:\a\_work\1\s\TextUtility\src\code\TextTablePar
ser.cs:line 150
   at System.Management.Automation.CommandProcessorBase.Complete()
CategoryInfo          : InvalidArgument: (:) [ConvertFrom-TextTable], PSArgumentException
FullyQualifiedErrorId : Argument,TextTableParser.ConvertTextTableCommand
InvocationInfo        :
    MyCommand        : ConvertFrom-TextTable
    ScriptLineNumber : 1
    OffsetInLine     : 22
    HistoryId        : 165
    Line             : yum repolist --all | ConvertFrom-TextTable -Skip 4
    PositionMessage  : At line:1 char:22
                       + yum repolist --all | ConvertFrom-TextTable -Skip 4
                       +                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    InvocationName   : ConvertFrom-TextTable
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1

Environment data

7.4

Version

0.5.0

Visuals

No response

ConvertFrom-TextTable dotnet tool search

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

If you run ConvertFrom-TextTable on output from dotnet tool search it will not accurately pick up the headers.

Package ID                        Latest Version      Authors                        Downloads      Verified

powershell                        7.3.6               Microsoft                      4376572        x
microsoft.dotnet-interactive      1.0.431302          Microsoft                      32442284       x
altcover.global                   8.6.68              Steve Gilham                   209871
ib                                5.10.4              Roman Kuzmin                   33355
newbe.build.psake                 3.0.1               Newbe                          13994
codecov.tool                      1.13.0              admiringworm, mathphysics      461968
utf8clip                          1.0.0               utf8clip                       614
ezdbcodegen.cli                   7.0.0               Ricardo Vega Jr.               13368
microsoft.psrule.tool             2.9.0               Microsoft                      2447           x
dacpac.tool                       3.0.512             Gustavo Américo                16465

Expected behavior

Package_ID                   Latest_Version Authors                   Downloads      Verified
----------                   -------------- -------                   ---------      --------
powershell                   7.3.6          Microsoft                 4376572        x
microsoft.dotnet-interactive 1.0.431302     Microsoft                 32442284       x
altcover.global              8.6.68         Steve Gilham              209871
ib                           5.10.4         Roman Kuzmin              33355
newbe.build.psake            3.0.1          Newbe                     13994
codecov.tool                 1.13.0         admiringworm, mathphysics 461968
utf8clip                     1.0.0          utf8clip                  614
ezdbcodegen.cli              7.0.0          Ricardo Vega Jr.          13368
microsoft.psrule.tool        2.9.0          Microsoft                 2447           x
dacpac.tool                  3.0.512        Gustavo Américo           16465

Actual behavior

dotnet tool search powershell | ConvertFrom-TextTable

Package_ID________________________Latest_Version______Authors________________________Downloads______Verified
------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
powershell                        7.3.6               Microsoft                      4376572        x
microsoft.dotnet-interactive      1.0.431302          Microsoft                      32442284       x
altcover.global                   8.6.68              Steve Gilham                   209871
ib                                5.10.4              Roman Kuzmin                   33355
newbe.build.psake                 3.0.1               Newbe                          13994
codecov.tool                      1.13.0              admiringworm, mathphysics      461968
utf8clip                          1.0.0               utf8clip                       614
ezdbcodegen.cli                   7.0.0               Ricardo Vega Jr.               13368
microsoft.psrule.tool             2.9.0               Microsoft                      2447           x
dacpac.tool                       3.0.512             Gustavo Américo                16465

### Error details

_No response_

### Environment data

```PowerShell
7.4-preview4

Version

0.5.0

Visuals

No response

Support Files in Base64 Cmdlets

More often than not, my use cases for base64 involve files. This leads to a large amount of boilerplate code. This could be reduced if support for reading from and writing to files is added to the Base64 cmdlets.

As a user, I would like to use ConvertFrom-Base64 to store the results in a file.

ConvertFrom-Base64 -Test 'This is a test' -OutFile 'test.base64.txt'

As a user, I would like to convert UTF-8 text contained in a file to Base64 using ConvertTo-Base64

ConvertTo-Base64 -TextFile 'I_Robot.txt'

As a user, I would like to convert a file containing binary data to Base64 using ConvertTo-Base64

ConvertTo-Base64 -BinaryFile 'kitten.png'

new table text converter should not include header separators

Summary of the new feature / enhancement

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
System Idle Process              0 Services                   0          8 K
System                           4 Services                   0      8,240 K

The output should not include the object with "==*" in all the fields

Proposed technical implementation details (optional)

those separators are really important for determining where the columns are, so it shouldn't happen before the columns are determined.

This is a test issue

Add Edit-File command

I would really love to see this command added. PS has been missing an easy way to modify files since v1. This comes in so handy for build systems. Our builds, first thing, edit a number of files to modify version numbers using this command e.g.:

# Update C# interop assembly version
Edit-FileContent -Path (Join-Path $rootPath 'src/Adapters/Acme.Wiley.Coyote.Api/Acme.Wiley.Coyote.Api.csproj') `
    -Pattern '(?<=\<Version\>)(.*?)(?=\</Version\>)' `
    -Replacement "${wileyVersion}.0"
Edit-FileContent -Path (Join-Path $rootPath 'src/Adapters/Acme.Wiley.Coyote.Api/Acme.Wiley.Coyote.Api.csproj') `
    -Pattern '(?<=\<InformationalVersion\>)(.*?)(?=\</InformationalVersion\>)' `
    -Replacement "${wileyVersion}${suffix}"

# WileyClient Conan C++ Support
Edit-FileContent -Path (Join-Path $rootPath 'src/WileyClient/conanfile.py') `
    -Pattern '(?<= version\s*=\s*'')(.*?)(?='')' `
    -Replacement "${wileyVersion}${suffix}"

# WileyManagement Conan C++ Support
Edit-FileContent -Path (Join-Path $rootPath 'src/WileyManagement/conanfile.py') `
    -Pattern '(?<= version\s*=\s*'')(.*?)(?='')' `
    -Replacement "${wileyVersion}${suffix}"
Edit-FileContent -Path (Join-Path $rootPath 'src/WileyManagement/conanfile.py') `
    -Pattern '(?<=WileyClient/)(.*?)(?=@Acme)' `
    -Replacement "${wileyVersion}${suffix}"

# NuGet C# Support
Edit-FileContent -Path (Join-Path $rootPath 'build/NuGet/WileyClientPkg/Acme.Wiley.Coyote.Api.nuspec') `
    -Pattern '(?<=\<version\>)(.*?)(?=\</version\>)' `
    -Replacement "${wileyVersion}${suffix}"
Edit-FileContent -Path (Join-Path $rootPath 'build/NuGet/WileyClientPkg/Acme.Wiley.Coyote.Api.nuspec') `
    -Pattern '(?<=id="runtime\.native\.Acme\.Wiley\.Coyote\.Api"\s+version=")(.*?)(?=")' `
    -Replacement "${wileyVersion}${suffix}"
Edit-FileContent -Path (Join-Path $rootPath 'build/NuGet/runtime.native.WileyClientPkg/runtime.native.Acme.Wiley.Coyote.Api.nuspec') `
    -Pattern '(?<=\<version\>)(.*?)(?=\</version\>)' `
    -Replacement "${wileyVersion}${suffix}"

# API Help Docs
$regexForHelpDocs = if ($Final) { '(?<== ")(\d)(.\d)*-?.* PRELIMINARY(?=")' } else { '(?<== ")(\d)(.\d)*-?.*(?= PRELIMINARY")' }
Edit-FileContent -Path (Join-Path $rootPath 'doc/CppReference/wiley-cpp-doxyfile') `
    -Pattern $regexForHelpDocs `
    -Replacement "${wileyVersion}${suffix}"

And here's the implementation - feel free to use as much (or little) as you want. https://github.com/Pscx/Pscx/blob/master/Src/Pscx/Commands/IO/EditFileCommand.cs

Publish releases

It would be great if releases could be published on GitHub so that it's easy to watch for new versions.

Implement ConvertFrom-Text

Equivalent to ConvertFrom-String for converting text output from native commands to an object, but will probably work differently than the WinPS cmdlet

Add Convert-LineEnding command

This would be like dos2unix and unix2dos. Running gci *.cpp | Convert-LineEnding with no "switch" parameter should convert line endings to match the current platform. I would expect -Windows and -Unix (or -Linux) mutually exclusive switches to force a specific line-ending independent of the current platform. Not sure if it would be worth adding -macOS switch (for \r line endings) since I "think" that is old, old macs and probably not relevant anymore.

Base64 cmdlets

Add parameterset to allow for conversion to and from byte array

ConvertTo-EOL cmdlet with -Unix and -Windows params

Often I use utilities like unixtodos.exe and dostounix.exe to fix the line endings of documents. It would be super helpful if PowerShell team consider it as a useful utility and provide it OOTB.

One solution is: https://stackoverflow.com/a/19128003 but it "almost" works as "mike z" mentioned.

Add Base64Url support

Base64Url support should be a built-in feature of PowerShell in my mind, but I could settle for importing a module that does it well.

There are a lot of blog posts showing tips on how to do it without importing a new module, and they all look like this:

It works, but it is horribly inefficient. I would rather have a Convert::ToBase64UrlString() function but we can't have everything 🤷

Base64Url is very common because of the popularity of JWTs, and everything that needs to be Base64 and URL-safe at the same time.

Implement Test-Spelling cmdlet

Consider adding Get-LoremIpsum command

Handy for generating random text. See https://github.com/Pscx/Pscx/blob/master/Src/Pscx/Commands/Text/GetLoremIpsumCommand.cs for a particular implementation. Feel free to borrow as much (or little) of this as you want.

TextTableParser namespace

Summary of the new feature / enhancement

Should TextTableParser namespace classes be in the module namespace Microsoft.PowerShell.TextUtility?

Proposed technical implementation details (optional)

No response

Implement ConvertTo-Sixel

Use https://github.com/saitoha/libsixel to support converting graphics to sixel