How can I store data from a CSV file into local variables for a Powershell script? - powershell

I'm trying to read a csv file and store its data into variables that I will use for Powershell scripts.
I managed to read the file by doing the following command:
$csv = Import-Csv C:\Users\[...]\Documents\ipaddresses.csv -delimiter "|"
$csv
The output looks like this:
IP Name
-- ----
165.XX.XX.1 n1
165.XX.XX.2 n2
165.XX.XX.3 n3
165.XX.XX.4 n4
165.XX.XX.5 n5
165.XX.XX.6 n6
The idea is that I could create 6 variables each like:
$Name = $IP
However, when I try to loop the objects of the $csv variable and store them into arrays like this:
foreach($item in $csv){
IP = $($item.IP)
Name = $($item.Name)
}
It returns me empty arrays. How do I fix this?

Taking your approach literal, you can use New-Variable, Get-Variable
## Q:\Test\2019\04\24\SO_55830910.ps1
$csv = Import-Csv $Env:USERPROFILE\Documents\ipaddresses.csv -delimiter "|"
foreach($item in $csv){
"{0} = {1}" -f $item.Name,$item.IP
New-Variable -Name $item.Name -Value $item.IP
}
1..6|%{Get-Variable "n$_"}
n1 = 165.XX.XX.1
n2 = 165.XX.XX.2
n3 = 165.XX.XX.3
n4 = 165.XX.XX.4
n5 = 165.XX.XX.5
n6 = 165.XX.XX.6
Name Value
---- -----
n1 165.XX.XX.1
n2 165.XX.XX.2
n3 165.XX.XX.3
n4 165.XX.XX.4
n5 165.XX.XX.5
n6 165.XX.XX.6

Your loop is returning nothing because you're not storing or yielding any result. Here's how it should be done :
$csv = Import-Csv C:\Users\[...]\Documents\ipaddresses.csv -delimiter "|"
$csv | Select-Object -Property IP,Name
# Or if you want a custom property
$csv | Select-Object -Property IP, #{Name="Custom Name", Expression = {"Hostname is : " + $_.Name}
Read more about Select-Object : https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/select-object?view=powershell-3.0

Related

Powershell - Prefix each line of Format-Table with String

I would like to know if there is an easy way of prefixing each line of a powershell table with a String.
For example, if I create an Array using the following code:
$Array = #()
$Object = #{}
$Object.STR_PARAM = "A"
$Object.INT_PARAM = 1
$Array += [PSCustomObject] $Object
$Object = #{}
$Object.STR_PARAM = "B"
$Object.INT_PARAM = 2
$Array += [PSCustomObject] $Object
Calling Format-Table give the following output:
$Array | Format-Table -AutoSize
STR_PARAM INT_PARAM
--------- ---------
A 1
B 2
Instead, I would like to have the following:
$Array | Format-Table-Custom -AutoSize -PrefixString " "
STR_PARAM INT_PARAM
--------- ---------
A 1
B 2
And if possible, I would also like to be able to use the Property parameter like this:
$SimpleFormat = #{Expression={$_.STR_PARAM}; Label="String Param"},
#{Expression={$_.INT_PARAM}; Label="Integer Param"};
$Array | Format-Table-Custom -Property $SimpleFormat -AutoSize -PrefixString "++"
++String Param Integer Param
++------------ -------------
++A 1
++B 2
Any help would be appreciated. Thanks.
You could just use the format expressions directly:
$f = #{Expression={"++" + $_.STR_PARAM}; Label="++String Param"},
#{Expression={$_.INT_PARAM}; Label="Integer Param"};
$Array | Format-Table $f -AutoSize
Output
++String Param Integer Param
-------------- -------------
++A 1
++B 2
Update to use expression and filter
Filter Format-Table-Custom
{
Param
(
[string]
$PrefixString,
[object]
$Property
)
end {
$rows = $input | Format-Table $property -AutoSize | Out-String
$lines = $rows.Split("`n")
foreach ($line in $lines) {
if ($line.Trim().Length -gt 0) {
$PrefixString + $line
}
}
}
}
$f = #{Expression={"--" + $_.STR_PARAM}; Label="--String Param"},
#{Expression={$_.INT_PARAM}; Label="Integer Param"};
$Array | Format-Table-Custom -Property $f -PrefixString "++"
Output
++--String Param Integer Param
++-------------- -------------
++--A 1
++--B 2

List down column headers and get the maximum length of string per column

I'm looking for a translation of my Excel formula in a form of a script in Powershell, vbscript or Excel VBA. I'm trying to get the list of column headers and the max length of string under it.
Normally, what I do is manually open the .txt file in Excel, from there I can get the header names.. next, I create an array formula =MAX(LEN(A1:A100,000)) for example. This will get the max length of string in the column. I'll do the same formula to other columns.
Right now I can't do this since files have increased to 1GB in size and i can't open them anymore, my desktop crashes. It is also maybe because theyre more than 1 million rows which Excel cant handle. My friend suggested Powershell but I have limited knowledge there.. don't know if it can be done in vbscript or Excel VBA.
Thanks in advance for your help.
Below code works for .csv files but does not with .txt delimited files -
$fileName = "C:\Desktop\EFile.csv"
<#
Sample format of c:\temp\data.csv
"id","name","grade","address"
"1","John","Grade-9","test1"
"2","Ben","Grade-9","test12222"
"3","Cathy","Grade-9","test134343"
#>
$colCount = (Import-Csv $fileName | Get-Member | Where-Object {$_.MemberType -eq 'NoteProperty'} | Measure-Object).Count
$csv = Import-Csv $fileName
$csvHeaders = ($csv | Get-Member -MemberType NoteProperty).name
$dict = #{}
foreach($header in $csvHeaders) {
$dict.Add($header,0)
}
foreach($row in $csv)
{
foreach($header in $csvHeaders)
{
if($dict[$header] -le ($row.$header).Length)
{
$dict[$header] =($row.$header).Length
}
}
}
$dict.Keys | % { "key = $_ , Column Length = " + $dict.Item($_) }
This is how I get my data.
$data = #"
"id","name","grade","address"
"1","John","Grade-9","test1"
"2","Ben","Grade-9","test12222"
"3","Cathy","Grade-9","test134343"
"#
$csv = ConvertFrom-Csv -Delimiter ',' $data
But you should get your data like this
$fileName = "C:\Desktop\EFile.csv"
$csv = Import-Csv -Path $fileName
And then
# Extract the header names
$headers = $csv | Get-Member -MemberType NoteProperty | Select-Object -ExpandProperty Name
# Capture output in $result variable
$result = foreach($header in $headers) {
# Select all items in $header column, find the longest, and select the item for output
$maximum = $csv | Select-Object -ExpandProperty $header | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
# Generate new object holding the information.
# This will end up in $results
[pscustomobject]#{
Header = $header
Max = $maximum.Length
String = $maximum
}
}
# Simple output
$result | Format-Table
This is what I get:
Header Max String
------ --- ------
address 10 test134343
grade 7 Grade-9
id 1 3
name 4 John
Alternatively, if you have memory issues dealing with large files, you may have to get a bit more dirty with the .NET framework. This snippet processes one csv line at a time, instead of reading the entire file into memory.
$fileName = "$env:TEMP\test.csv"
$delimiter = ','
# Open a StreamReader
$reader = [System.IO.File]::OpenText($fileName)
# Read the headers and turn it into an array, and trim away any quotes
$headers = $reader.ReadLine() -split $delimiter | % { $_.Trim('"''') }
# Prepare a hashtable for the results
$result = #{}
# So long as there's more data, keep running
while(-not $reader.EndOfStream) {
# Read a single line and process it as csv
$csv = $reader.ReadLine() | ConvertFrom-Csv -Header $headers -Delimiter $delimiter
# Determine if the item in the result hashtable is smaller than the current, using the header as a key
foreach($header in $headers) {
$item = $csv | Select-Object -ExpandProperty $header
if($result[$header].Maximum -lt $item.Length) {
$result[$header] = [pscustomobject]#{
Header = $header
Maximum = $item.Length
String = $item
}
}
}
}
# Clean up our spent resource
$reader.Close()
# Simple output
$result.Values | Format-Table

PowerShell - transpose results from a hashtable

I need to check the warranty of many servers, but the output returned by the module I found in https://www.powershellgallery.com/packages/HPWarranty/2.6.2 seems to be a hashtable and the first column contains what I want to be my rows.
the script below will return this where every 5 rows the fields repeat - output1.csv:
TYPE System.Management.Automation.PSCustomObject
"Component","Codecount"
"SerialNumber","CZ36092P5H"
"ProductNumber","727021-B21"
"OverallEntitlementStartDate","2016-03-04"
"OverallEntitlementEndDate","2019-04-02"
"ActiveEntitlement","true"
"SerialNumber","CZ36092P5K"
"ProductNumber","727021-B21"
"OverallEntitlementStartDate","2016-03-04"
"OverallEntitlementEndDate","2019-04-02"
"ActiveEntitlement","true"
How can I transpose the output so that SerialNumber, ProductNumber, OverallEntitlementStartDate, OverallEntitlementEndDate and ActiveEntitlement are the columns?
# variables
$dest_path = "C:\Scripts\HPE\HPWarranty"
$export_date = Get-Date -Format o | ForEach-Object {$_ -replace ":", "-"}
$myScriptName = $MyInvocation.MyCommand.Name
$transcriptPath = $dest_path + "\" + $myScriptName + "_transcript_" + $export_date + ".txt"
$csvPath = $dest_path + "\" + "hpe_list1.csv"
#Start transcript of script activities and set transcript location
start-transcript -append -path $transcriptPath | Out-Null
# import serials & part numbers to be processed
$csv_info = Import-Csv $csvPath
foreach ($line in $csv_info) {
$hash = (Get-HPEntWarrantyEntitlement -ProductNumber $line.ProductNumber -SerialNumber $line.SerialNumber)
&{$hash.getenumerator() |
ForEach-Object {new-object psobject -Property #{Component = $_.name;Codecount=$_.value}}
} | Export-Csv "C:\Scripts\HPE\HPWarranty\output1.csv" -Append
}
# Stop Transcript
Stop-Transcript | Out-Null
hpe_list1.csv that the script processes contains the details for two servers:
ProductNumber,SerialNumber
727021-B21,CZ36092P5H
727021-B21,CZ36092P5K
Cast the output hashtable to a [pscustomobject]:
$WarrantyInfo = foreach ($line in $csv_info) {
[pscustomobject](Get-HPEntWarrantyEntitlement -ProductNumber $line.ProductNumber -SerialNumber $line.SerialNumber)
}
$WarrantyInfo | Export-Csv "C:\Scripts\HPE\HPWarranty\output1.csv"

Powershell - Insert column in between specific columns in csv file

I have 2 csv files
First file:
firstName,secondName
1234,Value1
2345,Value1
3456,Value1
4567,Value3
7645,Value3
Second file:
firstName,fileSplitter,Csv2ColumnOne,Csv2ColumnTwo,Csv2ColumnThree
1234,,1234,abc,Value1
1234,,1234,asd,Value1
3456,,3456,qwe,Value1
4567,,4567,mnb,Value1
I want to insert column secondName in the second file in between columns firstName and fileSplitter.
The result should look like this:
firstName,secondName,fileSplitter,Csv2ColumnOne,Csv2ColumnTwo,Csv2ColumnThree
1234,Value1,,1234,abc,Value1
1234,Value1,,1234,asd,Value1
3456,Value1,,3456,qwe,Value1
4567,Value3,,4567,mnb,Value1
I'm trying the following code:
Function InsertColumnInBetweenColumns
{
Param ($FirstFileFirstColumnTitle, $firstFile, [string]$1stColumnName, [string]$2ndColumnName, [string]$columnMergedFileBeforeInput)
Write-Host "Creating hash table with columns values `"$1stColumnName`" `"$2ndColumnName`" From $OimFileWithMatches"
$hashFirstFileTwoColumns = #{}
Import-Csv $firstFile | ForEach-Object {$hashFirstFileTwoColumns[$_.$1stColumnName] = $_.$2ndColumnName}
Write-Host "Complete."
Write-Host "Appending Merge file with column `"$2ndColumnName`" from file $secondCsvFileWithLocalPath"
Import-Csv $outputCsvFileWithLocalPath | Select-Object $columnMergedFileBeforeInput, #{n=$2ndColumnName; e={
if ($hashFirstFileTwoColumns.ContainsKey($_.$FirstFileFirstColumnTitle)) {
$hashFirstFileTwoColumns[$_.$FirstFileFirstColumnTitle]
} Else {
'Not Found'
}}}, * | Export-Csv "$outputCsvFileWithLocalPath-temp" -NoType -Force
Move-Item "$outputCsvFileWithLocalPath-temp" $outputCsvFileWithLocalPath -Force
Write-Host "Complete."
Write-Host ""
}
This function will be called in a for loop for each column found in the first file (can contain an indefinite number). For testing, I am only using 2 columns from the first file.
I'm getting an error output resulting the following:
Select : Property cannot be processed because property "firstName" already exists.
At C:\Scripts\Tests\Compare2CsvFilesOutput1WithMatchesOnly.ps1:490 char:43
+ Import-Csv $outputCsvFileWithLocalPath | Select $columnMergedFileBeforeInput, # ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (#{firstName=L...ntName=asdfas}:PSObject) [Select-Object], PSArgume
ntException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyNoExpand,Microsoft.PowerShell.Commands.SelectObjectC
ommand
I know the issue is where it says Select-Object $columnMergedFileBeforeInput,.
How can I get the loop statement to insert the column in between the before column (name is specified), and append the rest using *?
Update
Just an fyi, changing this line Select-Object $columnMergedFileBeforeInput, #{n=$2ndColumnName..... to this line Select-Object #{n=$2ndColumnName..... works, it just attaches the columns out of order. That is why I'm trying to insert the column in between. Maybe if i do it this way but insert the columns in backwards using the for loop, this would work...
Not sure if this is the most efficient way to do it, but it should do the trick. It just adds the property to the record from file2, then reorders the output so secondName is the second column. You can output results to csv where required too (ConvertTo-Csv).
$file1 = Import-Csv -Path file1.csv
$file2 = Import-Csv -Path file2.csv
$results = #()
ForEach ($record In $file2) {
Add-Member -InputObject $record -MemberType NoteProperty -Name secondName -Value $($file1 | ? { $_.firstName -eq $record.firstName } | Select -ExpandProperty secondName)
$results += $record
}
$results | Select-Object -Property firstName,secondName,fileSplitter,Csv2ColumnOne,Csv2ColumnTwo,Csv2ColumnThree
I've created the following function. What it does is find the match (in this case "firstname") and adds the matching columnname to the new array afther the columnname on which the match is made (little difficult to explain in my poor English).
function Add-ColumnAfterMatchingColumn{
[CmdletBinding()]
param(
[string]$MainFile,
[string]$MatchingFile,
[string]$MatchColumnName,
[string]$MatchingColumnName
)
# Import data from two files
$file1 = Import-Csv -Path $MainFile
$file2 = Import-Csv -Path $MatchingFile
# Find column names and order them
$columnnames = $file2 | gm | where {$_.MemberType -like "NoteProperty"} | Select Name | %{$_.Name}
[array]::Reverse($columnnames)
# Find $MatchColumnName index and put the $MatchingColumnName after it
$MatchColumnNameIndex = [array]::IndexOf($columnnames, $MatchColumnName)
if($MatchColumnNameIndex -eq -1){
$MatchColumnNameIndex = 0
}
$columnnames = $columnnames[0..$MatchColumnNameIndex] + $MatchingColumnName + $columnnames[($MatchColumnNameIndex+1)..($columnnames.Length -1)]
$returnObject = #()
foreach ($item in $file2){
# Find corresponding value MatchingColumnName in $file1 and add it to the current item
$item | Add-Member -Name "$MatchingColumnName" -Value ($file1 | ?{$_."$($MatchColumnName)" -eq $item."$($MatchColumnName)"})."$MatchingColumnName" -MemberType NoteProperty
# Add current item to the returnObject array, in the correct order
$newItem = New-Object psobject
foreach ($columnname in [string[]]$columnnames){
$newItem | Add-Member -Name $columnname -Value $item."$columnname" -MemberType NoteProperty
}
$returnObject += $newItem
}
return $returnObject
}
When you run this function you will have the following output:
Add-ColumnAfterMatchingColumn -MainFile C:\Temp\file1.csv -MatchingFile C:\Temp\file2.csv -MatchColumnName "firstname" -MatchingColumnName "secondname" | ft
firstName secondname fileSplitter Csv2ColumnTwo Csv2ColumnThree Csv2ColumnOne
--------- ---------- ------------ ------------- --------------- -------------
1234 Value1 abc Value1 1234
1234 Value1 asd Value1 1234
3456 Value1 qwe Value1 3456
4567 Value3 mnb Value1 4567

PowerShell Import-Csv Issue - Why is my output being treated as a single column and not a CSV?

So I have a CSV file which I need to manipulate a bit, select the data I need and export to another CSV file.
The code I have is:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
Import-Csv -Header #("a","b","c","d") -Path $rawCSV |
select -Skip 7 |
Where-Object { $_.b.length -gt 1 } |
ft b,a,c,d |
Out-File $outputCSV
So this code uses the Import-Csv command to allow me to select just the columns I need, add some headers in the order I want and then I am simply putting the output in to a CSV file called $outputCSV. The contents of this output file look something like this:
b a c d
- - - -
john smith 29 England
mary poopins 79 Walton
I am not sure what the delimiter is in this output and rather than these columns being treated as individuals, they are treated as just one column. I have gone on further to replace all the spaces with a comma using the code:
$b = foreach ($line in $a)
{
$fields = $line -split '`n'
foreach ($field in $fields)
{
$field -replace " +",","
}
}
Which produces a file that looks like this:
b,a,c,d
john,smith,29,England
mary,poppins,79,Walton
But these are all still treated as one column instead of four separate columns as I need.
* UPDATE *
Using the answer given by #, I now get a file looking like this:
Don't use ft to reorder your columns - it's intended to format output for the screen, not really suitable for CSV.
"Manual" solution:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Write your headers to the output file
"b","a","c","d" -join ',' | Out-File $outputCSV -Force
$ReorderedData = foreach($Row in $Data){
# Reorder the columns in each row
'{0},{1},{2},{3}' -f $Row.b , $Row.a , $Row.c, $Row.d
}
# Write the reordered rows to the output file
$ReorderedData | Out-File $outputCSV -Append
Using Export-Csv:
As of PowerShell 3.0, you could also push the rows into a [pscustomobject] and pipe that to Export-Csv (pscustomobject preserves the order in which you supply the properties):
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Take the columns you're interested in, put them into new custom objects and export to CSV
$Data | ForEach-Object {
[pscustomobject]#{ "b" = $_.b; "a" = $_.a; "c" = $_.c; "d" = $_.d }
} | Export-Csv -NoTypeInformation $outputCSV
Export-Csv will take care of enclosing strings in quotes to escape ',' properly (one thing less for you to worry about)
First of all, what your raw CSV file looks like? If it's already like this
john,smith,29,England
mary,poppins,79,Walton
then import-csv will give you an array of objects which you can easily manipulate (and objects are the main reason to use PowerShell ;). For example, to check what you have after import:
$r = Import-Csv -Path $rawCSV -Header #("b","a","c","d")
$r.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
$r[0] | get-member
TypeName: System.Management.Automation.PSCustomObject
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
a NoteProperty System.String a=smith
b NoteProperty System.String b=john
c NoteProperty System.String c=29
d NoteProperty System.String d=England
For now you have array of objects with properties named "a","b","c","d". To manipulate objects you have select-object cmdlet:
$r | Select-Object a,b,c,d
a b c d
- - - -
smith john 29 England
poppins mary 79 Walton
And after all use export-csv to set the output file:
$r | where { $_.b.length -gt 1 } |
select a,b,c,d |
Export-Csv -NoTypeInformation -Encoding utf8 -path $outputCSV
I could think of two possible reasons why your data teated as one column:
consuming application expect different encoding and can't find
delimiters
delimiters are not commas but something else

Resources