Category Archives: PowerShell

Powershell scripts, snippets and nuggets of knowledge.

Automic Introspective REST Agent + Small life update

PowerShell + Craigslist for advanced apartment hunting

I was very, very tired of copying/pasting a template saying “Hi I’m Rex and I’m thinking about moving to NYC because of a job offer.. no pets, no kids, no wife, etc” to each place I found appealing. I decided to scrape craigslist and build PowerShell objects that I could sort and use to automagically spam my potential new roommates! 🙂

Here’s the script that’ll do the scraping and build an array of objects:

$AvailableRooms = @()
$URL = "http://newyork.craigslist.org"
$MaxPages = 10
For ($CurrentPage=0;$CurrentPage -lt $MaxPages;$CurrentPage++) {
    $WebPage = Invoke-WebRequest "http://newyork.craigslist.org/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1"
    $Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" }
    ForEach ($Item in $Results) { 
        $ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null
        $ID = ($Item -replace ".*pid\=`"","") -replace "`".*"
        $Price = ($Item -replace ".*class=price>","") -replace "</SPAN>.*"
        $DatePosted = ($Item -replace ".*class=date>","") -replace "</SPAN>.*"
        $Neighborhood = ($Item -replace ".*\<SMALL\>\(","") -replace "\)\</SMALL>.*"
        If ($Neighborhood -like "<*") { $Neighborhood = "N/A" } 
        $Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0]
        $Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";"
        $Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1]
        $ItemObject = New-Object -TypeName PSObject -Property @{
            'ID' = $ID
            'Price' = $Price
            'DatePosted' = $DatePosted
            'Neighborhood' = $Neighborhood
            'Link' = $Link
            'Description' = $Description
            'E-Mail' = $Email
        }
        $AvailableRooms += $ItemObject
    }
}

$AvailableRooms = @()

$URL = "http://newyork.craigslist.org"

$MaxPages = 10

For ($CurrentPage=0;$CurrentPage -lt $MaxPages;$CurrentPage++) {

$WebPage = Invoke-WebRequest "http://newyork.craigslist.org/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1"

$Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" }

ForEach ($Item in $Results) {

$ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null

$ID = ($Item -replace ".*pid\=`"","") -replace "`".*"

$Price = ($Item -replace ".*class=price>","") -replace ".*"

$DatePosted = ($Item -replace ".*class=date>","") -replace ".*"

$Neighborhood = ($Item -replace ".*\<SMALL\>$","") -replace "$\.*"

If ($Neighborhood -like "<*") { $Neighborhood = "N/A" }

$Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0]

$Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";"

$Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1]

$ItemObject = New-Object -TypeName PSObject -Property @{

'ID' = $ID

'Price' = $Price

'DatePosted' = $DatePosted

'Neighborhood' = $Neighborhood

'Link' = $Link

'Description' = $Description

'E-Mail' = $Email

}

$AvailableRooms += $ItemObject

}

And here’s what a sample object looks like, though you could probably guess by the object declaration:

Playing with data is fun, right?
Sample list of objects

There’s a ton that COULD be done with this, I’m only scratching the surface! 🙂

ForEach ($Room in $AvailableRooms) {  
    $Body = @"
Hi! My name is Rex and I'm looking for fun, sociable roommates! 

Quick Facts:
 - I am a non-smoker and do not have pets.       
 - I am single with no baggage.
 - I have a stable job!
 - I love PowerShell 
 - My phone number is 123-OMG-1337

Fun Fact - I automated this message!

-- I have appended the following information for ease of reference ---

Craiglist Posting: $($Room.Link)
Room Description:  $($Room.Description)
Room Neighborhood: $($Room.Neighborhood)
Room Price:  $($Room.Price)

"@
    Send-MailMessage -Subject "Room for rent in $($Room.Neighborhood), listing ID $($Room.ID)" -Body $Body -To $Room.'E-Mail' -From masterrex@outlook.com -SmtpServer smtp.domain.com 
}

ForEach ($Room in $AvailableRooms) {

$Body = @"

Hi! My name is Rex and I'm looking for fun, sociable roommates!

Quick Facts:

- I am a non-smoker and do not have pets.

- I am single with no baggage.

- I have a stable job!

- I love PowerShell

- My phone number is 123-OMG-1337

Fun Fact - I automated this message!

-- I have appended the following information for ease of reference ---

Craiglist Posting: $($Room.Link)

Room Description: $($Room.Description)

Room Neighborhood: $($Room.Neighborhood)

Room Price: $($Room.Price)

Send-MailMessage -Subject "Room for rent in $($Room.Neighborhood), listing ID $($Room.ID)" -Body $Body -To $Room.'E-Mail' -From masterrex@outlook.com -SmtpServer smtp.domain.com

}

———————————- UPDATE ———————————-

I’ve semi-wrapped it in a function for ease of use:

Function Find-Apartment { 
    param (
        [Parameter(Mandatory=$False)]$MinPrice="0",
        [Parameter(Mandatory=$False)]$MaxPrice="9999",
        [Parameter(Mandatory=$False)]$MaxPages="1",
        [Parameter(Mandatory=$False)]$URL = "http://burlington.craigslist.org"
    )
    $AvailableRooms = @()
    For ($CurrentPage=0;$CurrentPage -le $MaxPages;$CurrentPage++) {
        $WebPage = Invoke-WebRequest "$URL/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1"
        $Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" }
        ForEach ($Item in $Results) { 
            $ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null
            $ID = ($Item -replace ".*pid\=`"","") -replace "`".*"
            $Price = ($Item -replace ".*class=price>","") -replace "</SPAN>.*"
            $DatePosted = ($Item -replace ".*class=date>","") -replace "</SPAN>.*"
            $Neighborhood = ($Item -replace ".*\<SMALL\>\(","") -replace "\)\</SMALL>.*"
            If ($Neighborhood -like "<*") { $Neighborhood = "N/A" } 
            $Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0]
            $Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";"
            $Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1]
            $ItemObject = New-Object -TypeName PSObject -Property @{
                'ID' = $ID
                'Price' = $Price
                'DatePosted' = $DatePosted
                'Neighborhood' = $Neighborhood
                'Link' = $Link
                'Description' = $Description
                'E-Mail' = $Email
            }
            $AvailableRooms += $ItemObject
        }
    }
    Return $AvailableRooms
}

Function Find-Apartment {

param (

[Parameter(Mandatory=$False)]$MinPrice="0",

[Parameter(Mandatory=$False)]$MaxPrice="9999",

[Parameter(Mandatory=$False)]$MaxPages="1",

[Parameter(Mandatory=$False)]$URL = "http://burlington.craigslist.org"

)

$AvailableRooms = @()

For ($CurrentPage=0;$CurrentPage -le $MaxPages;$CurrentPage++) {

$WebPage = Invoke-WebRequest "$URL/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1"

$Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" }

ForEach ($Item in $Results) {

$ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null

$ID = ($Item -replace ".*pid\=`"","") -replace "`".*"

$Price = ($Item -replace ".*class=price>","") -replace ".*"

$DatePosted = ($Item -replace ".*class=date>","") -replace ".*"

$Neighborhood = ($Item -replace ".*\<SMALL\>$","") -replace "$\.*"

If ($Neighborhood -like "<*") { $Neighborhood = "N/A" }

$Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0]

$Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";"

$Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1]

$ItemObject = New-Object -TypeName PSObject -Property @{

'ID' = $ID

'Price' = $Price

'DatePosted' = $DatePosted

'Neighborhood' = $Neighborhood

'Link' = $Link

'Description' = $Description

'E-Mail' = $Email

}

$AvailableRooms += $ItemObject

}

Return $AvailableRooms

}

Using Find-Apartment

14,007 total views, 1 views today

OCR with Powershell

5 Replies

I wrote a little function that utilizes Microsoft Office Document Imaging (MODI) to retrieve text from images with OCR.

I have put a few notes in-line in the script and have dummy-proofed it somewhat, but ymmv! Below the snippet I’ll show an example where I compare 12pt font recognition with this technique.

Function Get-TextFromImage {
    param ( 
        [Parameter(Mandatory=$true)][string]$File
    )

    # Requires 32bit Powershell Session
    # Requires Office 2003 or 2007 Installation
    Try { 
        $FileObj = Get-ChildItem $File
        $FilePath = $FileObj.Fullname
        If ($FileObj.Extension -ne ".jpg" -or ".jpeg" -or ".bmp") {
            Write-Warning "$File does not have a known working extension, trying anyway..."
        }
    } Catch {
        Write-Error "Error Retrieving $File:"
        Write-Error $_
        Exit 1
    }

    Try {
        $MODIObj = New-Object -ComObject MODI.Document
    } Catch {
        Write-Error "Could not load MODI Com Object. Make sure you are running a 32bit Powershell sessio and have MS Office 2003/2007 installed."
        Write-Error "If you have MS Office 2010, try this: http://support.microsoft.com/kb/982760"
        Write-Error $_
        Exit 2
    }

    Try { 
        $MODIObj.Create($FilePath)
        $MODIObj.OCR()
        $Output = $MODIObj.Images.Item(0).Layout.Text.ToString().Trim()
        Write-Output $Output | Tee-Object -File $($FilePath + ".ocr.txt")
    } Catch {
        Write-Error "Failed to process file:"
        Write-Error $_
        Exit 3
    }   
}

Function Get-TextFromImage {

param (

[Parameter(Mandatory=$true)][string]$File

)

# Requires 32bit Powershell Session

# Requires Office 2003 or 2007 Installation

Try {

$FileObj = Get-ChildItem $File

$FilePath = $FileObj.Fullname

If ($FileObj.Extension -ne ".jpg" -or ".jpeg" -or ".bmp") {

Write-Warning "$File does not have a known working extension, trying anyway..."

}

} Catch {

Write-Error "Error Retrieving $File:"

Write-Error $_

Exit 1

}

Try {

$MODIObj = New-Object -ComObject MODI.Document

} Catch {

Write-Error "Could not load MODI Com Object. Make sure you are running a 32bit Powershell sessio and have MS Office 2003/2007 installed."

Write-Error "If you have MS Office 2010, try this: http://support.microsoft.com/kb/982760"

Write-Error $_

Exit 2

}

Try {

$MODIObj.Create($FilePath)

$MODIObj.OCR()

$Output = $MODIObj.Images.Item(0).Layout.Text.ToString().Trim()

Write-Output $Output | Tee-Object -File $($FilePath + ".ocr.txt")

} Catch {

Write-Error "Failed to process file:"

Write-Error $_

Exit 3

}

Here’s an example:

Image	Get-TextFromImage Output
	Windows Powershell NODI OCR Test Image l2pt COURIER NEW ABCDEFGHIJKLMNOPORSTUVLJXYZ abcdefghijklmnopqrstuvwxyz 01234567890 12 pt TAHOMA ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789 12 pt TERMINAL ABCDEFCH I JICLMNOPQRSTUUIIXYZ abc de f gh ii Ic inn o pqrs t tw wxyz 0123456789 12 pt VERDANA ABCD EFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789 12 pt CONSOLAS ABCDE FCHIJ KL KNOPQRSTUVHXYZ abcdefghij kirnnopqrst uvwxyz 0123456789 12 pt Times New Roman ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklnmopqrstuvwxyz 0123456739 ] pt OCR?A Extended ABCDEFGHI JKL1NOPQRSTUVLdXYZ abcde fghij kimnopqrstuvwxy z O]3456789

Image

Get-TextFromImage Output

Windows Powershell NODI OCR Test Image
l2pt COURIER NEW ABCDEFGHIJKLMNOPORSTUVLJXYZ
abcdefghijklmnopqrstuvwxyz
01234567890

12 pt TAHOMA
ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789

12 pt TERMINAL
ABCDEFCH I JICLMNOPQRSTUUIIXYZ
abc de f gh ii Ic inn o pqrs t tw wxyz
0123456789

12 pt VERDANA
ABCD EFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789

12 pt CONSOLAS
ABCDE FCHIJ KL KNOPQRSTUVHXYZ
abcdefghij kirnnopqrst uvwxyz
0123456789

12 pt Times New Roman
ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklnmopqrstuvwxyz
0123456739

] pt OCR?A Extended ABCDEFGHI JKL1NOPQRSTUVLdXYZ
abcde fghij kimnopqrstuvwxy z
O]3456789

The OCR-specific font failed miserably. Funny huh?

It appears that at 12pt in a jpg, Times New Roman is the best candidate for OCR using MODI via Powershell if you intend on having accurate results!

Relevant links:
http://stackoverflow.com/questions/316068/what-is-the-ideal-font-for-ocr
http://cerealnumber.livejournal.com/47638.html
http://stackoverflow.com/questions/9277571/how-can-i-retrieve-modi-reference-from-com-in-my-application

15,645 total views, 4 views today