I was very, very tired of copying/pasting a template saying “Hi I’m Rex and I’m thinking about moving to NYC because of a job offer.. no pets, no kids, no wife, etc” to each place I found appealing. I decided to scrape craigslist and build PowerShell objects that I could sort and use to automagically spam my potential new roommates! 🙂
Here’s the script that’ll do the scraping and build an array of objects:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
$AvailableRooms = @() $URL = "http://newyork.craigslist.org" $MaxPages = 10 For ($CurrentPage=0;$CurrentPage -lt $MaxPages;$CurrentPage++) { $WebPage = Invoke-WebRequest "http://newyork.craigslist.org/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1" $Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" } ForEach ($Item in $Results) { $ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null $ID = ($Item -replace ".*pid\=`"","") -replace "`".*" $Price = ($Item -replace ".*class=price>","") -replace "</SPAN>.*" $DatePosted = ($Item -replace ".*class=date>","") -replace "</SPAN>.*" $Neighborhood = ($Item -replace ".*\<SMALL\>\(","") -replace "\)\</SMALL>.*" If ($Neighborhood -like "<*") { $Neighborhood = "N/A" } $Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0] $Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";" $Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1] $ItemObject = New-Object -TypeName PSObject -Property @{ 'ID' = $ID 'Price' = $Price 'DatePosted' = $DatePosted 'Neighborhood' = $Neighborhood 'Link' = $Link 'Description' = $Description 'E-Mail' = $Email } $AvailableRooms += $ItemObject } } |
And here’s what a sample object looks like, though you could probably guess by the object declaration:
Playing with data is fun, right?
There’s a ton that COULD be done with this, I’m only scratching the surface! 🙂
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
ForEach ($Room in $AvailableRooms) { $Body = @" Hi! My name is Rex and I'm looking for fun, sociable roommates! Quick Facts: - I am a non-smoker and do not have pets. - I am single with no baggage. - I have a stable job! - I love PowerShell - My phone number is 123-OMG-1337 Fun Fact - I automated this message! -- I have appended the following information for ease of reference --- Craiglist Posting: $($Room.Link) Room Description: $($Room.Description) Room Neighborhood: $($Room.Neighborhood) Room Price: $($Room.Price) "@ Send-MailMessage -Subject "Room for rent in $($Room.Neighborhood), listing ID $($Room.ID)" -Body $Body -To $Room.'E-Mail' -From masterrex@outlook.com -SmtpServer smtp.domain.com } |
———————————-Â UPDATE ———————————-
I’ve semi-wrapped it in a function for ease of use:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
Function Find-Apartment { param ( [Parameter(Mandatory=$False)]$MinPrice="0", [Parameter(Mandatory=$False)]$MaxPrice="9999", [Parameter(Mandatory=$False)]$MaxPages="1", [Parameter(Mandatory=$False)]$URL = "http://burlington.craigslist.org" ) $AvailableRooms = @() For ($CurrentPage=0;$CurrentPage -le $MaxPages;$CurrentPage++) { $WebPage = Invoke-WebRequest "$URL/search/roo?=roo&s=$Start&query=&zoomToPosting=&minAsk=$MinPrice&maxAsk=$MaxPrice&hasPic=1" $Results = $WebPage.ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "<P class=row*" } ForEach ($Item in $Results) { $ItemObject=$ID=$Price=$DatePosted=$Neighborhood=$Link=$Description=$Email=$null $ID = ($Item -replace ".*pid\=`"","") -replace "`".*" $Price = ($Item -replace ".*class=price>","") -replace "</SPAN>.*" $DatePosted = ($Item -replace ".*class=date>","") -replace "</SPAN>.*" $Neighborhood = ($Item -replace ".*\<SMALL\>\(","") -replace "\)\</SMALL>.*" If ($Neighborhood -like "<*") { $Neighborhood = "N/A" } $Link = $URL + ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[0] $Email = (($(Invoke-WebRequest $Link).ParsedHtml.body.innerHTML.Split("`n") | ? { $_ -like "var displayEmail*" }) -replace "var displayEmail \= `"") -replace "`";" $Description = ((($Item -replace ".*\<A href\=`"","") -replace "\<.*") -split('">'))[1] $ItemObject = New-Object -TypeName PSObject -Property @{ 'ID' = $ID 'Price' = $Price 'DatePosted' = $DatePosted 'Neighborhood' = $Neighborhood 'Link' = $Link 'Description' = $Description 'E-Mail' = $Email } $AvailableRooms += $ItemObject } } Return $AvailableRooms } |
22,149 total views, 6 views today
I love this! Great work! Question : how did you paste your code with this really cool formatting?
Hi Stephen,
Thanks! I use the Crayon Syntax Highlighter WordPress plugin.
It isn’t perfect though! If you know of a better solution please let me know =)
Rex
Hi Rex,
I really like your idea here and I’m trying to modify it for my own use. I’m having trouble understanding a few of the variables you are creating and getting them to work for me, the $Link,$Email,$Description. Can you explain what you are doing with these?
Great job on the post!