Crawl & Scrape with Node.js

I’m a bit obsessed with web crawlers – I find them strangely fascinating. Previously I wrote a tutorial about building a web crawler that parses into a .csv file with Ruby, read the tutorial here – Build a Web Crawler in 5 Easy Steps. It’s literally 15 lines of code to get a basic Ruby crawler running.

Now having learnt some Node.js, I decided to write a JavaScript web crawler. I used the node-crawler npm package, similar to the spidr ruby gem. In the Ruby crawler I wrote, Nokogiri is used to parse the page using Xpath. For the Node.js crawler, Cheerio allows you to use jQuery selectors to grab HTML attributes, text, classes and more.

Here’s the repo, or continue reading for the code snippet.

 

This is the output:

Salies-MacBook-Pro:node-crawler salie$ node crawler.js
URL: https://en.wikipedia.org/wiki/Main_Page
Title: Wikipedia, the free encyclopedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Main Page
H2: From today's featured articleDid you know...In the newsOn this day...Today's
featured pictureOther areas of WikipediaWikipedia's sister projectsWikipedia la
nguagesNavigation menu
---------
URL: https://en.wikipedia.org/wiki/Hit_(baseball)
Title: Hit (baseball) - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Hit (baseball)
H2: ContentsScoring a hit[edit]Pitching a no-hitter[edit]History[edit]Major Leag
ue Baseball rules[edit]See also[edit]References[edit]Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Bat-and-ball_games
Title: Bat-and-ball games - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Bat-and-ball games
H2: ContentsList of bat-and-ball games[edit]References[edit]External links[edit]
Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Run_(baseball)
Title: Run (baseball) - Wikipedia
URL: https://en.wikipedia.org/wiki/Run_(baseball)
Title: Run (baseball) - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Run (baseball)
H2: ContentsSignificant run scoring records[edit]See also[edit]References[edit]E
xternal links[edit]Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Free_content
Title: Free content - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Free content
H2: ContentsDefinition[edit]Legal matters[edit]Usage[edit]See also[edit]Notes[ed
it]References[edit]Further reading[edit]External links[edit]Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Baseball_(ball)
Title: Baseball (ball) - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Baseball (ball)
H2: ContentsHistory[edit]Overview[edit]Famous baseballs[edit]See also[edit]Notes
and references[edit]External links[edit]Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Base_(baseball)
Title: Baseball field - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Baseball field
H2: ContentsSpecifications[edit]First base[edit]Second base[edit]Third base[edit
]Home plate[edit]Batter's box and catcher's box[edit]Foul poles[edit]Pitcher's m
ound[edit]Baseline[edit]Grass line[edit]Outfield[edit]Warning track[edit]Outfiel
d wall[edit]Bullpen[edit]On-deck circles[edit]Coach's boxes[edit]History[edit]Ma
intenance[edit]Honors and awards[edit]References[edit]External links[edit]Naviga
tion menu
---------
URL: https://en.wikipedia.org/wiki/Encyclopedia
Title: Encyclopedia - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Encyclopedia
H2: ContentsEtymologyCharacteristicsHistorySee alsoNotesReferencesExternal links
Navigation menu
---------
URL: https://en.wikipedia.org/wiki/Baseball
Title: Baseball - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Baseball
H2: ContentsHistoryRules and gameplayPersonnelStrategy and tacticsDistinctive el
ementsStatisticsPopularity and cultural impactSee alsoReferencesSourcesFurther r
eadingExternal linksNavigation menu
---------
URL: https://en.wikipedia.org/wiki/English_language
Title: English language - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: English language
H2: ContentsClassificationHistoryGeographical distributionPhonologyGrammarVocabu
laryWriting systemDialects, accents, and varietiesReferencesBibliographyExternal
linksNavigation menu
---------
URL: https://en.wikipedia.org/wiki/Wikipedia
Title: Wikipedia - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Wikipedia
H2: ContentsHistoryOpennessPolicies and lawsGovernanceCommunityLanguage editions
Critical receptionOperationAccess to contentCultural impactRelated projectsSee a
lsoReferencesFurther readingExternal linksNavigation menu
---------
URL: https://en.wikipedia.org/wiki/Out_(baseball)
Title: Out (baseball) - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Out (baseball)
H2: ContentsScoring[edit]See also[edit]Further reading[edit]References[edit]Navi
gation menu
---------
URL: https://en.wikipedia.org/wiki/Home_plate
Title: Baseball field - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Baseball field
H2: ContentsSpecifications[edit]First base[edit]Second base[edit]Third base[edit
]Home plate[edit]Batter's box and catcher's box[edit]Foul poles[edit]Pitcher's m
ound[edit]Baseline[edit]Grass line[edit]Outfield[edit]Warning track[edit]Outfiel
d wall[edit]Bullpen[edit]On-deck circles[edit]Coach's boxes[edit]History[edit]Ma
intenance[edit]Honors and awards[edit]References[edit]External links[edit]Naviga
tion menu
---------
URL: https://en.wikipedia.org/wiki/Pitcher
Title: Pitcher - Wikipedia
Meta Description: undefined
OG Title: undefined
OG Description: undefined
H1: Pitcher
H2: ContentsOverview[edit]Pitching in a game[edit]Rotation and specialization[ed
it]After the ball is pitched[edit]Pitching biomechanics[edit]Equipment[edit]See
also[edit]References[edit]Further reading[edit]Navigation menu
..........

Leave a Reply

Your email address will not be published. Required fields are marked *