Episode #100: Screen Scraping Gateway

Have you ever needed to incorporate information from third-party website, and found that they don't offer any kind of API? Sometimes the only way to get at the data you need is by good old-fashioned screen-scraping: pulling in webpages intended for human readers, and interpreting them programmatically. In this episode, you'll learn how to use the Ruby "Mechanize" gem to easily interact with and extract data from a website. But much more importantly, you'll learn how to use the Gateway Pattern to isolate the rest of your code from the complexity and inherent fragility of screen-scraping. And you'll see how to accomplish all this using test-driven development combined with interactive exploratory coding.

This page is just for members. Sign in or subscribe to gain access!