Webscraping in R - IMDb ETL Showcase

 

Web scraping in R is an ETL pipeline that perform web data mining by reading HTML tags and converting them to the structured format which can easily be visualized using tidyverse. Let's scrape movies from IMDb into a data frame in R by invoking the rvest library and then visualize the data frame using ggplot2 and qplot functions:

• Importing the key R libraries

library(rvest) #scraping

library(dplyr) #piping

library('ggplot2') #plotting

• Specifying the URL for desired website to be scraped

url <- 'http://www.imdb.com/search/title?count=100&release_date=2016,2016&title_type=feature'

• Reading the HTML code from the website

webpage <- read_html(url)

• Using CSS selectors to scrape the rankings section

rank_data_html <- html_nodes(webpage,'.text-primary')

• Converting the ranking data to text

rank_data <- html_text(rank_data_html)

• Let's have a look at the rankings

head(rank_data)

[1] "1." "2." "3." "4." "5." "6."

•  Using CSS selectors to scrape the title section

title_data_html <- html_nodes(webpage,'.lister-item-header a')

• Converting the title data to text

title_data <- html_text(title_data_html)

• Let's have a look at the title type it in R console

head(title_data)

[1] "Fantastic Beasts and Where to Find Them"            

[2] "Batman v Superman: Dawn of Justice Ultimate Edition"

[3] "Suicide Squad"                                      

[4] "Doctor Strange"                                     

[5] "Deadpool"                                           

[6] "Ah-ga-ssi"      

• Using CSS selectors to scrape the description section

description_data_html <- html_nodes(webpage,'.ratings-bar+ .text-muted')

• Converting the description data to text

description_data <- html_text(description_data_html)

• Data-Preprocessing: removing '\n'

description_data<-gsub("\n","",description_data)

• Let's have another look at the description data 

head(description_data)

[1] "The adventures of writer Newt Scamander in New York's secret community of witches and wizards seventy years before Harry Potter reads his book in school."                                                                              

[2] "Batman is manipulated by Lex Luthor to fear Superman. Superman´s existence is meanwhile dividing the world and he is framed for murder during an international crisis. The heroes clash and force the neutral Wonder Woman to reemerge."

[3] "A secret government agency recruits some of the most dangerous incarcerated super-villains to form a defensive task force. Their first mission: save the world from the apocalypse."                                                    

[4] "While on a journey of physical and spiritual healing, a brilliant neurosurgeon is drawn into the world of the mystic arts."                                                                                                             

[5] "A wisecracking mercenary gets experimented on and becomes immortal but ugly, and sets out to track down the man who ruined his looks."                                                                                                  

[6] "A woman is hired as a handmaiden to a Japanese heiress, but secretly she is involved in a plot to defraud her."                                                                                                                       

• Using CSS selectors to scrape the Movie runtime section

runtime_data_html <- html_nodes(webpage,'.text-muted .runtime')

• Converting the runtime data to text

runtime_data <- html_text(runtime_data_html)

• Let's have a look at the runtime

head(runtime_data)

[1] "132 min" "182 min" "123 min" "115 min" "108 min" "145 min"

• Using CSS selectors to scrape the Movie genre section

genre_data_html <- html_nodes(webpage,'.genre')

• Converting the genre data to text

genre_data <- html_text(genre_data_html)

• Data-Preprocessing: removing \n

genre_data<-gsub("\n","",genre_data)

• Data-Preprocessing: removing excess spaces

genre_data<-gsub(" ","",genre_data)

• Taking only the first genre of each movie

genre_data<-gsub(",.*","",genre_data)

• Convering each genre from text to factor

genre_data<-as.factor(genre_data)

• Let's have another look at the genre data

head(genre_data)

[1] Adventure Action    Action    Action    Action    Drama    

Levels: Action Adventure Animation Biography Comedy Crime Drama Horror

• Using CSS selectors to scrape the IMDB rating section

rating_data_html <- html_nodes(webpage,'.ratings-imdb-rating strong')

• Converting the ratings data to text

rating_data <- html_text(rating_data_html)

• Let's have a look at the ratings

head(rating_data)

[1] "7.2" "7.7" "5.9" "7.5" "8.0" "8.1"

• Using CSS selectors to scrape the votes section

votes_data_html <- html_nodes(webpage,'.sort-num_votes-visible span:nth-child(2)')

• Converting the votes data to text

votes_data <- html_text(votes_data_html)

• Let's have a look at the votes data

head(votes_data)

[1] "452,185" "20,529"  "670,449" "689,706" "991,800" "137,456"

• Data-Preprocessing: removing commas

votes_data<-gsub(",","",votes_data)

• Data-Preprocessing: converting votes to numerical

votes_data<-as.numeric(votes_data)

• Let's have another look at the votes data

head(votes_data)

[1] 452185  20529 670449 689706 991800 137456

• Using CSS selectors to scrape the directors section

directors_data_html <- html_nodes(webpage,'.text-muted+ p a:nth-child(1)')

• Converting the directors data to text

directors_data <- html_text(directors_data_html)

• Let's have a look at the directors data

head(directors_data)

[1] "David Yates"      "Amy Adams"        "David Ayer"      

[4] "Scott Derrickson" "Tim Miller"       "Park Chan-wook"  

• Using CSS selectors to scrape the actors section

actors_data_html <- html_nodes(webpage,'.lister-item-content .ghost+ a')

• Converting the gross actors data to text

actors_data <- html_text(actors_data_html)

• Let's have a look at the actors data

head(actors_data)

[1] "Eddie Redmayne"       "Will Smith"           "Benedict Cumberbatch"

[4] "Ryan Reynolds"        "Kim Min-hee"          "Ben Affleck" 

• Data-Preprocessing: converting actors data into factors

actors_data<-as.factor(actors_data)

• Using CSS selectors to scrape the metascore section

metascore_data_html <- html_nodes(webpage,'.metascore')

• Converting the runtime data to text

metascore_data <- html_text(metascore_data_html)

• Let's have a look at the metascore 

head(metascore_data)

[1] "66        " "40        " "72        " "65        " "84        "

[6] "44        "

• Data-Preprocessing: removing extra space in metascore

metascore_data<-gsub(" ","",metascore_data)

• Lets check the length of metascore data

length(metascore_data)

• Let's have a look at the metascore 

head(metascore_data)

[1] "66" "40" "72" "65" "84" "44"

• The visual inspection shows that the Metascore is missing for movies 39, 73, 80 and 89.

 

for (i in c(39,73,80,89)){

a<-metascore_data[1:(i-1)]

b<-metascore_data[i:length(metascore_data)]

metascore_data<-append(a,list("NA"))

metascore_data<-append(metascore_data,b)

}

• Data-Preprocessing: converting metascore to numerical

metascore_data<-as.numeric(metascore_data)

• Let's have another look at length of the metascore data

length(metascore_data)

• Data-Preprocessing: converting metascore to numerical

metascore_data<-as.numeric(metascore_data)

• Let's have another look at length of the metascore data

length(metascore_data)

[1] 101

head(metascore_data)

[1] 66 40 72 65 84 44

• Let's look at summary statistics

summary(metascore_data)

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 

  23.00   47.00   62.00   60.29   74.00   99.00       4 

• Using CSS selectors to scrape the gross revenue section

gross_data_html <- html_nodes(webpage,'.ghost~ .text-muted+ span')

• Converting the gross revenue data to text

gross_data <- html_text(gross_data_html)

• Let's have a look at the gross revenue data

head(gross_data)

[1] "$234.04M" "$325.10M" "$232.64M" "$363.07M" "$2.01M"   "$330.36M"

• Data-Preprocessing: removing '$' and 'M' signs

gross_data<-gsub("M","",gross_data)

gross_data<-substring(gross_data,2,6)

• Let's check the length of gross data

length(gross_data)

[1] 90

• Let's have a look at the gross revenue data

head(gross_data)

[1] "234.0" "325.1" "232.6" "363.0" "2.01"  "330.3"

• Filling missing entries with NA

for (i in c(17,39,49,52,57,64,66,73,76,77,80,87,88,89)){

a<-gross_data[1:(i-1)]

b<-gross_data[i:length(gross_data)]

gross_data<-append(a,list("NA"))

gross_data<-append(gross_data,b)

}

• Data-Preprocessing: converting gross to numerical

gross_data<-as.numeric(gross_data)

• Let's have another look at the length of gross data

length(gross_data)

[1] 104

summary(gross_data)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 

   0.18   26.84   56.95   95.85  126.20  532.10      14 

• Create the histogram.

hist(gross_data,xlab = "Gross $M ",col = "yellow",border = "blue")

Now we have successfully scraped all the 11 features for the 100 most 

popular feature films released in 2016. 

Let’s combine them to create a dataframe and inspect its structure.

• Combining all the lists to form a data frame

movies_df<-data.frame(Rank = rank_data, Title = title_data)

• Description = description_data, Runtime = runtime_data,
• Genre = genre_data, Rating = rating_data,
• Metascore = metascore_data1, Votes = votes_data, 
• Director = directors_data, Actor = actors_data, Gross = gross_data1)

• Structure of the data frame

str(movies_df)

movies_df<-data.frame(Rank = rank_data, Title = title_data)

• data visualization

movies_df1<-data.frame(Runtime = runtime_data, Genre = genre_data)

movies_df2<-data.frame(Rank = rank_data, Runtime = runtime_data, Votes=votes_data,Rating=rating_data)

sp=ggplot(movies_df2,aes(x=Rank,y=Runtime))+

geom_point(aes(size=Votes,col=Rating))

sp

# Change x and y axis labels, and limits

sp + scale_x_continuous(name="Rank", limits=c(1, 99),breaks=seq(1,99,5)) +

#  scale_y_continuous(name="Stopping distance", limits=c(0, 150))

#movies_df1<-data.frame(Runtime = runtime_data, Genre = genre_data)

#qplot(data = movies_df1,Runtime,fill = Genre,bins = 30)

movies_df3<-data.frame(Votes = votes_data, Rating = rating_data)

qplot(data = movies_df3,Votes,fill = Rating,bins = 30)

It is clear that one can analyze the data, drawing BI inferences from the above plots, training machine learning models over this data, etc.