> `"//p[@class='para 1']/a/@href"`

> _**NOTE:** Make sure to **not to use** the same set of quotes to enclose the paths._
>
> Example:
>
> `"//p[class="para 1"]"` or `'//p[class='para 1']'`

### **Scrapy Shell (Debugger)**

Now that we learned to access the elements, lets try to actually apply it. Its always a best practice to try out the xpath in the scrapy shell to check if the xpath works, this helps us avoid unnecessary headache in the later stage of our project.

**Step 1:** Open the terminal and activate the environment and enter the following in the terminal to open scrapy shell.

```bash
$ scrapy shell
```


**Step 2:** `fetch([URL])` will get the response from the URL provided and will store in the variable `response`. And you can view the response using `view()`.




**Step 3:** To get an element from the page, we use `response.xpath([PATH]).extract()`. In this case, we get the list of the portal. To get the class name of the element, right click and inspect the element.



Now that we are quite comfortable with the extracting data from the html files, lets start to create our first dataset.

# **TASK 2: Create an image classification dataset**

Great! Now that we know the basic usage of scrapy, let's get started to scrape some data to create our first dataset.

We'll be taking a more systematic approach so you can easily replicate the same steps to create your own datasets.

**

Step 1:

```bash
$ scrapy startproject imgClfDataset
$ cd imgClfDataset
```

Then We create a python script with which we'll code in the set instructions for our spider.

```bash
$ touch ./imgClfDataset/spiders/images.py
```

**

Step 2:

```python
# items.py

import scrapy

class ImgclfdatasetItem(scrapy.Item):

image_titles = scrapy.Field()
images = scrapy.Field()
image_urls = scrapy.Field()

```

**

Step 3:

Firstly, we'll scrape the images of `photographs` after which you can change the `classname` variable to `paintings`.

The following is the content for the `images.py` script.

```python
# images.py

import scrapy
from scrapy.loader import ItemLoader
from ..items import ImgclfdatasetItem

class images(scrapy.Spider):

name = 'images'
classname = "photographs" # [ "photographs", "paintings"]
start_urls = [
f"https://fineartamerica.com/art/{classname}"
]
page = 1
image_urls = []
image_titles = []
page_limit = 10

def parse(self, response):

self.image_titles = response.xpath('[ TYPE XPATH HERE ]').extract()
self.image_urls = response.xpath('[ TYPE XPATH HERE ]').extract()

self.page= self.page + 1

if self.page <= self.page_limit:
item = ImgclfdatasetItem()
item['image_urls'] = self.image_urls
item['image_titles'] = self.image_titles
yield item

next_page = f'https://fineartamerica.com/art/{self.classname}?page={self.page}'
yield scrapy.Request(next_page, callback=self.parse)

```

To save the requested images from the url, we have to add a line in the `settings.py` script to specify the download location.

```python
# settings.py
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}

IMAGES_STORE = './raw_dataset/paintings' # Add this line and also create the folder

```

**

Step 3:

The general syntax to call the spider to crawl is the following

```bash
$ scrapy crawl [NAME OF THE SPIDER]
```

In this case, it would be

```bash
$ scrapy crawl images
```

With that we conclude this task and will get back to the scraped content at the last section where we clean and preprocess the data.

As we have already applied various components to scrape content from web pages, it would be **easier to understand the architecture of scrapy.**

## **Scrapy Architecture**


### **Components**

**Scrapy Engine**
The engine is responsible for controlling the data flow between all components of the system, and triggering events when certain actions occur. See the Data Flow section above for more details.

**Scheduler**
The Scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them.

**Downloader**
The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders.

**Spiders**
Spiders are custom classes written by Scrapy users to parse responses and extract items from them or additional requests to follow. For more information see Spiders.

**Item Pipeline**
The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database).

**Downloader middlewares**
Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader, and responses that pass from Downloader to the Engine.

**Spider middlewares**
Spider middlewares are specific hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests).

# **TASK 3: News Headlines Dataset**

**Sentiment analysis** is a very important task in various field from politics to customer satisfaction. And hence we always need some data to test the sentiment analysis model and in this section we'll be mining news headlines about "Oil". This modelling could be extrapolated to any other topics. So lets get started!

**

Step 1:

```bash
$ scrapy startproject newsheadlines
$ cd newsheadlines
```

Then We create a python script with which we'll code in the set instructions for our spider.

```bash
$ touch ./newsheadlines/spiders/headlines.py
```

**

Step 2:

The following is the content for the `headlines.py` script.

```python
# headlines.py

import scrapy

class newsheadlines(scrapy.Spider):

name="news"
start_urls = [
'https://oilprice.com/Latest-Energy-News/World-News/'
]
page = 1
page_limit = 10

def parse(self, response):
if self.page <= self.page_limit:
news_title = response.xpath("[ TYPE XPATH HERE ]").extract()
news_meta = response.xpath("[ TYPE XPATH HERE ]").extract()
self.page+=1
for t, m in zip(news_title, news_meta):
news_raw = {
'title': t,
'meta' : m,
}

yield news_raw
next_page = f'https://oilprice.com/Latest-Energy-News/World-News/Page-{self.page}.html'
yield scrapy.Request(next_page, self.parse)
```

we also have to add a few lines in the `settings.py` script to specify the download location.

```python
# settings.py

FEED_FORMAT="csv"
FEED_URI="./raw_dataset/news_raw.csv" # Add this line and also create the folder

```

**

Step 3:

```bash
$ scrapy crawl news
```

We have crawled all the data that's required to for the next section and now lets start cleaning some data.

# **Data Cleaning and Preprocessing**

The following section of the workshop is documented in the `Data Cleaning.ipynb` notebook. Open the jupyter notebook and proceed with the workshop.

## **And with that we come to the end of the workshop...**

Thanks for attending the workshop! I hope it gave you a better idea about scraping data from web and cleaning the scraped data. These are a few study materials for reference or if you wish to explore more about this topic.

Feel free to fork this repository to keep track of the courses and project ideas.

# **Supplementary Materials**

## **Massive open online course (MOOC)**

- [ ] **Mining Massive Datasets:** The courses is quite an advanced course but the contents covered are really interesting. They have a youtube playlist along with a reference book.

- https://www.youtube.com/channel/UC_Oao2FYkLAUlUVkBfze4jg/videos
- https://www.youtube.com/channel/UC_Oao2FYkLAUlUVkBfze4jg/videos
- http://www.mmds.org/

- [ ] **Data Mining Specilization** The course dives deep into different aspects of the the topics dicussed in the workshop.

- https://www.coursera.org/specializations/data-mining

## **Some Project Ideas to explore**

- [ ] Create an IMDB movie dataset from scratch and build a simple recommendation system using the dataset.

- [ ] Create a multiclass image classification dataset of your choice and create a classification model using the dataset

# **FAQs for the workshop**

### **Web Scraping vs Web crawling**

### **Beautiful Soup vs Selenium vs Scrapy**

Selenium and Beautifulsoup are very easy to learn and are suitable if you are trying to mine a small of amount of data from the web and on the downside, you might get blocked in some websites which require Captcha.

On the other hand, Scrapy has a much steeper learning curve but it is much more robust and very efficient compared to others. It is very much advised to use Scrapy when mining much larger data from the web.

# **References**