The main objective of the configuration file is to specify extraction rules in terms of selector expressions and the attribute to be extracted. There are certain set forms of selector/attribute value pairs that perform various types of content extraction.

Selector expressions :

- CSS selector or XPath expressions that specify the tag to be selected.
- "url" to take the URL of the current page on which extraction is being performed.

Attribute selectors :

- "text" to extract the textual content from that tag.
- "href", "src" etc., to extract any of the other attributes of the selected tag.

Tutorials
---------

[For a more detailed tutorial, check out the [tutorial in the documentation](http://scrapple.readthedocs.org/en/latest/#experimentation-results)]

In this simple example for using Scrapple, we'll extract NBA player information from [the ESPN website](http://espn.go.com/nba/teams).

To first create the skeleton configuration file, we use the genconfig command.

$ scrapple genconfig nba http://espn.go.com/nba/teams --type=crawler --levels=2

This creates nba.json - a sample Scrapple configuration file for a crawler, which uses XPath expressions as selectors. This can be edited and the required follow link selector, data selectors and attributes can be specified.

~~~~ {.sourceCode .javascript}
{
"project_name": "nba",
"selector_type": "xpath",
"scraping": {
"url": "http://espn.go.com/nba/teams",
"data": [
{
"field": "",
"selector": "",
"attr": "",
"default": "",
"connector": ""
}
],
"next": [
{
"follow_link": "//*[@class='mod-content']//a[3]",
"scraping": {
"data": [
{
"field": "team",
"selector": "//h2",
"attr": "text",
"default": "",
"connector": ""
}
],
"next": [
{
"follow_link": "//*[@class='mod-content']/table[1]//tr[@class!='colhead']//a",
"scraping": {
"data": [
{
"field": "name",
"selector": "//h1",
"attr": "text",
"default": "",
"connector": ""
},
{
"field": "headshot_link",
"selector": "//*[@class='main-headshot']/img",
"attr": "src",
"default": "",
"connector": ""
},
{
"field": "number & position",
"selector": "//ul[@class='general-info']/li[1]",
"attr": "text",
"default": "<00> #",
"connector": ""
}
],
"table": [
{
"table_type": "rows",
"header": "//div[@class='player-stats']//table//th",
"prefix": "season_",
"suffix": "",
"selector": "//div[@class='player-stats']//table//tr[1]/td",
"attr": "text",
"default": "",
"connector": ""
},
{
"table_type": "rows",
"header": "//div[@class='player-stats']//table//th",
"prefix": "career_",
"suffix": "",
"selector": "//div[@class='player-stats']//table//tr[@class='career']/td",
"attr": "text",
"default": "",
"connector": ""
}
]
}
}
]
}
}
]
}
}
~~~~

The extractor can be run using the run command -

$ scrapple run nba nba_players -o json

This creates nba\_players.json which contains the extracted data. An example snippet of this data :

~~~~ {.sourceCode .javascript}
{

"project": "nba",
"data": [

# nba_players.json continues

{
"career_APG" : "9.9",
"career_PER" : "",
"career_PPG" : "18.6",
"career_RPG" : "4.4",
"headshot_link" : "http://a.espncdn.com/combiner/i?img=/i/headshots/nba/players/full/2779.png&w=350&h=254",
"name" : "Chris Paul",
"number & position" : "#3 PG",
"season_APG" : "9.2",
"season_PER" : "23.49",
"season_PPG" : "17.6",
"season_RPG" : "3.5",
"team" : "Los Angeles Clippers"
},
{
"career_APG" : "3.6",
"career_PER" : "",
"career_PPG" : "20.3",
"career_RPG" : "5.8",
"headshot_link" : "http://a.espncdn.com/combiner/i?img=/i/headshots/nba/players/full/662.png&w=350&h=254",
"name" : "Paul Pierce",
"number & position" : "#34 SF",
"season_APG" : "0.9",
"season_PER" : "7.55",
"season_PPG" : "5.0",
"season_RPG" : "2.6",
"team" : "Los Angeles Clippers"
},
{
"career_APG" : "2.9",
"career_PER" : "",
"career_PPG" : "3.7",
"career_RPG" : "1.8",
"headshot_link" : "http://a.espncdn.com/combiner/i?img=/i/headshots/nba/players/full/4182.png&w=350&h=254",
"name" : "Pablo Prigioni",
"number & position" : "#9 PG",
"season_APG" : "1.9",
"season_PER" : "8.72",
"season_PPG" : "2.3",
"season_RPG" : "1.5",
"team" : "Los Angeles Clippers"
},
{
"career_APG" : "2.0",
"career_PER" : "",
"career_PPG" : "11.1",
"career_RPG" : "1.9",
"headshot_link" : "http://a.espncdn.com/combiner/i?img=/i/headshots/nba/players/full/3024.png&w=350&h=254",
"name" : "J.J. Redick",
"number & position" : "#4 SG",
"season_APG" : "1.6",
"season_PER" : "18.10",
"season_PPG" : "15.9",
"season_RPG" : "1.5",
"team" : "Los Angeles Clippers"
},

# nba_players.json continues
]

}
~~~~

The run command can also be used to create a CSV file with the extracted data, using the --output\_type=csv argument.

The generate command can be used to generate a Python script that implements this extractor. In essence, it replicates the execution of the run command.

$ scrapple generate nba nba_script -o json

This creates nba\_script.py, which extracts the required data and stores the data in a JSON document.

Documentation
-------------

You can read the [complete documentation](http://scrapple.rtfd.org) for an extensive coverage on the background behind Scrapple, a thorough explanation on the Scrapple package implementation and a complete coverage of tutorials on how to use Scrapple to run your scraper/crawler jobs.

Authors
-------