We have now to teach the scraper how to extract key figures from the ad
page.

.. code:: python

class DummyScraper(Scraper):
# HEADS UP: Meta declarations are hidden here
# ...
# ...

# Extract data using a CSS Selector.
realtorName = RegexField('.realtor-title')
# Extract data using a CSS Selector and a Regex.
serviceCharge = RegexField('.description-list', 'charges : (.*)\s€')
# Extract data using a CSS Selector and a Regex.
# This will throw a custom exception if the field is missing.
livingSpace = RegexField('.description-list', 'surface :(\d*)', required=True, exception=reporting.SpaceMissingError)
# Extract the value directly, without using a Regex
totalRent = RegexField('.description-price', required=True, exception=reporting.RentMissingError)
# Store this value as a private property (begining with a underscore).
# It won't be saved in the database but it can be helpful as you we'll see.
_address = RegexField('.description-address')

Every attribute will be saved as an Ad's property, according to the Ad
model.

Some properties may not be extractable from the HTML. You may need to
use a custom function that received existing properties. For this reason
we created a second field type named ``ComputedField``. Since the
properties order of declaration is recorded, we can use previously
declared (and extracted) values to compute new ones.

.. code:: python

class DummyScraper(Scraper):
# ...
# ...

# Use existing properties `totalRent` and `livingSpace` as they were
# extracted before this one.
pricePerSqm = ComputedField(fn=lambda s, values: values["totalRent"] / values["livingSpace"])
# This full exemple uses private properties to find latitude and longitude.
# To do so we use a buid-in function named `convert` that transforms an
# address into a dictionary of coordinates.
_latLng = ComputedField(fn=lambda s, values: geocode(values['_address'], 'FRA') )
# Gets a the dictionary field we want.
latitude = ComputedField(fn=lambda s, values: values['_latLng']['lat'])
longitude = ComputedField(fn=lambda s, values: values['_latLng']['lng'])

All you need to do now is to create an instance of your class and run
the scraper.

.. code:: python

# When you script is executed directly
if __name__ == "__main__":
dummyScraper = DummyScraper()
dummyScraper.run()

API Doc
-------

``class`` Ad
~~~~~~~~~~~~

Attributes
^^^^^^^^^^

As seen above, every Ad attribute might be used as a Scraper attribute to declare which attribute extract.

+----------------------+--------------------------+---------------------------------------------------------------------------+
| Name | Type | Description |
+======================+==========================+===========================================================================+
| ``status`` | *String* | "listed" if needs more scraping, "scraped" if it's done |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``site`` | *String* | Name of the website |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``createdAt`` | *DateTime* | Date the ad was first scraped |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``siteId`` | *String* | The unique ID from the site where it's scrapped from |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``serviceCharge`` | *Float* | Extra costs (heating mostly) |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``baseRent`` | *Float* | Base costs (without heating) |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``totalRent`` | *Float* | Total cost |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``livingSpace`` | *Float* | Surface in square meters |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``pricePerSqm`` | *Float* | Price per square meter |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``furnished`` | *Bool* | True if the flat or house is furnished |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``realtor`` | *Bool* | True if realtor, n if rented by a physical person |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``realtorName`` | *Unicode* | The name of the realtor or person offering the flat |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``latitude`` | *Float* | Latitude |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``longitude`` | *Float* | Longitude |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``balcony`` | *Bool* | True if there is a balcony/terrasse |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``yearConstructed`` | *String* | The year the building was built |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``cellar`` | *Bool* | True if the flat comes with a cellar |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``parking`` | *Bool* | True if the flat comes with a parking or a garage |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``houseNumber`` | *String* | House Number in the street |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``street`` | *String* | Street name (incl. "street") |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``zipCode`` | *String* | ZIP code |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``city`` | *Unicode* | City |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``lift`` | *Bool* | True if a lift is present |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``typeOfFlat`` | *String* | Type of flat (no typology) |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``noRooms`` | *String* | Number of rooms |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``floor`` | *String* | Floor the flat is at |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``garden`` | *Bool* | True if there is a garden |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``barrierFree`` | *Bool* | True if the flat is wheelchair accessible |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``country`` | *String* | Country, 2 letter code |
+----------------------+--------------------------+---------------------------------------------------------------------------+
| ``sourceUrl`` | *String* | URL of the page |
+----------------------+--------------------------+---------------------------------------------------------------------------+

``class`` Scraper
~~~~~~~~~~~~~~~~~

Methods
^^^^^^^

The Scraper class defines a lot of method that we encourage you to
redefine in order to have the full control of your scraper behavior.

+----------------------+------------------------------------------------------------------------------------------------------+
| Name | Description |
+======================+======================================================================================================+
| ``extract_ad`` | Extract ads list from a page's soup. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``fail`` | Print out an error message. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``fetch_ad`` | Fetch a single ad page from the target website then create Ad instances by calling ``èxtract_ad``. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``fetch_series`` | Fetch a single list page from the target website then fetch an ad by calling ``fetch_ad``. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``find_ad_blocks`` | Extract ad block from a page list. Called within ``fetch_series``. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``get_ad_href`` | Extract a href attribute from an ad block. Called within ``fetch_series``. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``get_ad_id`` | Extract a siteId from an ad block. Called within ``fetch_series``. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``get_fields`` | Used internally to generate a list of property to extract from the ad. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``get_series`` | Fetch a list page from the target website. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``has_issue`` | True if we met issues with this ad before. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``is_scraped`` | True if we already scraped this ad before. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``ok`` | Print out an success message. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``prepare`` | Just before saving the values. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``run`` | Run the scrapper. |
+----------------------+------------------------------------------------------------------------------------------------------+
| ``transform_page`` | Transform HTML content of the series page before parsing it. |
+----------------------+------------------------------------------------------------------------------------------------------+

Start a migration
-----------------

Use Yoyo_:

::

yoyo new ./migrations -m "Your migration's description"

And apply it:

::

yoyo apply --database mysql://user:password@host/db ./migrations