https://github.com/andreoliwa/scrapy-tegenaria

🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢
https://github.com/andreoliwa/scrapy-tegenaria

crawler flask postgresql python python3 scrapy

Last synced: about 23 hours ago
JSON representation

🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢

• Host: GitHub
• URL: https://github.com/andreoliwa/scrapy-tegenaria
• Owner: andreoliwa
• License: agpl-3.0
• Created: 2015-08-04T20:40:20.000Z (about 9 years ago)
• Default Branch: develop
• Last Pushed: 2024-09-16T21:12:43.000Z (17 days ago)
• Last Synced: 2024-10-02T19:42:10.570Z (about 24 hours ago)
• Topics: crawler, flask, postgresql, python, python3, scrapy
• Language: Python
• Homepage:
• Size: 1.4 MB
• Stars: 2
• Watchers: 2
• Forks: 5
• Open Issues: 45
• Metadata Files:
  • Readme: README.rst
  • Changelog: HISTORY.rst
  • Contributing: CONTRIBUTING.rst
  • License: LICENSE
  • Authors: AUTHORS.rst

Awesome Lists containing this project

README

        
=========

Tegenaria

=========

.. image:: https://img.shields.io/travis/andreoliwa/scrapy-tegenaria.svg

        :target: https://travis-ci.org/andreoliwa/scrapy-tegenaria

Scrapy spiders to collect and browse different items (so far, apartment ads).

* Free software: BSD license

* Documentation: https://tegenaria.readthedocs.org.

How to setup the dev environment

--------------------------------

TODO

Metrics

-------

.. image:: https://badge.waffle.io/andreoliwa/scrapy-tegenaria.svg?label=ready&title=Ready

   :alt: Stories in Ready

   :target: http://waffle.io/andreoliwa/scrapy-tegenaria

.. image:: https://graphs.waffle.io/andreoliwa/scrapy-tegenaria/throughput.svg

   :alt: Throughput Graph

   :target: https://waffle.io/andreoliwa/scrapy-tegenaria/metrics

Quickstart

----------

First, set your app's secret key as an environment variable. For example, example add the following to ``.bashrc`` or ``.bash_profile``.

.. code-block:: bash

    export TEGENARIA_SECRET='something-really-secret'

Then run the following commands to bootstrap your environment.

::

    git clone https://github.com/andreoliwa/scrapy-tegenaria

    cd tegenaria

    pip install -r requirements/dev.txt

    flask run

You will see a pretty welcome screen.

Once you have installed your DBMS, run the following to create your app's database tables and perform the initial migration:

::

    flask db init

    flask db migrate

    flask db upgrade

    flask run

Deployment

----------

In your production environment, make sure the ``TEGENARIA_ENV`` environment variable is set to ``"prod"``.

Shell

-----

To open the interactive shell, run ::

    flask shell

By default, you will have access to ``app``, ``db``, and the ``User`` model.

Running Tests

-------------

To run all tests, run ::

    flask test

Migrations

----------

Whenever a database migration needs to be made. Run the following commands:

::

    flask db migrate

This will generate a new migration script. Then run:

::

    flask db upgrade

To apply the migration.

For a full migration command reference, run ``flask db --help``.