https://github.com/andreoliwa/scrapy-tegenaria
🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢
https://github.com/andreoliwa/scrapy-tegenaria
crawler flask postgresql python python3 scrapy
Last synced: about 1 year ago
JSON representation
🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢
- Host: GitHub
- URL: https://github.com/andreoliwa/scrapy-tegenaria
- Owner: andreoliwa
- License: agpl-3.0
- Created: 2015-08-04T20:40:20.000Z (almost 11 years ago)
- Default Branch: develop
- Last Pushed: 2025-03-17T19:19:20.000Z (over 1 year ago)
- Last Synced: 2025-03-26T22:21:34.542Z (about 1 year ago)
- Topics: crawler, flask, postgresql, python, python3, scrapy
- Language: Python
- Homepage:
- Size: 1.4 MB
- Stars: 2
- Watchers: 1
- Forks: 5
- Open Issues: 45
-
Metadata Files:
- Readme: README.rst
- Changelog: HISTORY.rst
- Contributing: CONTRIBUTING.rst
- License: LICENSE
- Authors: AUTHORS.rst
Awesome Lists containing this project
README
=========
Tegenaria
=========
.. image:: https://img.shields.io/travis/andreoliwa/scrapy-tegenaria.svg
:target: https://travis-ci.org/andreoliwa/scrapy-tegenaria
Scrapy spiders to collect and browse different items (so far, apartment ads).
* Free software: BSD license
* Documentation: https://tegenaria.readthedocs.org.
How to setup the dev environment
--------------------------------
TODO
Metrics
-------
.. image:: https://badge.waffle.io/andreoliwa/scrapy-tegenaria.svg?label=ready&title=Ready
:alt: Stories in Ready
:target: http://waffle.io/andreoliwa/scrapy-tegenaria
.. image:: https://graphs.waffle.io/andreoliwa/scrapy-tegenaria/throughput.svg
:alt: Throughput Graph
:target: https://waffle.io/andreoliwa/scrapy-tegenaria/metrics
Quickstart
----------
First, set your app's secret key as an environment variable. For example, example add the following to ``.bashrc`` or ``.bash_profile``.
.. code-block:: bash
export TEGENARIA_SECRET='something-really-secret'
Then run the following commands to bootstrap your environment.
::
git clone https://github.com/andreoliwa/scrapy-tegenaria
cd tegenaria
pip install -r requirements/dev.txt
flask run
You will see a pretty welcome screen.
Once you have installed your DBMS, run the following to create your app's database tables and perform the initial migration:
::
flask db init
flask db migrate
flask db upgrade
flask run
Deployment
----------
In your production environment, make sure the ``TEGENARIA_ENV`` environment variable is set to ``"prod"``.
Shell
-----
To open the interactive shell, run ::
flask shell
By default, you will have access to ``app``, ``db``, and the ``User`` model.
Running Tests
-------------
To run all tests, run ::
flask test
Migrations
----------
Whenever a database migration needs to be made. Run the following commands:
::
flask db migrate
This will generate a new migration script. Then run:
::
flask db upgrade
To apply the migration.
For a full migration command reference, run ``flask db --help``.