https://github.com/xi/pyjsonproxy

simple proxy and scraper
https://github.com/xi/pyjsonproxy

Last synced: about 2 months ago
JSON representation

simple proxy and scraper

• Host: GitHub
• URL: https://github.com/xi/pyjsonproxy
• Owner: xi
• Created: 2015-02-05T16:42:24.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2019-05-05T08:29:41.000Z (over 5 years ago)
• Last Synced: 2023-04-03T10:34:19.266Z (almost 2 years ago)
• Language: Python
• Homepage:
• Size: 35.2 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • Changelog: CHANGES.rst

Awesome Lists containing this project

README

        
PyJSONProxy - simple proxy and scraper

simple proxy

============

AJAX requests are restricted by the `same origin policy`_. This can be

bypassed by using either `JSONP`_, `CORS`_ or a local proxy. This

implements the third variant. So you can do something like this::

    $ curl http://localhost:5000/github/xi/

    {

      "login": "xi",

      ...

    }

With a configuration like this::

    ENDPOINTS = {

        'github': {

            'host': 'https://api.github.com/users/'

        }

    }

scraping

========

Maybe the more interesting part is that this also contains a simple

scraping mechanism. So if a service does not offer an API but only plain

HTML pages, PyJSONProxy can extract information from there::

    $ curl http://localhost:5000/github/xi/

    {

      "url": "https://github.com/xi/",

      "login": "xi",

      "activity": [

        ...

      ],

      "repos": [{

        ...

      }]

      ...

    }

::

    ENDPOINTS = {

        'github': {

            'host': 'https://github.com/',

            'fields': {

                'login': '.vcard-username',

                'fullname': '.vcard-fullname',

                'email': '.vcard-details .email',

                'join-date': '.vcard-details .join-date@datetime',

                'activity': {

                    'selector': '.contribution-activity-listing ul a'

                },

                'repos': {

                    'selector': '.popular-repos a.mini-repo-list-item',

                    'fields': {

                        'url': '@href',

                        'name': '.repo',

                        'description': '.repo-description'

                    }

                }

            }

        }

    }

Selectors are generally CSS-selectors with the additional option to

select an attribute by appending an ``@`` and the attribute name. If no

attribute is selected, the text content of the element will be used.

CORS header

===========

By setting ``ALLOW_CORS`` to ``True``, an

``Access-Control-Allow-Origin``-header with value ``*`` will be set for

all responses.

Documentation

=============

Some simple documentation is automatically generated and available under

``/`` (for all endpoints) or ``/{endpoint}/`` (for an individual

endpoint). To provide some input for this documentation, you can add a

description to both endpoints and fields::

    ENDPOINTS = {

        'github': {

            'host': 'https://github.com/',

            'doc': 'Access data about GitHub users',

            'fields': {

              'login': '.vcard-username',

              'fullname': '.vcard-fullname',

              'email': '.vcard-details .email'

              'join-date': '.vcard-details .join-date@datetime'

            },

            'fields_doc': {

              'login': 'github username',

              'fullname': 'the user\'s full name',

              'join-date': 'date when the user joined github in ISO-xx format'

            }

        }

    }

Note on security and performance

================================

Security and performance were not a priority in this project. So it

might be bad.

.. _same origin policy: https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy

.. _JSONP: https://en.wikipedia.org/wiki/JSONP

.. _CORS: https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS