https://github.com/kootenpv/requests_viewer

View requests objects with style
https://github.com/kootenpv/requests_viewer

Last synced: 2 months ago
JSON representation

View requests objects with style

• Host: GitHub
• URL: https://github.com/kootenpv/requests_viewer
• Owner: kootenpv
• Created: 2016-07-30T21:57:20.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2017-01-15T17:30:43.000Z (almost 8 years ago)
• Last Synced: 2024-09-25T16:57:36.053Z (4 months ago)
• Language: Python
• Size: 201 KB
• Stars: 42
• Watchers: 5
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• starred-awesome - requests_viewer - View requests objects with style (Python)

README

        
## requests_viewer

The idea is that `requests_viewer` can tell us information about our requests-objects quickly.

It opens up an HTML page with information for the request.

```python

import requests

from requests_viewer.web import view_request

view_request(requests.get("https://xkcd.com/"))

# or

from requests_viewer import main

main("https://xkcd.com/") # considers different mime types

```

### Main features

- HTML page is being shown as how the crawler sees it

    * Can extract the domain and hot-link so that it looks almost indistuinguishable

- Contains other nice functions to show lxml tree nodes

- Can visually show diffs between 2 html pages / trees

```python

from requests_viewer.web import view_diff_tree, get_tree

url1 = "http://xkcd.com/"

url2 = "http://xkcd.com/1/"

tree1 = get_tree(url1)  # get tree from request object directly

tree2 = get_tree(url2)  # could instead use `make_tree` if you already have a req

view_diff_tree(tree1, tree2)

```

Results in:







### Installation

    pip install requests_viewer

    pip3 install requests_viewer

Note that in order to do the real fancy stuff, you should install:

    pip install requests_viewer[fancy]

    pip3 install requests_viewer[fancy]

this will install `lxml`, `bs4` and `tldextract`.

### Types it can show currently:

- text/html

- image/*

- application/json

### Usability

Some example `web.py` functions:

``` python

def slugify(value):

def view_request(r, domain=None):

def view_html(x):

def view_node(node, attach_head=False, question_contains=None):

def view_tree(node):

def view_diff_tree(tree1, tree2, url, diff_method):

def view_diff_html(html1, html2, url, diff_method):

def view_diff(html1, html2, tree1, tree2, url, diff_method):

def make_parent_line(node, attach_head=False, question_contains=None):

def extract_domain(url):

def make_tree(html, domain=None):

def get_tree(url, domain=None):

def get_local_tree(url, domain=None):

```

### Contribute

This package is very small at the moment. I very much encourage you to contribute:

- Most likely we will want to show headers on the top of the package (html)

- Make the encoding an argument (instead of fixed utf8)

Note that I use [yapf](https://github.com/google/yapf) with max-line=100 to avoid any styling discussion.