https://github.com/scrapy/w3lib

Python library of web-related functions
https://github.com/scrapy/w3lib

hacktoberfest python

Last synced: 3 days ago
JSON representation

Python library of web-related functions

• Host: GitHub
• URL: https://github.com/scrapy/w3lib
• Owner: scrapy
• License: bsd-3-clause
• Created: 2011-04-16T22:23:16.000Z (over 13 years ago)
• Default Branch: master
• Last Pushed: 2024-04-22T07:25:34.000Z (7 months ago)
• Last Synced: 2024-05-16T00:10:17.641Z (6 months ago)
• Topics: hacktoberfest, python
• Language: Python
• Homepage:
• Size: 463 KB
• Stars: 382
• Watchers: 25
• Forks: 103
• Open Issues: 18
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

• starred-awesome - w3lib - Python library of web-related functions (Python)

README

        
=====

w3lib

=====

.. image:: https://github.com/scrapy/w3lib/actions/workflows/tests.yml/badge.svg

   :target: https://github.com/scrapy/w3lib/actions

.. image:: https://img.shields.io/codecov/c/github/scrapy/w3lib/master.svg

   :target: http://codecov.io/github/scrapy/w3lib?branch=master

   :alt: Coverage report

Overview

========

This is a Python library of web-related functions, such as:

* remove comments, or tags from HTML snippets

* extract base url from HTML snippets

* translate entites on HTML strings

* convert raw HTTP headers to dicts and vice-versa

* construct HTTP auth header

* converting HTML pages to unicode

* sanitize urls (like browsers do)

* extract arguments from urls

Requirements

============

Python 3.9+

Install

=======

``pip install w3lib``

Documentation

=============

See http://w3lib.readthedocs.org/

License

=======

The w3lib library is licensed under the BSD license.