Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/katyukha/lxmlrpc_monkey
xmlrpclib patch to reduce memory consumption on large strings
https://github.com/katyukha/lxmlrpc_monkey
Last synced: 24 days ago
JSON representation
xmlrpclib patch to reduce memory consumption on large strings
- Host: GitHub
- URL: https://github.com/katyukha/lxmlrpc_monkey
- Owner: katyukha
- License: other
- Created: 2017-06-08T13:42:16.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2017-06-22T09:32:18.000Z (over 7 years ago)
- Last Synced: 2024-09-20T13:37:13.272Z (about 2 months ago)
- Language: Python
- Size: 9.77 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
lxmlrpc - XMLRPClib patch
=========================
This module monkeypatch python's `xmlrpclib` to use `lxml `__ based parser
to reduce memory consumption on big xmlrpc requests / responses (100+ Mb)
NOTE
----
This module is useful only for ***python2.7***.
Use it **only** if you suffer from **high memory consumption** of **xmlrpclib**
Install
-------
This module is hosted on `PyPI `__
so it could be easily installed via *pip*:
.. code:: bash
pip install lxmlrpc_monkey
Usage
-----
To use this module just do following
.. code:: python
import xmlrpclib
from lxmlrpc_monkey import patch_xmlrpclib
# This line will monkey-patch xmlrpclib to use lxml for parser
patch_xmlrpclib()
Benchmarks (how to run)
-----------------------
To run benchmarks:
1. install `memory_profiler `__
2. prepare data with ``python generate_data.py --path --size 50000000``
3. run benchmarks with ``python benchmark.py --path ``
Benchmark results (50 Mb (real 65 Mb) data file)
================================================
look at ```p.feed(data)``` call in ```loads``` function of *xmlrpclib*
***Running unpatched loads***
Filename: /usr/lib/python2.7/xmlrpclib.py
====== ========= ========= =============
Line # Mem usage Increment Line Contents
====== ========= ========= =============
1134 104.7 MiB 0.0 MiB def loads(data, use_datetime=0):
1135 """data -> unmarshalled data, method name
1136
1137 Convert an XML-RPC packet to unmarshalled data plus a method
1138 name (None if not present).
1139
1140 If the XML-RPC packet represents a fault condition, this function
1141 raises a Fault exception.
1142 """
1143 104.7 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 622.4 MiB 517.7 MiB p.feed(data)
1145 558.0 MiB -64.4 MiB p.close()
1146 558.0 MiB 0.0 MiB return u.close(), u.getmethodname()
====== ========= ========= =============
***Running patched loads***
Filename: /usr/lib/python2.7/xmlrpclib.py
====== ========= ========= =============
Line # Mem usage Increment Line Contents
====== ========= ========= =============
1134 106.9 MiB 0.0 MiB def loads(data, use_datetime=0):
1135 """data -> unmarshalled data, method name
1136
1137 Convert an XML-RPC packet to unmarshalled data plus a method
1138 name (None if not present).
1139
1140 If the XML-RPC packet represents a fault condition, this function
1141 raises a Fault exception.
1142 """
1143 106.9 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 235.9 MiB 129.0 MiB p.feed(data)
1145 171.5 MiB -64.4 MiB p.close()
1146 171.5 MiB 0.0 MiB return u.close(), u.getmethodname()
====== ========= ========= =============
Filename: bechmark.py
====== ========= ========= =============
Line # Mem usage Increment Line Contents
====== ========= ========= =============
13 104.7 MiB 0.0 MiB @profile
14 def bench_load(xmldata):
15 104.7 MiB 0.0 MiB print ("Running unpatched loads")
16 106.9 MiB 2.2 MiB loads(xmldata)
17
18 106.9 MiB 0.0 MiB lxmlrpc.patch_xmlrpclib()
19
20 106.9 MiB 0.0 MiB print ("Running patched loads")
21 107.1 MiB 0.2 MiB loads(xmldata)
====== ========= ========= =============
Benchmark results (100 Mb (real 129 Mb) data file)
==================================================
***Running unpatched loads***
---
Filename: /usr/lib/python2.7/xmlrpclib.py
====== ========== ========== =============
Line # Mem usage Increment Line Contents
====== ========== ========== =============
1134 169.2 MiB 0.0 MiB def loads(data, use_datetime=0):
1135 """data -> unmarshalled data, method name
1136
1137 Convert an XML-RPC packet to unmarshalled data plus a method
1138 name (None if not present).
1139
1140 If the XML-RPC packet represents a fault condition, this function
1141 raises a Fault exception.
1142 """
1143 169.2 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 1203.0 MiB 1033.8 MiB p.feed(data)
1145 1074.2 MiB -128.8 MiB p.close()
1146 1074.2 MiB 0.0 MiB return u.close(), u.getmethodname()
====== ========== ========== =============
***Running patched loads***
---
Filename: /usr/lib/python2.7/xmlrpclib.py
====== ========== ========== =============
Line # Mem usage Increment Line Contents
====== ========== ========== =============
1134 171.6 MiB 0.0 MiB def loads(data, use_datetime=0):
1135 """data -> unmarshalled data, method name
1136
1137 Convert an XML-RPC packet to unmarshalled data plus a method
1138 name (None if not present).
1139
1140 If the XML-RPC packet represents a fault condition, this function
1141 raises a Fault exception.
1142 """
1143 171.6 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 429.4 MiB 257.8 MiB p.feed(data)
1145 300.6 MiB -128.8 MiB p.close()
1146 300.6 MiB 0.0 MiB return u.close(), u.getmethodname()
====== ========== ========== =============
Filename: bechmark.py
====== ========== ========== =============
Line # Mem usage Increment Line Contents
====== ========== ========== =============
13 169.2 MiB 0.0 MiB @profile
14 def bench_load(xmldata):
15 169.2 MiB 0.0 MiB print ("Running unpatched loads")
16 171.6 MiB 2.4 MiB loads(xmldata)
17
18 171.6 MiB 0.0 MiB lxmlrpc.patch_xmlrpclib()
19
20 171.6 MiB 0.0 MiB print ("Running patched loads")
21 171.8 MiB 0.2 MiB loads(xmldata)
====== ========== ========== =============