Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/douban/libmc

Fast and light-weight memcached client for C++ / #python / #golang #libmc
https://github.com/douban/libmc

c-plus-plus cache caching client-lib golang memcached python

Last synced: 6 days ago
JSON representation

Fast and light-weight memcached client for C++ / #python / #golang #libmc

Awesome Lists containing this project

README 

        
libmc

=====



|build_go| |build_py|

|status| |pypiv| |pyversions| |wheel| |license|



libmc is a memcached client library for Python without any other

dependencies at runtime. It's mainly written in C++ and Cython and

can be considered a drop in replacement for libmemcached and

`python-libmemcached `__.



libmc is developed and maintained by Douban Inc. Currently, it is

working in a production environment, powering all web traffic on

`douban.com `__

(`english wiki `__).



Build and Installation

----------------------



For users:



::



    pip install libmc



Usage:



.. code:: python



    import libmc



    mc = libmc.Client(['localhost:11211', 'localhost:11212'])

    mc.set('foo', 'bar')

    assert mc.get('foo') == 'bar'



Under the hood

--------------



Under the hood, libmc consists of 2 parts: an internal, fully-functional

memcached client implementation in C++ and a Cython wrapper around that

implementation. Dynamic memory allocation and memory-copy are slow, so

we've tried our best to avoid them. libmc also supports the ``set_multi``

command, which is not natively supported by the `memcached

protocol `__.

Some techniques have been applied to make ``set_multi`` command extremely fast

in libmc (compared to similiar libraries).



Configuration

-------------



.. code:: python



    import libmc

    from libmc import (

        MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT

    )



    mc = libmc.Client(

        [

        'localhost:11211',

        'localhost:11212',

        'remote_host',

        'remote_host mc.mike',

        'remote_host:11213 mc.oscar'

        ],

        do_split=True,

        comp_threshold=0,

        noreply=False,

        prefix=None,

        hash_fn=MC_HASH_MD5,

        failover=False

    )



    mc.config(MC_POLL_TIMEOUT, 100)  # 100 ms

    mc.config(MC_CONNECT_TIMEOUT, 300)  # 300 ms

    mc.config(MC_RETRY_TIMEOUT, 5)  # 5 s



-  ``servers``: a list of memcached server addresses. Each address

   should be formated as ``hostname[:port] [alias]``, where ``port`` and

   ``alias`` are optional. If ``port`` is not given, the default port ``11211``

   will be used. If given, ``alias`` will be used to compute the server hash,

   which would otherwise be computed based on ``host`` and ``port``

   (i.e. whichever portion is given).

-  ``do_split``: splits large values (up to 10MB) into chunks (<1MB). The

   memcached server implementation will not store items larger than 1MB,

   however in some environments it is beneficial to shard up to 10MB of data.

   Attempts to store more than that are ignored. Default: ``True``.

-  ``comp_threshold``: compresses large values using zlib. If

   ``buffer length > comp_threshold > 0`` (in bytes), the buffer will be

   compressed. If ``comp_threshold == 0``, the string buffer will never be

   compressed. Default: ``0``

-  ``noreply``: controls memcached's

   ``noreply`` `feature `__.

   Default: ``False``

-  ``prefix``: The key prefix. default: ``''``

-  ``hash_fn``: hashing function for keys. possible values:



   -  ``MC_HASH_MD5``

   -  ``MC_HASH_FNV1_32``

   -  ``MC_HASH_FNV1A_32``

   -  ``MC_HASH_CRC_32``



   default: ``MC_HASH_MD5``



   **NOTE:** fnv1\_32, fnv1a\_32, crc\_32 implementations in libmc are

   per each spec, but they're not compatible with corresponding

   implementions in libmemcached.



-  ``failover``: Whether to failover to next server when current server

   is not available. Default: ``False``



-  ``MC_POLL_TIMEOUT`` Timeout parameter used during set/get procedure.

   Default: ``300`` ms

-  ``MC_CONNECT_TIMEOUT`` Timeout parameter used when connecting to

   memcached server in the initial phase. Default: ``100`` ms

-  ``MC_RETRY_TIMEOUT`` When a server is not available due to server-end

   error, libmc will try to establish the broken connection in every

   ``MC_RETRY_TIMEOUT`` s until the connection is back to live. Default:

   ``5`` s



**NOTE:** The hashing algorithm for host mapping on continuum is always

md5.



Contributing to libmc

---------------------



Feel free to send a **Pull Request**. For feature requests or any

questions, please open an **Issue**.



For **SECURITY DISCLOSURE**, please disclose the information responsibly

by sending an email to [email protected] directly instead of creating

a GitHub issue.



FAQ

---



Does libmc support PHP?

^^^^^^^^^^^^^^^^^^^^^^^



No, but, if you like, you can write a wrapper for PHP based on the C++

implementation.



Is Memcached binary protocol supported ?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^



No. Only Memcached ASCII protocol is supported currently.



Why reinventing the wheel?

^^^^^^^^^^^^^^^^^^^^^^^^^^



Before libmc, we were using

`python-libmemcached `__,

which is a python extention for

`libmemcached `__.

libmemcached is quite weird and buggy. After nearly one decade, there're

still some unsolved bugs.



Is libmc thread-safe ?

^^^^^^^^^^^^^^^^^^^^^^



Yes. ``libmc.ThreadedClient`` is a thread-safe client implementation. To hold

access for more than one request, ``libmc.ClientPool`` can be used with Python

``with`` statements. ``libmc.Client``, however, is a single-threaded memcached

client. If you initialize a standard client in one thread but reuse that in

another thread, a Python ``ThreadUnsafe`` Exception will be raised.



Is libmc compatible with gevent?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^



Yes, with the help of `greenify `__,

libmc is friendly to gevent. Read ``tests/shabby/gevent_issue.py`` for

details. ``libmc.ThreadedClient`` and ``libmc.ClientPool`` are not compatible.

[#]_



**Notice:**



``gevent.monkey.patch_all()`` will override

``threading.current_thread().ident`` to Greenlet's ID,

this will cause libmc to throw a ThreadUnSafe error

or run into dead lock, you should only patch the things

that you need, e.g.



.. code:: python



    from gevent import monkey

    monkey.patch_socket()



Acknowledgments

---------------



-  Thanks to `@fahrenheit2539 `__ and

   the llvm project for the standalone.

   `SmallVector `__

   implementation.

-  Thanks to `@miloyip `__ for the high

   performance `i64toa `__

   implementation.

-  Thanks to `Ivan Novikov `__ for the

   research in `THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED

   INJECTIONS `__.

-  Thanks to the PolarSSL project for the md5 implementation.

-  Thanks to `@lericson `__ for the `benchmark

   script in

   pylibmc `__.

-  Thanks to the libmemcached project and some other projects possibly

   not mentioned here.



Contributors

------------



-  `@mckelvin `__

-  `@zzl0 `__

-  `@windreamer `__

-  `@lembacon `__

-  `@seansay `__

-  `@mosasiru `__

-  `@jumpeiMano `__



Who is using

------------



- `豆瓣 `__

- `下厨房 `__

- `Some other projects on GitHub `__

- Want to add your company/organization name here?

  Please feel free to send a PR!



Documentation

-------------



https://github.com/douban/libmc/wiki



Footnotes

---------



.. [#] In order to use a single executable for multiple greenlet contexts,

   gevent has to `copy thread memory

   `__

   to and from the same stack space. This doesn't affect Python references,

   which are handed off through gevent, but makes it impossible for shared

   libraries to pass memory addresses across greenlets, which is required for

   the worker pool.



LICENSE

-------



Copyright (c) 2014-2020, Douban Inc. All rights reserved.



Licensed under a BSD license:

https://github.com/douban/libmc/blob/master/LICENSE.txt



.. |build_go| image:: https://github.com/douban/libmc/actions/workflows/golang.yml/badge.svg

   :target: https://github.com/douban/libmc/actions/workflows/golang.yml



.. |build_py| image:: https://github.com/douban/libmc/actions/workflows/python.yml/badge.svg

   :target: https://github.com/douban/libmc/actions/workflows/python.yml



.. |pypiv| image:: https://img.shields.io/pypi/v/libmc

   :target: https://pypi.org/project/libmc/



.. |status| image:: https://img.shields.io/pypi/status/libmc

.. |pyversions| image:: https://img.shields.io/pypi/pyversions/libmc

.. |wheel| image:: https://img.shields.io/pypi/wheel/libmc

.. |license| image:: https://img.shields.io/pypi/l/libmc?color=blue