{"id":16828454,"url":"https://github.com/jameslamb/lgb-glibc-demo","last_synced_at":"2026-01-04T00:17:07.317Z","repository":{"id":79614391,"uuid":"484857485","full_name":"jameslamb/lgb-glibc-demo","owner":"jameslamb","description":"Code and notes from investigating https://github.com/microsoft/LightGBM/issues/5106","archived":false,"fork":false,"pushed_at":"2022-04-25T04:25:14.000Z","size":26,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T01:54:24.557Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jameslamb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-23T21:11:59.000Z","updated_at":"2024-07-11T14:59:51.000Z","dependencies_parsed_at":"2023-05-13T14:30:50.815Z","dependency_job_id":null,"html_url":"https://github.com/jameslamb/lgb-glibc-demo","commit_stats":{"total_commits":7,"total_committers":1,"mean_commits":7.0,"dds":0.0,"last_synced_commit":"ed84dc8bd9232cd3578be6a4b08596fa7ff8f4b7"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameslamb%2Flgb-glibc-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameslamb%2Flgb-glibc-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameslamb%2Flgb-glibc-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jameslamb%2Flgb-glibc-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jameslamb","download_url":"https://codeload.github.com/jameslamb/lgb-glibc-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244109599,"owners_count":20399559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:26:44.735Z","updated_at":"2026-01-04T00:17:07.277Z","avatar_url":"https://github.com/jameslamb.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lgb-glibc-demo\n\nCode and documentation for investigating the error described in https://github.com/microsoft/LightGBM/issues/5106.\n\nWhen LightGBM's Python package is build from source on Linux, under certain conditions compilation can succeed by importing the library can fail with an error similar to the following.\n\n\u003e OSError: /root/miniforge/bin/../lib/libstdc++.so.6: version 'GLIBCXX_3.4.30' not found (required by /usr/local/src/LightGBM/python-package/compile/lib_lightgbm.so)\n\n## Reproducible Examples\n\nThis describes the steps to reproduce the error.\n\nFirst, build an Ubuntu 22.04 container with the necessary system libraries set up and `conda` installed.\n\n```shell\nmake build\n```\n\nNext, try building `lightgbm` from source and installing it in the `base` conda environment.\n\nInstallation and importing the library should succeed.\n\n```shell\ndocker run \\\n    --rm \\\n    --workdir /usr/local/src/LightGBM/python-package \\\n    -it lgb-glibc-demo:local \\\n    /bin/bash -c \"pip install . \u0026\u0026 python -c 'import lightgbm'\"\n```\n\nBut installing `libcxx-ng` (or anything that results in that being installed, like `dask`) will cause importing `lightgbm` to fail.\n\n```shell\n# fails\ndocker run \\\n    --rm \\\n    --workdir /usr/local/src/LightGBM/python-package \\\n    -it lgb-glibc-demo:local \\\n    /bin/bash -c \"conda install -y -n base libstdcxx-ng \u0026\u0026 pip install . \u0026\u0026 python -c 'import lightgbm'\"\n```\n\n\u003e OSError: /root/miniforge/bin/../lib/libstdc++.so.6: version 'GLIBCXX_3.4.30' not found (required by /usr/local/src/LightGBM/python-package/compile/lib_lightgbm.so)\n\n```shell\n# fails\ndocker run \\\n    --rm \\\n    --workdir /usr/local/src/LightGBM/python-package \\\n    -it lgb-glibc-demo:local \\\n    /bin/bash -c \"conda install -y -n base dask \u0026\u0026 pip install . \u0026\u0026 python -c 'import lightgbm'\"\n```\n\n\u003e OSError: /root/miniforge/bin/../lib/libstdc++.so.6: version 'GLIBCXX_3.4.30' not found (required by /usr/local/src/LightGBM/python-package/compile/lib_lightgbm.so)\n\n## Investigation\n\n### Root Cause\n\nTThe root cause of this issue is as follows:\n\n\u003e When `lib_lightgbm.so` is compiled with `gcc` not managed by `conda`, the compiler links against a C++ implementation in a location like `/usr/lib/x86_64-linux-gnu/libstdc++.so.6`.\n\u003e\n\u003e That linking is *dynamic*...it's expected that when the library is loaded in a Python session, `libstdc++.so.6` will be loaded at runtime.\n\u003e\n\u003e `conda`'s distribution of Python\n\u003e\n\u003e Using a `conda` distribution of CPython-based Python, when `lib_lightgbm.so` is loaded using `ctypes.cdll.LoadLibrary()`, if there are any `libstdc++.so.6` in `conda`'s lib paths, one of those will be loaded instead of the one in `/usr/lib`.\n\u003e\n\u003e This is because `conda` alters `ctypes`'s behavior to prefer directories managed by `conda`.\n\nAs described in https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html, GNU C++ is architected for forward compatibility.\n\n\u003e It is not possible to take program binaries linked with the latest version of a library binary in a release series (with additional symbols added), substitute in the initial release of the library binary, and remain link compatible.\n\nSo if the C++ compiler links against a given `libstdc++.so` at build time, then at runtime it's necessary to link against a version of `libstdc++.so` that is *at least that new*.\n\nSo the issue here comes in when the `libstdc++.so` version available from the operating system (the one linked to by like `/usr/bin/g++`, for example) is newer than whatever ones are found in a given conda environment when loading the library.\n\nThere are two mechanisms by which errors like this can result from trying to load a `.so`/ `.dll` in `conda`'s Python:\n\n1. via `conda`'s patches to `ctypes` and Python itself\n2. beccause of the `RPATH` values se on `conda`'s \n\n`conda` makes this situation much more likely via its patch(es?) to `ctypes`.\n\n```shell\nLIB_LIGHTGBM='/root/miniforge/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so'\n\n# fails with conda Python\n/root/miniforge/bin/python -c \\\n    \"import ctypes; ctypes.cdll.LoadLibrary('${LIB_LIGHTGBM}')\"\n\n# succeeds with non-conda Python\n/usr/bin/python3 -c \\\n    \"import ctypes; ctypes.cdll.LoadLibrary('${LIB_LIGHTGBM}')\"\n```\n\n### Research\n\n## Workarounds that do not require modifying LightGBM\n\nCode below should be run inside a container using the base image created above.\n\n```shell\ndocker run \\\n    --rm \\\n    --workdir /usr/local/src/LightGBM/python-package \\\n    -it lgb-glibc-demo:local \\\n    /bin/bash\n```\n\n1. Use `conda`'s CMake and compilers to build LightGBM from source.\n\nFrom https://conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#using-the-compiler-packages\n\n\u003e Instead of `gcc`, the executable name of the compiler you use will be something like `x86_64-conda_cos6-linux-gnu-gcc`.\n\n\u003e Many build tools such as make and CMake search by default for a compiler named simply `gcc`, so we set environment variables to point these tools to the correct compiler.\n\n\u003e We set these variables in conda activate.d scripts, so any environment in which you will use the compilers must first be activated so the scripts will run. Conda-build does this activation for you using activation hooks installed with the compiler packages in `CONDA_PREFIX/etc/conda/activate.d`.\n\n```shell\n# install the problematic library\nconda install -y -n base \\\n    libstdcxx-ng\n\n# confirm that it resulting in a `libstdc++.so.6` being added in conda env\nfind / -name 'libstdc++.so.6'\n# /root/miniforge/lib/libstdc++.so.6\n# /root/miniforge/pkgs/libstdcxx-ng-11.2.0-he4da1e4_16/lib/libstdc++.so.6\n# /usr/lib/x86_64-linux-gnu/libstdc++.so.6\n\n# get conda compilers\nconda install -y -n base \\\n    cmake \\\n    gcc_linux-64 \\\n    gxx_linux-64\n\n# it's important to activate the target conda env, to set\n# the relevant environment variables pointing to conda's compilers\nsource activate base\n\n# you can see the effect of this by checking env variables\necho $CC\n# /root/miniforge/bin/x86_64-conda-linux-gnu-cc\n\necho $CXX\n# /root/miniforge/bin/x86_64-conda-linux-gnu-c++\n\ncd /usr/local/src/LightGBM\npip uninstall -y lightgbm\nrm -rf ./build\nrm -f ./lib_lightgbm.so\n\ncd ./python-package\npip install .\n\n# confirm that importing works\npython -c \"import lightgbm; print(lightgbm.__version__)\"\n# 3.3.2.99\n\n# confirm that the maximum GLIBCXX version is less than\n# the one from the error message, and that the libstdc++.so.6 linked\n# is the one from /root/miniforge\nLIB_LIGHTGBM_IN_CONDA=$(\n    find /root/miniforge -name 'lib_lightgbm.so' \\\n    | head -1\n)\nldd -v \\\n    \"${LIB_LIGHTGBM_IN_CONDA}\"\n```\n\n2. Set `LD_PRELOAD` prior to loading `libstdc++.so.6`.\n\n```shell\n# install the problematic library\nconda install -y -n base \\\n    libstdcxx-ng\n\n# confirm that it resulted in a `libstdc++.so.6` being added in conda env\nfind / -name 'libstdc++.so.6'\n# /root/miniforge/lib/libstdc++.so.6\n# /root/miniforge/pkgs/libstdcxx-ng-11.2.0-he4da1e4_16/lib/libstdc++.so.6\n# /usr/lib/x86_64-linux-gnu/libstdc++.so.6\n\n# build LightGBM from source\ncd /usr/local/src/LightGBM\npip uninstall -y lightgbm\nrm -rf ./build\nrm -f ./lib_lightgbm.so\ncd ./python-package\npip install .\n\n# try loading lightgbm (this will fail)\npython -c \"import lightgbm; print(lightgbm.__version__)\"\n\n# try loading lightgbm with LD_LIBRARY_PATH set to the same paths\n# referenced in lib_lightgbm.so\nLD_PRELOAD=\"${LD_PRELOAD}:/usr/lib/x86_64-linux-gnu/libstdc++.so.6\" \\\npython -c \"import lightgbm; print(lightgbm.__version__)\"\n```\n\nNOTE: this cannot be done from inside Python. The following code will fail.\n\n```python\nimport os\nos.environ[\"LD_PRELOAD\"] = \"/usr/lib/x86_64-linux-gnu/libstdc++.so.6\"\nimport lightgbm\n```\n\n3. Modify `lib_lightgbm.so`'s DT_RPATH tag so that it points at the place where it found `libstdc++.so.6`.\n\nSee https://man7.org/linux/man-pages/man3/dlopen.3.html and https://stackoverflow.com/a/20333550/3986677.\nrpath is a way to embed a hint about where to find include dirs in a shared object.\n\n```shell\ncd /root/miniforge/lib/python3.9/site-packages/lightgbm/\ncp lib_lightgbm.so lib_lightgbm2.so\n\n# shows no rpath\nchrpath -l lib_lightgbm2.so\n\n# fails\npython -c \\\n    \"import ctypes; ctypes.cdll.LoadLibrary('/root/miniforge/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so')\"\n\n# patch the rpath\npatchelf --set-rpath '/usr/lib/x86_64-linux-gnu' lib_lightgbm2.so\n\n# shows rpath\nchrpath -l lib_lightgbm2.so\n\n# succeeds!\npython -c \\\n    \"import ctypes; ctypes.cdll.LoadLibrary('/root/miniforge/lib/python3.9/site-packages/lightgbm/lib_lightgbm2.so')\"\n```\n\n4. Inspect `lib_lightgbm.so` to figure out where the compiler found certain libraries, and copy any found from outside `conda` libraries into `conda`'s `lib/` directory.\n\n```shell\n# install the problematic library\nconda install -y -n base \\\n    libstdcxx-ng\n\n# confirm that it resulted in a `libstdc++.so.6` being added in conda env\nfind / -name 'libstdc++.so.6'\n# /root/miniforge/lib/libstdc++.so.6\n# /root/miniforge/pkgs/libstdcxx-ng-11.2.0-he4da1e4_16/lib/libstdc++.so.6\n# /usr/lib/x86_64-linux-gnu/libstdc++.so.6\n\ncd /usr/local/src/LightGBM\npip uninstall -y lightgbm\nrm -rf ./build\nrm -f ./lib_lightgbm.so\n\ncd ./python-package\npip install .\n\n# try loading lightgbm (this will fail)\npython -c \"import lightgbm; print(lightgbm.__version__)\"\n\n# find libraries linked against lib_lightgbm.so but not in conda's lib path\nLIB_LIGHTGBM_IN_CONDA=$(\n    find \"${CONDA}\" -name 'lib_lightgbm.so' \\\n    | head -1\n)\nLINKED_LIBRARIES_NOT_IN_CONDA_LIB_PATH=$(\n    ldd ${LIB_LIGHTGBM_IN_CONDA} \\\n    | grep -oP '(?=\\\u003e ).*(?= )' \\\n    | tr -d '\\\u003e ' \\\n    | grep -v -E \"^${CONDA}\"\n)\n\nfor libfile in ${LINKED_LIBRARIES_NOT_IN_CONDA_LIB_PATH}; do\n    echo \"copying '${libfile}' into ${CONDA}/lib\"\n    cp \"${libfile}\" \"${CONDA}/lib/\"\ndone\n\n# now this should work\npython -c \"import lightgbm; print(lightgbm.__version__)\"\n# 3.3.2.99\n```\n\n## Ways LightGBM could mitigate this issue\n\n1. Try setting RPATH at build time to create a \"relocatable executable\"\n    - either by setting `LDFLAGS` or by using `patchelf` after compilation\n    - see https://nehckl0.medium.com/creating-relocatable-linux-executables-by-setting-rpath-with-origin-45de573a2e98\n2. In `setup.py`, after compilation (incl. when using `--precompile`), detect that we're using `conda`, then run `ldd` to check what is linked into `lib_lightgbm.so`, and try to copy into the current conda environment's `lib/` dir any shared objects that are not already found there.\n    - conda patches `sys.prefix` to be the path to the current conda env, so that can be relied on\n3. try-catch library loading, and if an `OSError` about missing libraries is raised, at least raise an informative error describing the workarounds listed above\n\n## Attempted workarounds that do not work\n\n## References\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameslamb%2Flgb-glibc-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjameslamb%2Flgb-glibc-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameslamb%2Flgb-glibc-demo/lists"}