{"id":18627500,"url":"https://github.com/yugr/shlibvisibilitychecker","last_synced_at":"2025-04-11T05:31:44.732Z","repository":{"id":41437120,"uuid":"130516320","full_name":"yugr/ShlibVisibilityChecker","owner":"yugr","description":"Tool for locating internal symbols unnecessarily exported from shared libraries.","archived":false,"fork":false,"pushed_at":"2022-12-03T12:01:49.000Z","size":105,"stargazers_count":32,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-03-08T22:25:48.659Z","etag":null,"topics":["abi","interfaces","shared-library","static-analysis","visibility"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yugr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-21T22:22:31.000Z","updated_at":"2023-03-01T14:17:53.000Z","dependencies_parsed_at":"2023-01-23T05:00:32.313Z","dependency_job_id":null,"html_url":"https://github.com/yugr/ShlibVisibilityChecker","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yugr%2FShlibVisibilityChecker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yugr%2FShlibVisibilityChecker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yugr%2FShlibVisibilityChecker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yugr%2FShlibVisibilityChecker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yugr","download_url":"https://codeload.github.com/yugr/ShlibVisibilityChecker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223460474,"owners_count":17148756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abi","interfaces","shared-library","static-analysis","visibility"],"created_at":"2024-11-07T04:42:36.595Z","updated_at":"2024-11-07T04:42:37.299Z","avatar_url":"https://github.com/yugr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](http://img.shields.io/:license-MIT-blue.svg)](https://github.com/yugr/ShlibVisibilityChecker/blob/master/LICENSE.txt)\n[![Build Status](https://github.com/yugr/ShlibVisibilityChecker/actions/workflows/ci.yml/badge.svg)](https://github.com/yugr/ShlibVisibilityChecker/actions)\n[![codecov](https://codecov.io/gh/yugr/ShlibVisibilityChecker/branch/master/graph/badge.svg)](https://codecov.io/gh/yugr/ShlibVisibilityChecker)\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/yugr/ShlibVisibilityChecker.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/yugr/ShlibVisibilityChecker/alerts/)\n[![Coverity Scan](https://scan.coverity.com/projects/yugr-ShlibVisibilityChecker/badge.svg)](https://scan.coverity.com/projects/yugr-ShlibVisibilityChecker)\n\n# What's this?\n\nShlibVisibilityChecker is a small tool which locates internal symbols\nthat are unnecessarily exported from shared libraries.\nSuch symbols are undesirable because they cause\n* slower startup time (due to [slower relocation processing by dynamic linker](https://lwn.net/Articles/341309/), see a [real-world example](https://lore.kernel.org/lkml/CAKwvOdk0nxxUATg2jEKgx4HutXCMXcW92SX3DT+uCTgqBwQHBg@mail.gmail.com/) for Linux kernel)\n* performance slowdown (due to indirect function calls, compiler's inability\n  to optimize exportable functions e.g. inline them, effective turnoff of `--gc-sections`)\n* leak of implementation details\n  (if some clients start to use private functions instead of regular APIs)\n* bugs due to runtime symbol clashing\n  - [crash in Apache due to symbol clash with libasn1](https://github.com/DCIT/perl-CryptX/issues/68)\n  - more real-world examples in [Flameeyes blog](https://flameeyes.blog/2008/02/09/flex-and-linking-conflicts-or-a-possible-reason-why-php-and-recode-are-so-crashy/)\n\nShlibVisibilityChecker compares APIs declared in public headers\nagainst APIs exported from shared libraries and warns about discrepancies.\nIn majority of cases such symbols are internal library symbols which should be hidden\n(in rare cases these are internal symbols which are used by other libraries or executables\nin the same package and `shlibvischeck-debian` tries hard to not report such cases).\n\nSuch discrepancies should then be fixed by recompiling package\nwith `-fvisibility=hidden` (see [here](https://gcc.gnu.org/wiki/Visibility) for details).\nA typical fix, for a typical Autoconf project can be found\n[here](https://github.com/cacalabs/libcaca/issues/33#issuecomment-387656546).\n\nShlibVisibilityChecker _not_ meant to be 100% precise but rather provide assistance in locating packages\nwhich may benefit the most from visibility annotations (and to understand how bad the situation\nwith visibility is in modern distros).\n\n# How to use\n\nTo check a raw package, i.e. a bunch of headers and shared libs,\ncollect source and binary interfaces and compare them:\n```\n$ bin/read_header_api --only-args /usr/include/xcb/* \u003e api.txt\n$ ./read_binary_api --permissive /usr/lib/x86_64-linux-gnu/libxcb*.so \u003e abi.txt\n$ vimdiff api.txt abi.txt  # Or `comm -13 api.txt abi.txt'\n```\n\nAnother useful scenario is locating symbols that are exported from\nDebian package's shared libraries but are not declared in it's headers.\nThe main tool for this is a `shlibvischeck-debian` script.\n\nTo apply it to a package, run\n```\n$ shlibvischeck-debian libacl1\nThe following exported symbols in package 'libacl1' are private:\n  __acl_extended_file\n  __acl_from_xattr\n  __acl_to_xattr\n  __bss_start\n  _edata\n  _end\n  _fini\n  _init\n  closed\n  head\n  high_water_alloc\n  next_line\n  num_dir_handles\n  walk_tree\n```\nTo skip autogenerated symbols like `_init` or `_edata` (caused by [ld linker scripts](https://sourceware.org/ml/binutils/2018-04/msg00326.html) and [libgcc startup files](https://gcc.gnu.org/ml/gcc-help/2018-04/msg00097.html)) add `--permissive`.\n\nYou can also check visibility issues in arbitrary set of headers and libraries:\n```\n$ shlibvischeck-common --permissive --cflags=\"-I/usr/include -I$AUDIT_INSTALL/include -I/usr/lib/llvm-5.0/lib/clang/5.0.0/include\" $AUDIT_INSTALL/include/*.h $AUDIT_INSTALL/lib/*.so*\n```\n\n# How to install\n\nBuild-time prerequisites are `python3` (`setuptools` module), `clang`,\n`llvm`, `libclang-dev`, `g++` and `make`.\nRun-time dependencies are `python3` (`python-magic` module), `pkg-config` and `aptitude`.\nTo install everything on Ubuntu, run\n```\n$ sudo apt-get install python3 clang llvm libclang-dev g++ make pkg-config aptitude\n$ sudo python3 -mensurepip\n$ sudo pip3 install setuptools python-magic\n```\n(you could also use script `scripts/install-deps.sh`).\n\nYou also need to enable access to Ubuntu source packages via\n```\n$ sudo sed -Ei 's/^# *deb-src /deb-src /' /etc/apt/sources.list\n$ sudo apt-get update\n```\n\nPython and binary components are built separately:\n```\n$ make clean all \u0026\u0026 make install\n$ ./setup.py build \u0026\u0026 pip3 install .\n```\n\nDuring analysis `shlibvischeck-debian` installs new Debian packages so it's recommended to run it under chroot or in VM.\nThere are many instructions on setting up chroot e.g. [this one](https://github.com/yugr/debian_pkg_test).\n\n# Where to find packages\n\nA list of packages for analysis can be obtained from [Debian rating](https://popcon.debian.org/by_vote):\n```\n$ curl https://popcon.debian.org/by_vote | awk '/^[0-9]+ +lib/{print $2}' \u003e by_vote\n$ shlibvischeck-debian $(head -500 by_vote | tr '\\n' ' ')\n```\n\n# How to fix a package\n\nOnce you found a problematic package, you can fix it by restricting visibility of internal symbols.\nThe best way to control symbol visibility in a package is to\n* hide all symbols by default by adding `-fvisibility=hidden` to `CFLAGS` in project buildscripts\n  (`Makefile.in` or `CMakeLists.txt`)\n* explicitly annotate publicly visible functions with\n  `__attribute__((visibility(\"default\")))`\n\nSee [fix in libcaca](https://github.com/cacalabs/libcaca/pull/34/files)\nfor example.\n\n# Issues and limitations\n\nAt the moment tool works only on Debian-based systems (e.g. Ubuntu).\nThis should be fine as buildscripts are the same across all distros\nso detecting issues on Ubuntu would serve everyone else too.\n\nAn important design issue is that the tool can not detect symbols which are used indirectly\ni.e. not through an API but through `dlsym` or explicit out-of-header prototype declaration\nin source file. This happens in plugins or tightly interconnected shlibs within the same project.\nSuch cases should hopefully be rare.\n\nShlibVisibilityChecker is a heuristic tool so it will not be able to analyze all packages.\nCurrent success rate is around 60%.\nMajor reasons for errors are\n* badly-structured headers i.e. the ones which do not \\#include all their dependencies \n  (e.g. `libatasmart` [fails to include `stddef.h`](https://github.com/Rupan/libatasmart/issues/1)\n  and `tdb` [fails to include `sys/types.h`](https://bugzilla.samba.org/show_bug.cgi?id=13398)).\n* internal headers which should not be \\#included directly (e.g. `lzma/container.h`)\n* experimental headers which require custom macro definitions (not listed in\n  pkgconfig) (e.g. `dpkg/macros.h` requires `LIBDPKG_VOLATILE_API`)\n* missing dependencies (e.g. `libverto-dev` uses Glib headers but does not declare this)\n\nOther issues:\n* TODOs are scattered all over the codebase\n* would be interesting to go over dependent packages and check if they use invalid symbols\n\n# Trophies\n\nThe tool found huge number of packages that lacked visibility annotations (in practice every second package\nhas spurious exports). Here are some which I tried to fix:\n\n* Bzip2: [Hide unused symbols in libbz2](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896750)\n* Expat: [Private symbols exported from shared library](https://github.com/libexpat/libexpat/issues/195) (fixed)\n* Libaudit: [Exported private symbols in audit-userspace](https://www.redhat.com/archives/linux-audit/2018-April/msg00119.html) (partially fixed)\n* Gdbm: [sr #347: Add visibility annotations to hide private symbols](https://puszcza.gnu.org.ua/support/index.php?347)\n* Libnfnetfilter: [\\[RFC\\]\\[PATCH\\] Hide private symbols in libnfnetlink](https://marc.info/?l=netfilter-devel\u0026m=152481166515881) (fixed)\n* Libarchive: [Hide private symbols in libarchive.so](https://github.com/libarchive/libarchive/issues/1017) ([fixed](https://github.com/libarchive/libarchive/pull/1751))\n* Libcaca: [Hide private symbols in libcaca](https://github.com/cacalabs/libcaca/issues/33) (fixed)\n* Libgmp: [Building gmp with -fvisibility=hidden](https://gmplib.org/list-archives/gmp-discuss/2018-April/006229.html)\n* Vorbis: [Remove private symbols from Vorbis shared libs](https://github.com/xiph/vorbis/issues/43)\n\nMore perspective packages (from Debian top-100): libpopt1, libgpg-error0, libxml2, libwrap0, libpcre3, libkeyutils1, libedit2, liblcms2-2.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyugr%2Fshlibvisibilitychecker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyugr%2Fshlibvisibilitychecker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyugr%2Fshlibvisibilitychecker/lists"}