{"id":21644587,"url":"https://github.com/samsung/cas","last_synced_at":"2025-08-21T15:33:27.473Z","repository":{"id":40240597,"uuid":"478100096","full_name":"Samsung/CAS","owner":"Samsung","description":"Code Aware Services (CAS) is a set of tools for extracting information from a (especially large) source code trees. It consists of Build Awareness Service (BAS) and Function/Type database (FTDB). BAS is a tool for extracting information how particular S/W image is created from ongoing builds. FTDB transforms predefined source code information (like information about functions and types) into easily accessible format (like JSON) which can be used by a number of applications.","archived":false,"fork":false,"pushed_at":"2024-10-05T20:01:03.000Z","size":6455,"stargazers_count":45,"open_issues_count":0,"forks_count":7,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-12-09T00:43:57.704Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Samsung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-05T11:35:02.000Z","updated_at":"2024-11-26T16:14:57.000Z","dependencies_parsed_at":"2023-11-25T12:25:14.462Z","dependency_job_id":"2d17bee8-658b-409a-b022-4ca40eb417da","html_url":"https://github.com/Samsung/CAS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FCAS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FCAS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FCAS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FCAS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Samsung","download_url":"https://codeload.github.com/Samsung/CAS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230520392,"owners_count":18238948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T05:41:12.222Z","updated_at":"2024-12-20T01:16:39.706Z","avatar_url":"https://github.com/Samsung.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Code Aware Services\n\nCode Aware Services (CAS) is a set of tools for extracting information from a (especially large) source code trees. It consists of Build Awareness Service (BAS) and Function/Type database (FTDB). BAS is a tool for extracting information how particular S/W image is created from ongoing builds. FTDB transforms predefined source code information (like information about functions and types) into easily accessible format (like JSON) which can be used by a number of applications.\n\n### Introduction\n\nImagine you need to fix a S/W issue in a large S/W product (like Mobile Phone). This kind of project usually consists of a large number of S/W modules (like kernel, modem, system libraries and applications) communicating with each other. You need to make a general overview of the source code structure to find interesting source files and identify dependencies between modules. You also need detailed information how the S/W was built (i.e. exact compilation switches for specific source files) to be able to locate S/W errors that depend on specific configuration.\n\nBAS works by tracing the full S/W image build (using specialized Linux kernel module) and storing information about all opened files and executed processes. This is later post-processed to extract detailed compilation information from compilation processes. All that information is stored in database (JSON file) which can be accessed freely after the build.\n\nExamples of potential uses of BAS are:\n\n* extracting the list of all files from entire source tree which were used to create the final S/W image (which can be much smaller number (like ~10% in case of Linux kernel build) than the entire source repository) which can radically improve code search tools;\n\n* generating compilation database (i.e. JSON file with all compilation switches used to build the final S/W image) which can be later used by other tools operating on source code, like static analyzers, source code transformation tools, custom source processors etc.;\n\n* generating custom makefiles that can incrementally rebuild specifically selected parts of the source tree which can speed-up the change/build/test iteration during S/W development\n\n* generating code description files to various IDEs (like Eclipse IDE) which can significantly improve development process for user space applications (or OS kernel)\n\n* analyzing build related information to locate S/W errors\n\nand more.\n\nNow imagine that you want to write a tool that operates on source code, i.e. custom static analyzer which looks for specific erroneous patterns in the code and informs the developer about potential issues to focus on. Writing such tool requires a parsed source code representation to be available but writing parsers (especially for complex languages like C++) is extremely difficult. FTDB is a tool that uses existing libraries (i.e. libclang library) to parse the source code and store selected information from the parsed representation in an easily accessible format (i.e. JSON database). Such database can be later used by applications (i.e. Python programs) which can implement a set of distinct functionalities that takes source code information as an input.\n\nTogether BAS with FTDB forms a tightly related toolset which was collectively named CAS (Code Aware Services).\n\n\n### Details\n\nCAS is split into three separate functionalities (stored in three separate source directories):\n* build tracer ([tracer](tracer) directory)\n\nThis is the Linux kernel module for tracing specific syscalls executed during build. The module file produced is **bas_tracer.ko**. More details about tracer operation can be found in [readme](tracer/README.md) file.\n\n* BAS tools ([bas](bas) directory)\n\nThis is a set of user space tools used to process and operate on the trace file produced by the build tracer. More details can be found in [readme](bas/README.md) file.\n\n\n* FTDB processor ([clang-proc](clang-proc) directory)\n\nThis is the clang processor used to parse source files (currently only C sources are supported (but C++, at least partiallyl is on its way)) to extract relevant information from the parsed representation and store it in simple JSON database file. More details can be found in [readme](clang-proc/README.md) file.\n\n### How to build\n\nFirst setup the build environment and download sources:\n```bash\nsudo apt install git cmake llvm clang libclang-dev python3-dev gcc-9-plugin-dev build-essential linux-headers-$(uname -r) python-futures flex bison libssl-dev\ngit clone https://github.com/Samsung/CAS.git \u0026\u0026 cd CAS\nexport CAS_DIR=$(pwd)\n```\n\nYou can build all components using any decent compiler however for FTDB processor to work you'll need `llvm` and `clang` in version at least 10 (up to 17). In standard Ubuntu 20.04 distribution the above command will install this version by default. In case of other OS arrangements please make sure that the `llvm-config --version` returns the proper supported number. Alternatively you can pass `-DLLVM_CONFIG_BIN=llvm-config-1x` to cmake invocation to properly setup the clang processor based on clang 1x libraries.\n\nBuilding the tracer:\n\n```bash\n(cd ${CAS_DIR}/tracer \u0026\u0026 make \u0026\u0026 sudo make modules_install)\n```\n\nSome kernel configurations may require modules to be signed. To do so, follow the instructions in [kernel module signing docs](https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html).\n\nAfter the tracer module is installed setup the tracing wrapper:\n```bash\n(cd ${CAS_DIR} \u0026\u0026 sudo ./etrace_install.sh)\n```\n\nThis will modify your sudoers file to allow for using the tracer kernel module by any user in the system. In case of problems with running the etrace command (i.e. asking for sudo password) please make sure that the following line is located at the end of the `/etc/sudoers` file\n```\n#includedir /etc/sudoers.d\n```\n\nBuilding CAS libraries:\n```bash\nmkdir -p ${CAS_DIR}/build_release \u0026\u0026 cd ${CAS_DIR}/build_release\n# This will configure to build with standard system compiler 'cc'\ncmake -DCMAKE_BUILD_TYPE=Release ..\n# Use this if you want to use clang explicitly\ncmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..\nmake -j16 \u0026\u0026 make install\n```\n\n### How to use\n\n**etrace** - this is the wrapper script that runs the command (most probably build) under the tracer.\n\n```bash\nexport PATH=${CAS_DIR}:$PATH\n# Run the COMMAND and save tracing log file (.nfsdb) in the current directory\n$ etrace \u003cCOMMAND\u003e\n# Run the COMMAND and save tracing log file (.nfsdb) in the WORK_DIR directory\n$ etrace -w \u003cWORK_DIR\u003e \u003cCOMMAND\u003e\n# Run the COMMAND and save tracing log file (.nfsdb) in the current directory (also keep kernel log file along the way)\n$ etrace -l \u003cCOMMAND\u003e\n# Run the COMMAND and save tracing log file (.nfsdb) in the WORK_DIR directory (also keep kernel log file along the way)\n$ etrace -w \u003cWORK_DIR\u003e -l \u003cCOMMAND\u003e\n```\n\n  All the above commands also produce `.nfsdb.stats` file. These are the statistics from the underlying ftrace infrastructure. Most common usage is to check for possible dropped events (meaning the underlying kernel buffer should be larger):\n\n  ```bash\n  $ cat .nfsdb.stats | grep dropped\n  ```\n\n  The `-l` option saves the kernel log file (dmesg) during execution to the `.nfsdb.klog` file. This can be used for debugging of the tracer kernel module.\n\n  Beware, whole kernel log file is saved (even events before the execution of the tracer). To get rid of those use `sudo dmesg -C` before running the tracer.\n\n**etrace_parser** - this will parse the tracing log file (.nfsdb) and produce the build information in the form of simple JSON file (for the description of JSON format please refer to the [readme](bas/README.md) file).\n\n**cas** - this is the command line tool that controls various CAS suite operations. It can create the BAS database or query it for specific data. It provides high level functionalities that takes the BAS database as a source of data.\n\nTo parse the tracing log file navigate to the directory where the `.nfsdb` file is located and simply execute the following:\n```bash\ncas parse\n```\nThe result is written into `.nfsdb.json` file.\n\nAnother step is to post-process the `.nfsdb.json` file and include additional information regarding compilations and other.\n\n```bash\ncas postprocess\n```\n\nIf your first-stage parsing `.nfsdb.json` is located in different directory than the root of your source tree you can provide the source root information using the `--set-root` option:\n```bash\ncas postprocess --set-root=\u003cpath to source root directory\u003e\n```\n\nThere are times when the post-processed JSON file is really large (in case of full Android builds for latest OS version it can easily take few GB of size). Reading such JSON can be very time consuming and take a few minutes just to parse it. There is option to create a specialized cached version of the JSON file strictly dedicated for reading only. This will also (try to) precompute the file dependency information for all linked modules created during the build and save it into the (separate) cache file.\n\n```bash\ncas cache [--set-version=\u003cversion string\u003e] [--set-root=\u003cpath to source root directory\u003e] [--shared-argvs=\u003cshared linking spec\u003e] [--jobs=\u003cint\u003e] [--deps-threshold DEPS_THRESHOLD]\n```\n\nThis will produce the `.nfsdb.img` file with in-memory representation of the parsed JSON data which can be read back into memory almost instantly.\n\nThe shared linking spec provides a list (delimited by `:`) of linker switches (that were used during the build) used to tell a linker that it should produce a shared library. For most of the possible usages it should be enough to provide `-shared:--shared` values (it's actually the default value).\n\nThe number of jobs used during the cache creation specifies how many processors should be used for the file dependency computation of the linked modules (which defaults to the number of cores available in the system).\n\nWhen cache file is created the list of dependencies (both direct and full dependency list) for each processed module is created. If a number of computed dependencies for a given module exceeds some predefined value (90k at the current default implementation) the error is produced and the cache creation stops. The dependency threshold can be overriden using the `--deps-threshold` (or `-DT`) option.\n\nThe cache creation process can be also split into two independent phases. When the `--create` option is passed only the core BAS database is saved into the cache file. Executing next with option `--deps-create` will compute the dependency list for each module and add that information to the cache as well.\n\nFor more information regarding the `cas` command line tool (and other clients to access the database content) please see [client readme](CLIENT.md) file.\n\n### Examples\n\nLet's see some quick introduction through several simple examples. Let's build our CAS suite again but this time with the tracer enabled.\n\n```bash\ncd ${CAS_DIR}/build_release\nmake clean\netrace make -j16 \u0026\u0026 make install\ncas parse\ncas postprocess --set-root=${CAS_DIR}\ncas cache --set-root=${CAS_DIR}\n```\n \nNow strike up some editor and write-up the following Python code:\n```python\n#!/usr/bin/env python3\n\nimport sys\nimport libetrace\n\nnfsdb = libetrace.nfsdb()\nnfsdb.load(sys.argv[1],quiet=True)\n\nprint (\"Source root of the build: %s\"%(nfsdb.source_root))\nprint (\"Database version: %s\"%(nfsdb.dbversion))\n\n# List all linked modules\nL = [x for x in nfsdb if x.is_linking()]\nprint (\"# Linked modules\")\nfor x in L:\n    print (\"  %s\"%(x.linked_path))\nprint()\n\n# List compiled files\nC = [x for x in nfsdb if x.has_compilations()]\nprint (\"# Compiled files\")\nfor x in C:\n    for u in x.compilation_info.file_paths:\n        print (\"  %s\"%(u))\nprint()\n\n# List all compilation commands\nprint (\"# Compilation commands\")\nfor x in C:\n    print (\"$ %s\\n\"%(\" \".join(x.argv)))\n```\n\nOr try existing example:\n```bash\nexport PYTHONPATH=${CAS_DIR}:$PYTHONPATH\n${CAS_DIR}/examples/etrace_show_info .nfsdb.img\n```\n\nThis should give you a list of all linked modules, compiled files and compilation commands used during the CAS suite build.\n\nWe can generate FTDB database for all C compiled files.\n```bash\nexport CLANG_PROC=${CAS_DIR}/clang-proc/clang-proc\n${CAS_DIR}/examples/extract-cas-info-for-ftdb .nfsdb.img\n${CAS_DIR}/clang-proc/create_json_db -P $CLANG_PROC\n${CAS_DIR}/tests/ftdb_cache_test --only-ftdb-create db.json\n```\n\nFirst command extracts the compilation database (i.e. list of all compilation commands used to compile the program which is later used by the clang processor). Second command creates the FTDB database (`db.json` file in our case). Finally we create the cached version of the FTDB database (you can perfectly use the original database `db.json` file if that suits you however as soon will be shown this JSON file can also get really large very quickly for complex modules).\n\nSome simple information regarding the FTDB database:\n```bash\n${CAS_DIR}/examples/ftdb_show_info db.img \n```\n\nOk, let's see some more serious example now. First clone Linux kernel source for AOSP version 12 (make sure you have a few GB of space available).\n\n```bash\ncd ${CAS_DIR}\nexamples/clone-avdkernel.sh\n```\n\nSources should be downloaded to `avdkernel` directory. To build it you'll need clang at least in version 11.\n```bash\nsudo apt-get install clang-11 llvm-11 lld-11\ncd avdkernel\netrace ${CAS_DIR}/examples/build-avdkernel.sh\ncas parse\ncas postprocess\ncas cache\n```\n\nWe will now extract information required to create the FTDB database for the Linux kernel (and all accompanying modules). This time we will get some more information as we will build specialized version of FTDB suited for the automated off-target generation (please see [AoT](https://github.com/Samsung/auto_off_target) project for more details).\n\n```bash\n${CAS_DIR}/examples/extract-avdkernel-info-for-ftdb .nfsdb.img\n```\n\nYou should see confirmation that three files were created along the way. Apart from compilation database we also extract compilation dependency map (i.e. list of compiled files for every module of interest) and reverse dependency map (i.e. for every source file we get information which module used that file).\n```\ncreated compilation database file (compile_commands.json)\ncreated compilation dependency map file (cdm.json)\ncreated reverse dependency map file (rdm.json)\n```\n\nNow you're ready for FTDB generation.\n```bash\n${CAS_DIR}/clang-proc/create_json_db -o db.json -P $CLANG_PROC -m android_common_kernel -V \"12.0.0_r15-eng\" -mr ${CAS_DIR}/clang-proc/vmlinux-macro_replacement.json -A -DD ${CAS_DIR}/clang-proc/vmlinux-additional_defs.json -cdm cdm.json -j4\n```\n\nThe invocation crates the `db.json` file (with some additions helpful for the `AoT` project). If you have infinite amounts of RAM you can skip the `-j4` option and it'll run on full speed. Otherwise I suggest to keep it that way.\n\nThe JSON database reached a few GB of size. Now it's fully justified to use the cache.\n```bash\n${CAS_DIR}/tests/ftdb_cache_test --only-ftdb-create db.json\n${CAS_DIR}/examples/ftdb_show_info db.img\n```\n\nNow the FTDB database can be used for further analysis and application development.\n\n\n### Running in docker\n\nRunning in docker environment requires some extended capabilities so running with --privileged switch is recommended. Also --pid=\"host\" is required for proper capture of tracing data from mounted `/sys/kernel/debug`. \n\nContainer should enable user to load kernel module from host so `/lib/modules/` must be mounted and `kmod` system package are required.\n\nExample of Ubuntu 22.04 usage:\n\n```\ndocker run -it --pid=\"host\" --privileged -v /lib/modules/:/lib/modules/ -v /sys/kernel/debug:/sys/kernel/debug -v ${CAS_DIR}:/cas ubuntu:22.04\n\n// Setup\nroot@5f01ed8691a8:/# apt update\nroot@5f01ed8691a8:/# apt install kmod\nroot@5f01ed8691a8:/# cd /cas/\nroot@5f01ed8691a8:/cas# ./etrace_install.sh\n// Testing\nroot@5f01ed8691a8:/cas# cd /tmp/\nroot@5f01ed8691a8:/tmp# /cas/etrace ls /\n// Verify\nroot@5f01ed8691a8:/tmp# ll /tmp/\ntotal 20\ndrwxrwxrwt 1 root root 4096 Jan 10 09:46 ./\ndrwxr-xr-x 1 root root 4096 Jan 10 09:46 ../\n-rw-r--r-- 1 root root 8114 Jan 10 09:46 .nfsdb\n-rw-r--r-- 1 root root 1363 Jan 10 09:46 .nfsdb.stats\n```\n\nTracing is working fine if .nfsdb contains more than one line.\n\n### Virtual Environment\n\nFor those who are cautious to load and run custom Linux kernel modules on their Linux machines there is also virtual environment (based on QEMU) provided. Please refer to the [readme](tools/virtual_environment/README.md) file.\n\n### Talks\n\nTalk about Code Aware Services from the Linux Security Summit NA 2022 can be found [here](https://youtu.be/M7gl7MFU_Bc?t=648)\n\nTalk about how Code Aware Services can be used to improve the developer productivity in the complex low-level systems world can be found [here](https://www.youtube.com/watch?v=FZrhHgor4NE\u0026t=1s)\n\n### Documentation\n\nUsing CAS as a [DPE tool](https://samsung.github.io/CAS/) in the Complex Low-Level Systems World\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fcas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamsung%2Fcas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fcas/lists"}