{"id":23190215,"url":"https://github.com/lefred/mysql-component-profiler","last_synced_at":"2025-04-05T06:41:32.348Z","repository":{"id":267516147,"uuid":"872653620","full_name":"lefred/mysql-component-profiler","owner":"lefred","description":"CPU and Memory profiler component for MySQL","archived":false,"fork":false,"pushed_at":"2025-01-13T16:51:07.000Z","size":1888,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-10T14:25:19.612Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lefred.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-14T20:26:47.000Z","updated_at":"2025-01-13T16:51:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"a01b5b77-4003-43a6-9b3b-ef2d994360a4","html_url":"https://github.com/lefred/mysql-component-profiler","commit_stats":null,"previous_names":["lefred/mysql-component-profiler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefred%2Fmysql-component-profiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefred%2Fmysql-component-profiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefred%2Fmysql-component-profiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefred%2Fmysql-component-profiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lefred","download_url":"https://codeload.github.com/lefred/mysql-component-profiler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299790,"owners_count":20916185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-18T12:13:48.084Z","updated_at":"2025-04-05T06:41:32.328Z","avatar_url":"https://github.com/lefred.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mysql-component-profiler\n\nThis is component collection to extend MySQL providing capabilities using gperftools (https://github.com/gperftools/gperftools).\n\nWhen using tcmalloc, you have the possibility to profile the memory, the CPU or both. \n\nWhen using jemalloc, you have the possibility to profile the memory.\n\nThe component uses `pprof` to generate text reports or dot output for tcmalloc and `jeprof` for jemalloc.\n\n## installation \u0026 prerequisities\n\nThere are 4 components:\n\n- `component_profiler.so`: the main one. It's a dependency of the other ones \n- `component_profiler_cpu.so`: to porfile CPU. It requires `component_profiler.so` to be installed.\n- `component_profiler_memory.so`: to profile memory. It requires `component_profiler.so` to be installed.\n- `component_profiler_jemalloc_memory.so`: to profile memory. It requires `component_profiler.so` to be installed.\n\n`component_profiler_cpu.so`, `component_profiler_memory.so` and `component_profiler_jemalloc_memory.so` can be installed independently.\n\nTo install the components copy the `component_profiler.so` and the memory and/or cpu files into the plugins directory, `/usr/lib64/mysql/plugin/` on Oracle Linux.\n\n### when using tcmalloc\n\n```\nMySQL \u003e install component 'file://component_profiler';\nQuery OK, 0 rows affected (0.0017 sec)\n\nMySQL \u003e install component 'file://component_profiler_cpu';\nQuery OK, 0 rows affected (0.0017 sec)\n\nMySQL \u003e install component 'file://component_profiler_memory';\nQuery OK, 0 rows affected (0.0017 sec)\n\nMySQL \u003e select * from mysql.component;\n+--------------+--------------------+----------------------------------+\n| component_id | component_group_id | component_urn                    |\n+--------------+--------------------+----------------------------------+\n|            1 |                  1 | file://component_profiler        |\n|            2 |                  2 | file://component_profiler_cpu    |\n|            3 |                  3 | file://component_profiler_memory |\n+--------------+--------------------+----------------------------------+\n3 rows in set (0.0006 sec)\n```\n\nDuring the installation of the components, the following lines will be added in the error log:\n\n```\n2024-12-12T17:47:19.949654Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'initializing…'\n2024-12-12T17:47:19.949691Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'new UDF 'profiler_cleanup()' has been registered successfully.'\n2024-12-12T17:47:19.949775Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'new variable 'profiler.dump_path' has been registered successfully.'\n2024-12-12T17:47:19.949832Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'new variable 'profiler.pprof_binary' has been registered successfully.'\n2024-12-12T17:47:19.952361Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'PFS table has been registered successfully.'\n2024-12-12T17:55:29.929491Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'initializing…'\n2024-12-12T17:55:29.929533Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'new UDF 'memprof_start()' has been registered successfully.'\n2024-12-12T17:55:29.929549Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'new UDF 'memprof_stop()' has been registered successfully.'\n2024-12-12T17:55:29.929560Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'new UDF 'memprof_dump()' has been registered successfully.'\n2024-12-12T17:55:29.929571Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'new UDF 'memprof_report()' has been registered successfully.'\n2024-12-12T17:55:29.929581Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'new UDF 'memprof_diff()' has been registered successfully.'\n2024-12-12T17:55:29.929625Z 8 [Note] [MY-011071] [Server] Component profiler_memory reported: 'Status variable(s) registered'\n2024-12-12T17:55:34.998066Z 8 [Note] [MY-011071] [Server] Component profiler_cpu reported: 'initializing…'\n2024-12-12T17:55:34.998130Z 8 [Note] [MY-011071] [Server] Component profiler_cpu reported: 'new UDF 'cpuprof_start()' has been registered successfully.'\n2024-12-12T17:55:34.998151Z 8 [Note] [MY-011071] [Server] Component profiler_cpu reported: 'new UDF 'cpuprof_stop()' has been registered successfully.'\n2024-12-12T17:55:34.998164Z 8 [Note] [MY-011071] [Server] Component profiler_cpu reported: 'new UDF 'cpuprof_report()' has been registered successfully.'\n2024-12-12T17:55:34.998228Z 8 [Note] [MY-011071] [Server] Component profiler_cpu reported: 'Status variable(s) registered'\n```\n\nAs we can see several UDFs were created:\n\n```\nMySQL \u003e SELECT UDF_NAME FROM performance_schema.user_defined_functions where udf_name like '%PROF_%';\n+------------------+\n| UDF_NAME         |\n+------------------+\n| CPUPROF_REPORT   |\n| CPUPROF_STOP     |\n| MEMPROF_DIFF     |\n| MEMPROF_START    |\n| PROFILER_CLEANUP |\n| MEMPROF_DUMP     |\n| MEMPROF_REPORT   |\n| MEMPROF_STOP     |\n| CPUPROF_START    |\n+------------------+\n9 rows in set (0.0018 sec)\n```\n\n### when using jemalloc\n\n```\nMySQL \u003e install component 'file://component_profiler';\nQuery OK, 0 rows affected (0.0017 sec)\n\nMySQL \u003e install component 'file://component_profiler_jemalloc_memory';\nQuery OK, 0 rows affected (0.0015 sec)\n\nMySQL \u003e select * from mysql.component;\n+--------------+--------------------+-------------------------------------------+\n| component_id | component_group_id | component_urn                             |\n+--------------+--------------------+-------------------------------------------+\n|            1 |                  1 | file://component_profiler                 |\n|            2 |                  2 | file://component_profiler_jemalloc_memory |\n+--------------+--------------------+-------------------------------------------+\n2 rows in set (0.0009 sec)\n```\n\nDuring the installation of the components, the following lines will be added in the error log:\n\n```\n2024-10-30T22:52:08.981065Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'initializing…'\n2024-10-30T22:52:08.981240Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'new variable 'profiler.dump_path' has been registered successfully.'\n2024-10-30T22:52:08.981338Z 8 [Note] [MY-011071] [Server] Component profiler reported: 'new variable 'profiler.pprof_binary' has been registered successfully.'\n2024-10-30T22:52:15.324135Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'initializing…'\n2024-10-30T22:52:15.324217Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'new UDF 'memprof_jemalloc_start()' has been registered successfully.'\n2024-10-30T22:52:15.324245Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'new UDF 'memprof_jemalloc_stop()' has been registered successfully.'\n2024-10-30T22:52:15.324265Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'new UDF 'memprof_jemalloc_dump()' has been registered successfully.'\n2024-10-30T22:52:15.324284Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'new UDF 'memprof_jemalloc_report()' has been registered successfully.'\n2024-10-30T22:52:15.324354Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'Status variable(s) registered'\n2024-10-30T22:52:15.324463Z 8 [Note] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'new variable 'profiler.jeprof_binary' has been registered successfully.\n```\n\nAs we can see several UDFs were created:\n\n```\nSELECT UDF_NAME FROM performance_schema.user_defined_functions where udf_name like '%PROF_%';\n+-------------------------+\n| UDF_NAME                |\n+-------------------------+\n| MEMPROF_JEMALLOC_REPORT |\n| MEMPROF_JEMALLOC_STOP   |\n| MEMPROF_JEMALLOC_START  |\n| MEMPROF_JEMALLOC_DUMP   |\n+-------------------------+\n4 rows in set (0.0007 sec)\n```\n\nTo use the components, the user needs to have the `SENSITIVE_VARIABLES_OBSERVER` privilege granted.\n\n## global variables\n\n```\nMySQL \u003e show global variables like 'profiler.%';\n+------------------------+--------------------+\n| Variable_name          | Value              |\n+------------------------+--------------------+\n| profiler.dump_path     | /tmp/mysql.memprof |\n| profiler.jeprof_binary | /usr/bin/jeprof    |\n| profiler.pprof_binary  | /usr/bin/pprof     |\n+------------------------+--------------------+\n3 rows in set (0.0045 sec)\n```\n\n### profiler.dump_path\n\nThis defines where the collected data should be dumped on the server.\n\n### profiler.jeprof_binary\n\nThis variable is installed by `component_profiler_jemalloc_memory` and defines where the `jeprof` binary is installed.\n\n### profiler.pprof_binary\n\nThe only way to parse the collected data is the use the `pprof` program. This variables defines where is installed the pprof binary executable file.\n\n## status variables\n\n### tcmalloc\n\n```\nMySQL \u003e show status like 'profiler.%';\n+------------------------+---------+\n| Variable_name          | Value   |\n+------------------------+---------+\n| profiler.cpu_status    | STOPPED |\n| profiler.memory_status | STOPPED |\n+------------------------+---------+\n2 rows in set (0.0107 sec)\n```\n### jemalloc\n\n```\nMySQL \u003e show global status like 'profiler%';\n+---------------------------------+---------+\n| Variable_name                   | Value   |\n+---------------------------------+---------+\n| profiler.jemalloc_memory_status | STOPPED |\n+---------------------------------+---------+\n1 row in set (0.0009 sec)\n```\n\nThese status variables provides the status of the profiling operations.\n\n## CPU profiling\n\n### start\n\nTo start the CPU profiling, we use the following command:\n\n```\nMySQL \u003e select cpuprof_start();\n+-----------------------+\n| cpuprof_start()       |\n+-----------------------+\n| cpu profiling started |\n+-----------------------+\n1 row in set (0.0022 sec)\n```\n\nNow the status variable changed:\n\n```\nMySQL \u003e show status like 'profiler.cpu_status';\n+---------------------+---------+\n| Variable_name       | Value   |\n+---------------------+---------+\n| profiler.cpu_status | RUNNING |\n+---------------------+---------+\n1 row in set (0.0018 sec)\n```\n\n### stop\n\nTo stop the collection, we use the following statement:\n\n```\nMySQL \u003e select cpuprof_stop();\n+-----------------------+\n| cpuprof_stop()        |\n+-----------------------+\n| cpu profiling stopped |\n+-----------------------+\n1 row in set (0.0033 sec)\n```\n\nThis generates a file `.prof` in the directory and file defined by `profiler.dump_path`:\n\n```\n$ ls -lh /tmp/dimk/\ntotal 68K\n-rw-rw---- 1 fred fred 68K Oct 14 21:08 mysql.prof\n```\n\n### report\n\nNow we can generate a report in two format: TEXT (the default) or DOT.\n\n#### text\n\nTo generate the report we use the following statement:\n\n```\nMySQL \u003e select cpuprof_report()\\G\n*************************** 1. row ***************************\ncpuprof_report(): Total: 95 samples\n      39  41.1%  41.1%       39  41.1% __futex_abstimed_wait_common\n       6   6.3%  47.4%        6   6.3% __GI___lll_lock_wake\n       4   4.2%  51.6%       45  47.4% ___pthread_cond_timedwait\n       4   4.2%  55.8%        5   5.3% __clock_gettime_2\n       3   3.2%  58.9%        3   3.2% __getrusage\n       3   3.2%  62.1%        6   6.3% pfs_end_mutex_wait_v1\n       3   3.2%  65.3%        3   3.2% syscall\n       2   2.1%  67.4%        3   3.2% OSMutex::enter (inline)\n       2   2.1%  69.5%        2   2.1% PFS_single_stat::aggregate_value (inline)\n       2   2.1%  71.6%        2   2.1% log_update_concurrency_margin\n       2   2.1%  73.7%       47  49.5% os_event::timed_wait\n       2   2.1%  75.8%        2   2.1% pfs_start_mutex_wait_v1\n       1   1.1%  76.8%        1   1.1% 0x00007ffb339e8af4\n       1   1.1%  77.9%        1   1.1% IB_thread::state (inline)\n       1   1.1%  78.9%        1   1.1% Log_files_capacity::next_file_earlier_margin (inline)\n       1   1.1%  80.0%       11  11.6% PolicyMutex::enter\n       1   1.1%  81.1%        3   3.2% PolicyMutex::pfs_begin_lock (inline)\n       1   1.1%  82.1%        1   1.1% ___pthread_mutex_lock\n       1   1.1%  83.2%        1   1.1% ___pthread_mutex_unlock\n       1   1.1%  84.2%        1   1.1% __condvar_dec_grefs\n       1   1.1%  85.3%        1   1.1% __memset_evex_unaligned_erms\n       1   1.1%  86.3%        1   1.1% __pthread_cleanup_pop\n       1   1.1%  87.4%        1   1.1% __strcpy_evex\n... \n```\n\n#### dot\n\nWe can generate the content of a dot file that can be used to generate an image:\n\n```\nMySQL \u003e select cpuprof_report('dot') into outfile 'cpu.dot';\nQuery OK, 1 row affected (5.6931 sec)\n```\n\n```\n$ dot -Tpng cpu.dot -o cpu.png\n```\n\n![CPU](examples/cpu.png)\n\n## Memory profiling - tcmalloc\n\n### start\n\nTo start the profiling, we need to use the following statement:\n\n```\nMySQL \u003e select memprof_start();\n+--------------------------+\n| memprof_start()          |\n+--------------------------+\n| memory profiling started |\n+--------------------------+\n1 row in set (0.0022 sec) \n```\n\nWe can confirm this from the status variable:\n\n```\nMySQL \u003e show status like 'profiler.memory_status';\n+------------------------+---------+\n| Variable_name          | Value   |\n+------------------------+---------+\n| profiler.memory_status | RUNNING |\n+------------------------+---------+\n1 row in set (0.0071 sec)\n```\n\nIt's also possible to provide a timeout in seconds to stop the profiling automatically:\n\n```\nMySQL \u003e select memprof_start(60);\n+-----------------------------------------+\n| memprof_start(60)                       |\n+-----------------------------------------+\n| memory profiling started for 60 seconds |\n+-----------------------------------------+\n1 row in set (0.0022 sec) \n```\n\n\n### dump\n\nDifferently than for the CPU profiling, we have the possibility to dump the collected data for the memory\nto disk manually:\n\n```\nMySQL \u003e select memprof_dump();\n+------------------------------+\n| memprof_dump()               |\n+------------------------------+\n| memory profiling data dumped |\n+------------------------------+\n1 row in set (0.0036 sec) \n```\n\nWe need to dump at the data we want to use for the reporting. Starting and stopping without dumping, won't produce\nany file to parse.\n\nWe can also provide information about the dump that will be printed to error log: \n\n```\nMySQL \u003e select memprof_dump('after a large select');\n+--------------------------------------+\n| memprof_dump('after a large select') |\n+--------------------------------------+\n| memory profiling data dumped         |\n+--------------------------------------+\n1 row in set (0.0014 sec)\n```\n\nWhen no string is provided, the default `user request` is used.\n\nWe can see in error log:\n\n```\nDumping heap profile to /tmp/dimk/mysql.0001.heap (user request)\nDumping heap profile to /tmp/dimk/mysql.0002.heap (after a large select)\n```\n\nAnd on the filesystem:\n\n```\n$ ls -lh /tmp/dimk/*heap\n-rw-rw---- 1 fred fred 301K Oct 14 21:32 /tmp/dimk/mysql.0001.heap\n-rw-rw---- 1 fred fred 302K Oct 14 21:33 /tmp/dimk/mysql.0002.heap\n```\n\n### stop\n\nBefore being able to generate a report, we need to stop the memory profiling:\n\n```\nMySQL \u003e select memprof_stop();\n+--------------------------+\n| memprof_stop()           |\n+--------------------------+\n| memory profiling stopped |\n+--------------------------+\n1 row in set (0.0035 sec)\n```\n\n### report\n\nNow we can generate a report for the memory in two format: TEXT (the default) or DOT.\n\n#### text\n\nTo generate the report we use the following statement:\n\n```\nMySQL \u003e select memprof_report()\\G\n*************************** 1. row ***************************\nmemprof_report(): Total: 1.0 MB\n     0.7  68.1%  68.1%      0.7  68.1% redirecting_allocator (inline)\n     0.3  27.6%  95.7%      0.3  27.6% ut::detail::malloc (inline)\n     0.0   1.5%  97.2%      0.0   1.5% __gnu_cxx::__aligned_membuf::_M_addr (inline)\n     0.0   1.4%  98.7%      0.0   1.4% std::__cxx11::basic_string::_M_local_data (inline)\n     0.0   1.0%  99.7%      0.0   1.5% std::pair::pair (inline)\n     0.0   0.1%  99.8%      0.0   0.4% Table_cache::add_used_table (inline)\n     0.0   0.1%  99.8%      0.0   0.1% std::__new_allocator::allocate (inline)\n     0.0   0.1%  99.9%      0.0   0.1% std::vector::reserve (inline)\n     0.0   0.0%  99.9%      0.0   0.0% std::__detail::_Hashtable_alloc::_M_allocate_buckets [clone .isra.0]\n     0.0   0.0% 100.0%      0.0   2.9% std::construct_at (inline)\n     0.0   0.0% 100.0%      0.0   0.0% void* my_internal_malloc [clone .lto_priv.0] (inline)\n     0.0   0.0% 100.0%      0.0   0.0% ngs::Socket_events::callback_timeout (inline)\n     0.0   0.0% 100.0%      0.0   0.0% std::__cxx11::basic_string::_M_capacity (inline)\n     0.0   0.0% 100.0%      0.0   2.3% Field_blob::clone\n     0.0   0.0% 100.0%      0.0   0.2% Field_blob::store_internal\n     0.0   0.0% 100.0%      0.0   1.4% Field_enum::clone\n     0.0   0.0% 100.0%      0.0   0.7% Field_long::clone\n     0.0   0.0% 100.0%      0.0   3.6% Field_longlong::clone\n...\n1 row in set (5.0192 sec)\n```\n\n#### dot\n\nWe can generate the ouput in dot format:\n\n```\nMySQL \u003e select memprof_report('dot') into outfile 'memory.dot';\nQuery OK, 1 row affected (4.7576 sec)\n```\n\nWe can use the file to generate an image:\n\n```\n$ dot -Tpng memory.dot -o memory.png\n```\n![Memory](examples/memory.png)\n\n\nIf you don't pass any dump file, all the dump files are used to generate the report. If you want to use a specific file, you can pass the filename as argument:\n\n```\nMySQL \u003e select memprof_report('TEXT', 'mysql.memprof.0001.heap')\\G\n*************************** 1. row ***************************\nmemprof_report('TEXT', 'mysql.memprof.0001.heap'): Total: 0.0 MB\n     0.0  58.4%  58.4%      0.0  58.4% void* my_internal_malloc [clone .lto_priv.0] (inline)\n     0.0  35.6%  94.1%      0.0  35.6% ut::detail::malloc (inline)\n     0.0   4.2%  98.3%      0.0  36.9% ut::detail::Alloc_pfs::alloc (inline)\n     0.0   0.9%  99.2%      0.0   0.9% dd::cache::Element_map::Element_map (inline)\n     0.0   0.4%  99.6%      0.0   0.4% addProfiler_element (inline)\n     0.0   0.2%  99.8%      0.0   0.2% ngs::Socket_events::callback_timeout (inline)\n     0.0   0.1% 100.0%      0.0   0.1% std::pair::pair (inline)\n     0.0   0.0% 100.0%      0.0   0.0% udf_impl::memprof_start_udf (inline)\n     0.0   0.0% 100.0%      0.0   1.0% Detached_thread::operator (inline)\n     0.0   0.0% 100.0%      0.0  11.5% Field_long::clone\n     0.0   0.0% 100.0%      0.0   0.4% Item::send\n     0.0   0.0% 100.0%      0.0   0.4% Item_func_udf_str::val_str\n     0.0   0.0% 100.0%      0.0   3.4% Log_files_dict::add\n     0.0   0.0% 100.0%      0.0   3.4% Log_files_dict::add (inline)\n     0.0   0.0% 100.0%      0.0  24.5% MEM_ROOT::Alloc (inline)\n     0.0   0.0% 100.0%      0.0  41.6% MEM_ROOT::AllocBlock\n     0.0   0.0% 100.0%      0.0  41.6% MEM_ROOT::AllocSlow\n     ....\n```\n\n## Memory profiling - jemalloc\n\n### start\n\nTo start the profiling, we need to use the following statement:\n\n```\nMySQL \u003e select memprof_jemalloc_start();\n+--------------------------+\n| memprof_jemalloc_start() |\n+--------------------------+\n| memory profiling started |\n+--------------------------+\n1 row in set (0.0002 sec)\n```\n\nWe can confirm this from the status variable:\n\n```\nMySQL \u003e show status like 'profiler.jemalloc_memory_status';\n+---------------------------------+---------+\n| Variable_name                   | Value   |\n+---------------------------------+---------+\n| profiler.jemalloc_memory_status | RUNNING |\n+---------------------------------+---------+\n1 row in set (0.0071 sec)\n```\n\n### dump\n\nAs for tcmalloc, we have to dump the collected data for the memory\nto disk manually:\n\n```\nMySQL \u003e select memprof_jemalloc_dump();\n+------------------------------+\n| memprof_jemalloc_dump()      |\n+------------------------------+\n| memory profiling data dumped |\n+------------------------------+\n1 row in set (0.0027 sec)\n```\n\nWe need to dump at the data we want to use for the reporting. Starting and stopping without dumping, won't produce\nany file to parse.\n\nWe can see in error log:\n\n```\nUsing local file /home/fred/workspace/mysql-server/BIN-DEBUG/runtime_output_directory/mysqld.\nUsing local file /tmp/mysql.memprof.0001.heap.\n```\n\nAnd on the filesystem:\n\n```\n$ ls -lh /tmp/*heap\n-rw-r----- 1 fred fred 92K Oct 30 23:52 /tmp/mysql.memprof.0001.heap\n```\n\n### stop\n\nBefore being able to generate a report, we need to stop the memory profiling:\n\n```\nMySQL \u003e select memprof_jemalloc_stop();\n+--------------------------+\n| memprof_jemalloc_stop()  |\n+--------------------------+\n| memory profiling stopped |\n+--------------------------+\n1 row in set (0.0005 sec) \n```\n\n### report\n\nNow we can generate a report for the memory in two format: TEXT (the default) or DOT.\n\n#### text\n\nTo generate the report we use the following statement:\n\n```\nMySQL \u003e select memprof_jemalloc_report()\\G\n*************************** 1. row ***************************\nmemprof_jemalloc_report(): Total: 368.6 MB\n   139.6  37.9%  37.9%    139.6  37.9% pfs_malloc\n   125.1  33.9%  71.8%    125.1  33.9% ut::detail::malloc (inline)\n    75.7  20.5%  92.4%     75.7  20.5% ut::detail::calloc (inline)\n    22.5   6.1%  98.5%     22.5   6.1% void* my_internal_malloc [clone .lto_priv.0] (inline)\n     3.1   0.8%  99.3%      3.1   0.8% operator delete[]\n     1.5   0.4%  99.7%      1.5   0.4% my_once_alloc\n     0.5   0.1%  99.9%      0.5   0.1% std::__to_chars_bfloat16_t@@GLIBCXX_3.4.31\n     0.5   0.1% 100.0%      0.5   0.1% my_openssl_malloc (inline)\n     0.0   0.0% 100.0%      0.5   0.1% ::DDSE_dict_recover\n     0.0   0.0% 100.0%      1.5   0.4% ::store_plugin_and_referencing_views_metadata\n     0.0   0.0% 100.0%      0.5   0.1% AIO::AIO\n     0.0   0.0% 100.0%      0.5   0.1% AIO::create\n     0.0   0.0% 100.0%      0.5   0.1% AIO::start\n     0.0   0.0% 100.0%      0.8   0.2% Acl_cache_lock_guard::lock [clone .part.0]\n     0.0   0.0% 100.0%     20.0   5.4% Buf_flush_list_added_lsns::Buf_flush_list_added_lsns (inline)\n     0.0   0.0% 100.0%     20.0   5.4% Buf_flush_list_added_lsns::create\n     0.0   0.0% 100.0%      0.5   0.1% CRYPTO_malloc@@OPENSSL_3.0.0\n     0.0   0.0% 100.0%      0.5   0.1% CRYPTO_zalloc@@OPENSSL_3.0.0\n     0.0   0.0% 100.0%      1.5   0.4% Clone_persist_gtid::flush_gtids\n     0.0   0.0% 100.0%      1.5   0.4% Clone_persist_gtid::periodic_write\n     0.0   0.0% 100.0%      0.5   0.1% Cost_constant_cache::reload\n     0.0   0.0% 100.0%      0.5   0.1% Datafile::read_first_page\n     0.0   0.0% 100.0%      1.5   0.4% Detached_thread::operator (inline)\n     0.0   0.0% 100.0%      7.6   2.0% Double_write::Double_write\n     0.0   0.0% 100.0%      7.6   2.0% Double_write::create_v2\n     0.0   0.0% 100.0%     58.1  15.8% Double_write::load\n     0.0   0.0% 100.0%      0.5   0.1% EVP_CIPHER_fetch@@OPENSSL_3.0.0\n     0.0   0.0% 100.0%      1.0   0.3% Events::init \n...\n1 row in set (5.0192 sec)\n```\n\n#### dot\n\nWe can generate the ouput in dot format:\n\n```\nMySQL \u003e select memprof_jemalloc_report('dot') into outfile 'jemalloc.dot';\nQuery OK, 1 row affected (4.7576 sec)\n```\n\nWe can use the file to generate an image:\n\n```\n$ dot -Tpng jemalloc.dot -o jemalloc.png\n```\n![Memory](examples/jemalloc.png)\n\n## performance_schema table - profiler_actions\n\nAll actions are recorded in a `performance_schema` table called `profiler_actions`:\n\n### tcmalloc\n\n```\nMySQL \u003e select * from performance_schema.profiler_actions;\n+---------------------+-----------+--------+---------+------------------------------+-------------------+\n| LOGGED              | ALLOCATOR | TYPE   | ACTION  | FILENAME                     | EXTRA             |\n+---------------------+-----------+--------+---------+------------------------------+-------------------+\n| 2024-11-03 15:51:54 | tcmalloc  | memory | started |                              |                   |\n| 2024-11-03 15:52:06 | tcmalloc  | memory | dumped  | /tmp/mysql.memprof.0001.heap | user request      |\n| 2024-11-03 15:52:13 | tcmalloc  | memory | dumped  | /tmp/mysql.memprof.0002.heap | after large query |\n| 2024-11-03 15:52:20 | tcmalloc  | memory | stopped |                              |                   |\n| 2024-11-03 15:52:35 | profiler  | cpu    | started | /tmp/mysql.memprof.prof      |                   |\n| 2024-11-03 15:52:42 | profiler  | cpu    | stopped | /tmp/mysql.memprof.prof      |                   |\n| 2024-11-03 15:53:47 | profiler  | cpu    | report  |                              | text              |\n| 2024-11-03 15:53:59 | tcmalloc  | memory | report  |                              | text              |\n| 2024-11-03 15:54:38 | tcmalloc  | memory | report  |                              | dot               |\n+---------------------+-----------+--------+---------+------------------------------+-------------------+\n9 rows in set (0.0008 sec)\n```\n\n### jemalloc\n\n```\nMySQL \u003e select * from performance_schema.profiler_actions;\n+---------------------+-----------+--------+---------+------------------------------+-------+\n| LOGGED              | ALLOCATOR | TYPE   | ACTION  | FILENAME                     | EXTRA |\n+---------------------+-----------+--------+---------+------------------------------+-------+\n| 2024-11-03 16:02:11 | jemalloc  | memory | started |                              |       |\n| 2024-11-03 16:02:23 | jemalloc  | memory | dumped  | /tmp/mysql.memprof.0001.heap |       |\n| 2024-11-03 16:02:29 | jemalloc  | memory | stopped |                              |       |\n| 2024-11-03 16:02:40 | jemalloc  | memory | report  |                              | text  |\n| 2024-11-03 16:02:48 | jemalloc  | memory | report  |                              | dot   |\n+---------------------+-----------+--------+---------+------------------------------+-------+\n5 rows in set (0.0030 sec)\n```\n\n## cleanup collected dump files\n\nIt's possible to also cleanup the collected dump files. This could be dangerous as\nit could be used to try deleting other important files.\n\nThe UDF `profiler_cleanup` cleans up all the files in the directory and matching the filenmae specified in the `profiler.dump_path` variable:\n\n```\nMySQL \u003e show global variables like 'profiler.dump_path';\n+--------------------+--------------------+\n| Variable_name      | Value              |\n+--------------------+--------------------+\n| profiler.dump_path | /tmp/mysql.memprof |\n+--------------------+--------------------+\n1 row in set (0.0007 sec)\n\nMySQL \u003e select profiler_cleanup();\n+------------------------------------------------------------------------+\n| profiler_cleanup()                                                     |\n+------------------------------------------------------------------------+\n| Profiling data matching /tmp/mysql.memprof prefix has been cleaned up. |\n+------------------------------------------------------------------------+\n1 row in set (0.0008 sec)\n```\n\n## errors, warnings, messages\n\n### dependency\n\nWhen installing `component_profiler_cpu`, `component_profiler_memory` or `component_profiler_jemalloc_memory`, if `component_profiler` is not installed, the following error is displayed: \n\n```\nERROR: 3534 (HY000): Cannot satisfy dependency for service 'profiler_var'\nrequired by component 'mysql:profiler_cpu_service'.\n```\n\n### privilege\n\nThe privilege is checked to modify variables and call the UDFs:\n\n```\nMySQL \u003e select memprof_report('dot') into outfile 'memory2.dot' ;\nERROR: 1227 (42000): Access denied; you need (at least one of) the \n       SENSITIVE_VARIABLES_OBSERVER privilege(s) for this operation\n\nMySQL \u003e set global profiler.dump_path='/tmp/dimk/mysql2';\nERROR: 1227 (42000): Access denied; you need (at least one of) the\n        SENSITIVE_VARIABLES_OBSERVER privilege(s) for this operation\n```\n\n### folders and privileges\n\nIf the folder doesn't exist or mysql user has no rights, an error will be returned:\n\n```\nMySQL \u003e set global profiler.dump_path='/tmp/lefred/mysql';\nERROR: 3200 (HY000): memprof UDF failed; we don't have access to write in that folder.\n```\n\n### previous existing data\n\nIf there is already data with the same name as defined in `profiler.dump_path`, not profiling can be started:\n\n```\nMySQL \u003e select memprof_start();\nERROR: 3200 (HY000): profiler UDF failed; There is already a heap dump, \n       change the 'profiler.dump_path' value first.\n```\n\nThe previous data must be deleted or another name must be provided.\n\n### report without stopping\n\nAny report must be generated only if the profiling is stopped:\n\n```\nMySQL \u003e select memprof_report()\\G\nERROR: 3200 (HY000): profiler UDF failed; memory profiler is still running, you need to stop it first.\n```\n\n### tcmalloc\n\nMySQL must be started using a gperftools-libs library (`LD_PRELOAD=/usr/lib64/libtcmalloc_and_profiler.so`):\n\n```\nMySQL \u003e install component 'file://component_profiler';\nERROR: 1126 (HY000): Can't open shared library '/home/fred/workspace/mysql-server/BIN-DEBUG/lib/plugin/component_profiler.so'\n(errno: 0 /home/fred/workspace/mysql-server/BIN-DEBUG/lib/plugin/component_profiler.so: undefined symbol: ProfilerStart)\n```\n\nFor memory profiling, you need to use one of these libraries:\n- `libtcmalloc.so`\n- `libtcmalloc_and_profiler.so`\n\nFor CPU profiling, you need to use one of those:\n- `libprofiler.so`\n- `libtcmalloc_and_profiler.so`\n\n### jemalloc\n\nThe `jemalloc` library used needs to support profiling. It needs to have been compiled with `--enable-prof`, you can check by using the following command:\n\n```\n$ jemalloc-config --config | grep prof\n```\n\nIf you get `--enable-prof` then it means it's OK. If not, you need to compile `jmalloc` or ask me for an rpm ;)\n\nMySQL needs also to be started using `LD_PRELOAD=/usr/lib64/libjemalloc.so` and the profiling must be enabled using `MALLOC_CONF=\"prof:true\"`,\nif not it won't work and you will get the following error:\n\n```\nERROR: 1126 (HY000): Can't open shared library '/home/fred/workspace/mysql-server/BIN-DEBUG/lib/plugin/component_profiler_jemalloc_memory.so'`\n(errno: 0 /home/fred/workspace/mysql-server/BIN-DEBUG/lib/plugin/component_profiler_jemalloc_memory.so: undefined symbol: mallctl)\n```\n\n### my.cnf or SET PERSIST problem \n\nIt seems that the system variables defined in `my.cnf` or saved using `SET PERSIST` are not used when\nrestarting MySQL.\n\nThis is because the user `bootstrap@localhost` doesn't have the `SENSITIVE_VARIABLES_OBSERVER` privilege:\n\n```\n2024-11-04T21:43:50.176044Z 6 [ERROR] [MY-011071] [Server] Component profiler_jemalloc_memory reported: 'user (bootstrap@localhost) has no access to set profiler.jeprof_path variable (privilege SENSITIVE_VARIABLES_OBSERVER required).'\n2024-11-04T21:43:50.176070Z 6 [ERROR] [MY-011268] [Server] Configuring persisted options failed: \"Access denied; you need (at least one of) the SENSITIVE_VARIABLES_OBSERVER privilege(s) for this operation\".\n```\n\nHowever, that user doesn't exist on the system?!? (I need to investigate with the development team).\n\nTo fix it, just create the user and assign the privilege:\n\n```\nMySQL \u003e CREATE USER bootstrap@localhost IDENTIFIED by RANDOM PASSWORD;\nMySQL \u003e CREATE USER bootstrap@127.0.0.1 IDENTIFIED by RANDOM PASSWORD;\n\nMySQL \u003e GRANT SENSITIVE_VARIABLES_OBSERVER ON *.* TO 'bootstrap'@'127.0.0.1';\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flefred%2Fmysql-component-profiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flefred%2Fmysql-component-profiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flefred%2Fmysql-component-profiler/lists"}