{"id":19865267,"url":"https://github.com/mnms/share","last_synced_at":"2025-09-22T03:46:18.473Z","repository":{"id":35574194,"uuid":"197524228","full_name":"mnms/share","owner":"mnms","description":"documents and tools about LightningDB","archived":false,"fork":false,"pushed_at":"2022-03-29T21:58:19.000Z","size":31555,"stargazers_count":0,"open_issues_count":4,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-01T00:32:56.058Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mnms.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-18T06:19:41.000Z","updated_at":"2019-11-19T08:52:40.000Z","dependencies_parsed_at":"2022-08-08T09:16:46.849Z","dependency_job_id":null,"html_url":"https://github.com/mnms/share","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mnms/share","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnms%2Fshare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnms%2Fshare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnms%2Fshare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnms%2Fshare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mnms","download_url":"https://codeload.github.com/mnms/share/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnms%2Fshare/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276342525,"owners_count":25625581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-22T02:00:08.972Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T15:21:56.206Z","updated_at":"2025-09-22T03:46:18.432Z","avatar_url":"https://github.com/mnms.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Installation Guide\n==============================================================================\n\nInstalling Prerequisites\n-------------------\n\n### For Mac osx\n- java 1.8  \n- xcode gcc 사용 (xcode 9.0 이상)\n-  xcode-select --install\n\n``` console\nbrew install maven automake libtool pkg-config sshd boost\ngem install redis\nmvn compiler:compile\n```\n\n\n### For Linux\n- Oracla JDK 1.8\n- bash\n- gcc 4.8.5 이상 \n- glibc 2.17 이상 (CentOS 7.2 이후 버전은 기본적으로 2.17 이상이 설치되어 있습니다.)\n- unzip\n- ssh \n\n``` console\nyum -y install vim ruby unzip\nsudo yum udpate\nsudo yum install epel-release\nsudo yum install boost boost-thread boost-devel \n```\n``` console\nssh-keygen -t rsa\nchmod 0600 ~/.ssh/authorized_keys\ncat .ssh/id_rsa.pub | ssh localhost \"cat \u003e\u003e .ssh/authorized_keys\"\n```\n### Ruby and redis ruby module install\n- Ruby 1.x 이상 \n- Ruby module : redis (3.3.0 이상) \n- rubygem : 모듈 설치를 위함 \n\n#### Update for Centos 7 (installing gem redis module version 4.1.2 compatible with flashbase)\n``` console\nsudo yum install centos-release-scl\nsudo yum install rh-ruby23\nscl enable rh-ruby23 bash\ngem install redis\n```\n\n#### Previous method\n``` console\n$gem install redis\n\n'gem install redis' 가 제대로 수행이 되지 않거나,\n설치 후 'redis restart --reset --cluster --yes' 수행 시 아래와 같은 오류가 발생하면 제대로 환경설정이 되지 않은 것이므로,\n\n/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError)\nfrom /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'\nfrom /home/hynix/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/bin/redis-trib.rb:25:in `\u003cmain\u003e'\n\n아래 순서로 확인한다.\n\n$gem env\ngem 의 환경설정으로 붉은색 부분이 설정되어 있는지 확인한다.\n\nRubyGems Environment:\n  - RUBYGEMS VERSION: 2.0.14.1\n  - RUBY VERSION: 2.0.0 (2015-12-16 patchlevel 648) [x86_64-linux]\n  - INSTALLATION DIRECTORY: /home/hynix/.gem\n  - RUBY EXECUTABLE: /usr/bin/ruby\n  - EXECUTABLE DIRECTORY: /home/hynix/.gem/bin\n  - RUBYGEMS PLATFORMS:\n    - ruby\n    - x86_64-linux\n  - GEM PATHS:\n     - /home/hynix/.gem\n     - /home/hynix/.gem/ruby/gems\n     - /usr/share/gems\n     - /usr/local/share/gems\n  - GEM CONFIGURATION:\n     - :update_sources =\u003e true\n     - :verbose =\u003e true\n     - :backtrace =\u003e false\n     - :bulk_threshold =\u003e 1000\n  - REMOTE SOURCES:\n     - https://rubygems.org/\n\n\n$gem list\n\ngem 에 설치된 패키지 리스트로 'redis xxx '가 없으면 제대로 설치가 안된 것이다.\n\n*** LOCAL GEMS ***\n\nabrt (0.3.0)\nbigdecimal (1.2.0)\nbundler (1.7.8)\nio-console (0.4.2)\njson (1.7.7)\nnet-http-persistent (2.8)\npsych (2.0.0)\nrdoc (4.0.0)\nredis (3.3.1, 3.3.0, 3.0.7)\nredis-stat (0.4.13)\nthor (0.19.1)\n\n만약 없다면 다시 시도하거나, \n아래 두 경로에 아래 파일들을 복사해서 넣는다.(기존 운영하던 서버 또는 다른 PATH(version)로 설치된 위치에서...)\n\n\u003c\u003c\u003c\u003c *.gemspec 파일 복사 후 있는지 확인 \u003e\u003e\u003e\u003e\n[hynix@d216 specifications]$ pwd\n/usr/share/gems/specifications\n[hynix@d216 specifications]$ ls\nabrt-0.3.0.gemspec        io-console-0.4.2.gemspec         rdoc-4.0.0.gemspec   redis-stat-0.4.13.gemspec\nbigdecimal-1.2.0.gemspec  json-1.7.7.gemspec               redis-3.0.7.gemspec  thor-0.19.1.gemspec\nbundler-1.7.8.gemspec     net-http-persistent-2.8.gemspec  redis-3.3.0.gemspec\ndefault                   psych-2.0.0.gemspec              redis-3.3.1.gemspec\n\n\u003c\u003c\u003c\u003c .gem dir에 redis 관련 dir가 존재하는지 확인 \u003e\u003e\u003e\u003e\n/usr/share/gems/gems\n[hynix@d216 gems]$ ls\nansi256-0.2.5                   insensitive_hash-0.3.3    rack-1.6.4             sinatra-1.3.6\ndaemons-1.1.9                   json-1.8.3                rack-protection-1.5.3  tabularize-0.2.9\nelasticsearch-1.0.18            lps-0.2.1                 redis-3.0.7            thin-1.5.1\nelasticsearch-api-1.0.18        multi_json-1.12.1         redis-3.3.0            tilt-1.4.1\nelasticsearch-transport-1.0.18  multipart-post-2.0.0      redis-3.3.1            nicode-display_width-0.1.1\neventmachine-1.2.0.1            option_initializer-1.5.1  redis-stat-0.4.13\nfaraday-0.9.2                   parallelize-0.4.1         si-0.1.4\n```\n\n- 'envsubst: command not found'\n``` console\nbrew install gettext\nbrew install wget\n\ncd ~/Downloads\nwget http://ftp.gnu.org/gnu/gettext/gettext-0.18.2.tar.gz\ntar -zxvf gettext-0.18.2.tar.gz\ncd gettext-0.18.2\n./configure\nmake\n\nmake check\nsudo make install\n\nbrew link --force gettext\n\n```\n### Intel MKL library\n\n#### (1) Intel MKL 2019 library install\n\n- go to the website: https://software.intel.com/en-us/mkl/choose-download/macos\n- register and login\n- select product named \"Intel * Math Kernel Library for Linux\" or \"Intel * Math Kernel Library for Mac\" from the select box \"Choose Product to Download\"\n- Choose a Version \"2019 Update 2\" and download\n- unzip the file and execute the install.sh file with root account or (sudo command)\n``` console\n    sudo ./install.sh\n```\n- choose custom install and configure the install directory /opt/intel (with sudo, /opt/intel is the default installation path, just confirm it)\n``` console\nmatthew@fbg05 /opt/intel $ pwd\n/opt/intel\nmatthew@fbg05 /opt/intel $ ls -alh\n합계 0\ndrwxr-xr-x  10 root root 307  3월 22 01:34 .\ndrwxr-xr-x.  5 root root  83  3월 22 01:34 ..\ndrwxr-xr-x   6 root root  72  3월 22 01:35 .pset\ndrwxr-xr-x   2 root root  53  3월 22 01:34 bin\nlrwxrwxrwx   1 root root  28  3월 22 01:34 compilers_and_libraries -\u003e compilers_and_libraries_2019\ndrwxr-xr-x   3 root root  19  3월 22 01:34 compilers_and_libraries_2019\ndrwxr-xr-x   4 root root  36  1월 24 23:04 compilers_and_libraries_2019.2.187\ndrwxr-xr-x   6 root root  63  1월 24 22:50 conda_channel\ndrwxr-xr-x   4 root root  26  1월 24 23:01 documentation_2019\nlrwxrwxrwx   1 root root  33  3월 22 01:34 lib -\u003e compilers_and_libraries/linux/lib\nlrwxrwxrwx   1 root root  33  3월 22 01:34 mkl -\u003e compilers_and_libraries/linux/mkl\nlrwxrwxrwx   1 root root  29  3월 22 01:34 parallel_studio_xe_2019 -\u003e parallel_studio_xe_2019.2.057\ndrwxr-xr-x   5 root root 216  3월 22 01:34 parallel_studio_xe_2019.2.057\ndrwxr-xr-x   3 root root  16  3월 22 01:34 samples_2019\nlrwxrwxrwx   1 root root  33  3월 22 01:34 tbb -\u003e compilers_and_libraries/linux/tbb\n```\n\n\n#### (2) Intel MKL 2019 library environment settings\n\n- append the following statement into ~/.bashrc\n```console\n# INTEL MKL enviroment variables for ($MKLROOT, can be checked with the value export | grep MKL)\nsource /opt/intel/mkl/bin/mklvars.sh intel64\n```\n\n- apache hadoop 2.6 이상(정상 작동을 확인한 버전은 2.6.0 입니다.)\n- apache spark 2.1.1 on hadoop 2.6 \n\n\n선택 설치 항목\n-----------------------\n- ntp : 서버 간의 시간 동기화를 위함 [CentOS only]\n- Gateway에서 mysql을 사용할 경우 mysql 패키지 필요(mariadb, mariadb-server) [CentOS only]\n\n\nSystem Parameters \n-------------------\n/etc/sysctl.conf [CentOS only]\n\u003cpre\u003e\u003ccode\u003evm.swappiness = 0 \nvm.overcommit_memory = 1 \nvm.overcommit_ratio = 50 \nfs.file-max = 6815744 \nnet.ipv4.ip_local_port_range = 32768 65535 \nnet.core.rmem_default = 262144 \nnet.core.wmem_default = 262144 \nnet.core.rmem_max = 16777216 \nnet.core.wmem_max = 16777216\nnet.ipv4.tcp_max_syn_backlog = 4096 \nnet.core.somaxconn = 65535\n\u003c/code\u003e\u003c/pre\u003e\n\u003e 주의) runtime  반영하려면 sudo sysctl -p 수행\n\n/etc/security/limits.conf [CentOS only]\n```bash\n* soft nofile 262144\n* hard nofile 262144\n* soft nproc 131072 \n* hard nproc 131072\n[account name] * soft nofile 262144\n[account name] * hard nofile 262144\n[account name] * soft nproc 131072 \n[account name] * hard nproc 131072\n```\n\u003e 주의) runtime 반영하려면, ulimit -n 65535, ulimit -u 131072 수행\n\nRemove SWAP Partition [CentOS only]\n\u003e FlashBase를 구동할 서버는 /etc/fstab 파일에서 SWAP partition 부분을 comment 처리 후 재시작 합니다.\n\u003cpre\u003e\u003ccode\u003e# /etc/fstab\n# Created by anaconda on Fri Sep 15 03:59:39 2017\n# Accessible filesystems, by reference, are maintained under '/dev/disk'\n# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info\n/dev/sda1 /home xfs defaults 0 0 \n/dev/sda2 / xfs defaults 0 0 \n/dev/sda3 /var xfs defaults 0 0 \n/dev/sda5 /tmp xfs defaults 0 0 \n/dev/sdb1 /data01 xfs defaults 0 0 \n/dev/sdc1 /data02 xfs defaults 0 0 \n/dev/sdd1 /data03 xfs defaults 0 0 \n/dev/sde1 /data04 xfs defaults 0 0\n/dev/mapper/centos-swap swap swap defaults 0 0\n\u003c/code\u003e\u003c/pre\u003e\n\u003e 재시작이 불가능할 경우 아래 명령으로 swap partition의 mount를 해제 합니다. (swap을 사용중일 경우 시간이 오래 소요됩니다.)\n\u003cpre\u003e\u003ccode\u003eswapoff -a\u003c/code\u003e\u003c/pre\u003e\n\nSession configuration files\n\n.bashrc\n```bash\n# .bashrc\n\nif [ -f /etc/bashrc ]; then\n. /etc/bashrc\nfi\n\n# User specific environment and startup programs\n\nPATH=$PATH:$HOME/.local/bin:$HOME/bin\n\nHADOOP_HOME=/home/nvkvs/hadoop\nHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop\nYARN_CONF_DIR=$HADOOP_HOME/etc/hadoop\nSPARK_HOME=/home/nvkvs/spark\n\nPATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$HOME/sbin\n\nexport PATH SPARK_HOME HADOOP_HOME HADOOP_CONF_DIR YARN_CONF_DIR\nalias cfc='source ~/.use_cluster'\n```\n\n.use_cluster\n```bash\n#!/bin/bash\n\n## set cluster-#{NUM} path\nexport PATH=\"/bin/:/sbin/:/usr/local/bin/:/usr/local/sbin\"\nexport SR2_HOME=${HOME}/tsr2/cluster_$1/tsr2-assembly-1.0.0-SNAPSHOT\n\nsource ${HOME}/.bash_profile\n\necho $PATH | grep ${SR2_HOME} \u003e /dev/null\nRET=$?\nif [[ $RET -eq 1 ]]; then\n    PATH=$PATH:$SR2_HOME/bin:$SR2_HOME/sbin\nfi\n\n## source command auto-complate\nsource $SR2_HOME/sbin/tsr2-helper\n\nif [ \"$#\" -le \"1\" ]; then\n    return 0\nelse\n\tshift\n    \"$@\"\n\treturn $?\nfi\n```\n\nOther system configuration\n===========================\n```bash\nroot@fbg01 ~]# cat /etc/init.d/disable-transparent-hugepages\n#!/bin/bash\n### BEGIN INIT INFO\n# Provides:          disable-transparent-hugepages\n# Required-Start:    $local_fs\n# Required-Stop:\n# X-Start-Before:    mongod mongodb-mms-automation-agent\n# Default-Start:     2 3 4 5\n# Default-Stop:      0 1 6\n# Short-Description: Disable Linux transparent huge pages\n# Description:       Disable Linux transparent huge pages, to improve\n#                    database performance.\n### END INIT INFO\n \ncase $1 in\n  start)\n    if [ -d /sys/kernel/mm/transparent_hugepage ]; then\n      thp_path=/sys/kernel/mm/transparent_hugepage\n    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then\n      thp_path=/sys/kernel/mm/redhat_transparent_hugepage\n    else\n      return 0\n    fi\n \n    echo 'never' \u003e ${thp_path}/enabled\n    echo 'never' \u003e ${thp_path}/defrag\n \n    re='^[0-1]+$'\n    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]\n    then\n      # RHEL 7\n      echo 0  \u003e ${thp_path}/khugepaged/defrag\n    else\n      # RHEL 6\n      echo 'no' \u003e ${thp_path}/khugepaged/defrag\n    fi\n \n    unset re\n    unset thp_path\n    ;;\nesac\n[root@fbg01 ~]# chmod 755 /etc/init.d/disable-transparent-hugepages\n[root@fbg01 ~]# chkconfig --add disable-transparent-hugepages\n```\n\n\n\nFlashBase Install\n=================\n\ndeploy directory를 생성한다.\n\u003cpre\u003e\u003ccode\u003emkdir ~/deploy\n\u003c/code\u003e\u003c/pre\u003e\n\nFlashBase 설치 파일(e.g. flashbase.xxx.bin)을 아래 경로에 복사한다. ('bin/'에 있음))\n\u003cpre\u003e\u003ccode\u003ecp flashbase.xxx.bin ~/deploy\u003c/code\u003e\u003c/pre\u003e\n\ndeploy-flashbase.sh를 '~/deploy'에 복사한다.('scripts'에 있음)\n\u003cpre\u003e\u003ccode\u003ecp deploy-flashbase.sh ~/deploy/ \u003c/code\u003e\u003c/pre\u003e\n\ndeploy-flashbase.sh를 열어서 아래와 같이 수정한다.\n\n1. 'line 3'에서 cluster를 구성할 노드(서버)를 지정한다. 여러 대의 서버에 설치할 경우에는 'line 1'과 같이 설정한다.\n2. 'line 17'에 cluster 번호를 입력한다. 'cluster_1'의 경우 '1'로 입력하면 된다. '1 2 3' 과 같이 여러 cluster에 동시에 설치하는 것도 가능하다.\n3. 설치는 './deploy-flashbase.sh [file name]'와 같이 입력하면 된다.\n\u003cpre\u003e\u003ccode\u003e./deploy-flashbase.sh ./flashbase.xxx.bin \u003c/code\u003e\u003c/pre\u003e\n\n``` console\n  1 #nodes=(\"nv-accel-d01\" \"nv-accel-d02\" \"nv-accel-d03\")\n  2 #nodes=(\"nv-accel-w01\" \"nv-accel-w02\" \"nv-accel-w03\" \"nv-accel-w04\" \"nv-accel-w05\" \"nv-accel-w06\")\n  3 nodes=( \"localhost\")\n  4\n  5 INSTALLER_PATH=$1\n  6\n  7 [[ $INSTALLER_PATH == \"\" ]] \u0026\u0026 echo \"NO ARGS\" \u0026\u0026 echo \"cmd \u003cpath of installer.bin\u003e\" \u0026\u0026 exit 1\n  8 [[ ! -e $INSTALLER_PATH ]] \u0026\u0026 echo \"NO FILE: $INSTALLER_PATH\" \u0026\u0026 exit 1\n  9\n 10 INSTALLER_BIN=$(basename $INSTALLER_PATH)\n 11 DATEMIN=`date +%Y%m%d%H%M%S`\n 12 TSR2_DIR=~/tsr2\n 13 echo \"DATEMIN: $DATEMIN\"\n 14 echo \"INSTALLER PATH: $INSTALLER_PATH\"\n 15 echo \"INSTALLER NAME: $INSTALLER_BIN\"\n 16\n 17 for cluster_num in \"1\";\n 18 do\n 19     CLUSTER_DIR=$TSR2_DIR/cluster_${cluster_num}\n 20     BACKUP_DIR=\"${CLUSTER_DIR}_bak_$DATEMIN\"\n 21     CONF_BACKUP_DIR=\"${CLUSTER_DIR}_conf_bak_$DATEMIN\"\n 22     SR2_HOME=${CLUSTER_DIR}/tsr2-assembly-1.0.0-SNAPSHOT\n 23     SR2_CONF=${SR2_HOME}/conf\n 24\n 25     echo \"======================================================\"\n 26     echo \"DEPLOY CLUSTER $cluster_num\"\n 27     echo \"\"\n 28     echo \"CLUSTER_DIR: $CLUSTER_DIR\"\n 29     echo \"SR2_HOME: $SR2_HOME\"\n 30     echo \"SR2_CONF: $SR2_CONF\"\n 31     echo \"BACKUP_DIR: $BACKUP_DIR\"\n 32     echo \"CONF_BACKUP_DIR: $CONF_BACKUP_DIR\"\n 33     echo \"======================================================\"\n 34     echo \"backup...\"\n 35     mkdir -p ${CONF_BACKUP_DIR}\n 36     cp -rf ${SR2_CONF}/* $CONF_BACKUP_DIR\n 37\n 38     echo \"\"\n 39\n 40     for node in ${nodes[@]};\n 41     do\n 42         echo \"DEPLOY NODE $node\"\n 43        # ssh $node \"mv ${CLUSTER_DIR} ${BACKUP_DIR}\"\n 44         ssh $node \"mkdir -p ${CLUSTER_DIR}\"\n 45         scp -r $INSTALLER_PATH $node:${CLUSTER_DIR}\n 46         ssh $node \"PATH=${PATH}:/usr/sbin; ${CLUSTER_DIR}/${INSTALLER_BIN} --full ${CLUSTER_DIR}\"\n 47         rsync -avr $CONF_BACKUP_DIR/* $node:${SR2_CONF}\n 48     done\n 49\n 50     echo \"\"\n 51 done\n\n```\n\n\n'.use_cluster' 파일을 이용하여 cluster 경로를 설정한다.('scripts'에 있음)\n\u003cpre\u003e\u003ccode\u003ecp .use_cluster ~/ \u003c/code\u003e\u003c/pre\u003e\n\n아래와 같이 입력하면 cluster_1 로 경로가 설정된다.\n\u003cpre\u003e\u003ccode\u003esource ~/.use_cluster 1 \u003c/code\u003e\u003c/pre\u003e\n\n아래와 같이 경로가 설정되어 있는지 확인한다.\n\u003cpre\u003e\u003ccode\u003e$ which flashbase\n/Users/admin/tsr2cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/sbin/flashbase\n\n$ flashbase version\nFlashBase(TM) version '1.1.9'\nLast updated on 2018.08.06\n\u003c/code\u003e\u003c/pre\u003e\n\n아래와 같이 process 수와 replication, disk 정보를 입력한다.\n\n'line 4~5'에서 master node의 서버와 프로세스 수를 지정한다.(PORT 수가 프로세스 수)\n\n'line 8~9'에서 slave node의 서버와 프로세스 수를 지정한다.\n\n'line 13~15'에서 파일을 저장하기 위한 경로를 지정한다.\n\n'line 18'에서 디스크 수를 지정한다.(mac에서는 directory 수,ssd_1~ssd_3)\n\u003cpre\u003e\u003ccode\u003e$ flashbase edit\n\n  1 #!/bin/bash\n  2\n  3 ## Master hosts and ports\n  4 export SR2_REDIS_MASTER_HOSTS=( \"127.0.0.1\" )\n  5 export SR2_REDIS_MASTER_PORTS=( $(seq 18100 18104) )\n  6\n  7 ## Slave hosts and ports (optional)\n  8 export SR2_REDIS_SLAVE_HOSTS=( \"127.0.0.1\" )\n  9 export SR2_REDIS_SLAVE_PORTS=( $(seq 18600 18604) )\n 10\n 11 ## only single data directory in redis db and flash db\n 12 ## Must exist below variables; 'SR2_REDIS_DATA', 'SR2_REDIS_DB_PATH' and 'SR2_FLASH_DB_PATH'\n 13 export SR2_REDIS_DATA=\"/sata_ssd/ssd_\"\n 14 export SR2_REDIS_DB_PATH=\"/sata_ssd/ssd_\"\n 15 export SR2_FLASH_DB_PATH=\"/sata_ssd/ssd_\"\n 16\n 17 ## multiple data directory in redis db and flash db\n 18 export SSD_COUNT=3\n\u003c/code\u003e\u003c/pre\u003e\n\n'redis-master.conf.template'과 'redis-master.conf.template'를 아래와 같이 해당 cluster의 conf directory에 복사한다.('TSR2-package/conf'에 있음)\n\u003cpre\u003e\u003ccode\u003ecp redis-master.conf.template ~/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/conf\ncp redis-slave.conf.template ~/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/conf\n\u003c/code\u003e\u003c/pre\u003e\n\nFlashBase 실행하기\n==============================================================================\n\n아래 순서로 실행한다.\n\n1. 경로 설정\n\u003cpre\u003e\u003ccode\u003esource ~/.use_cluster 1\u003c/code\u003e\u003c/pre\u003e\n\n2. 실행중인 프로세스 중지\n\u003cpre\u003e\u003ccode\u003eflashbase stop\u003c/code\u003e\u003c/pre\u003e\n\n3. 기존 파일 삭제\n\u003cpre\u003e\u003ccode\u003eflashbase clean\u003c/code\u003e\u003c/pre\u003e\n\n4. 'flashbase edit template'을 입력하여, 설정값을 확인 및 변경\n\u003cpre\u003e\u003ccode\u003eflashbase edit template\u003c/code\u003e\u003c/pre\u003e\n\n5. 초기화와 함께 클러스터 구성\n\u003cpre\u003e\u003ccode\u003eflashbase restart --reset --cluster --yes\u003c/code\u003e\u003c/pre\u003e\n\n6. 초기화 및 실행이 안되는 경우 대처\n\n  - 아래 위치로 가서 로그를 확인한다.\n\u003cpre\u003e\u003ccode\u003ecd $SR2_HOME/logs/redis\u003c/code\u003e\u003c/pre\u003e\n\n  - 로그에 아무런 정보가 없으면 아래와 같이 프로세스 하나만 실행해서 발생한 로그 메시지를 확인한다.\n  - 'cluster_1' 부분은 클러스터에 맞게 수정 후 사용\n\u003cpre\u003e\u003ccode\u003eDYLD_LIBRARY_PATH=~/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/lib/native/ ~/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/bin/redis-server ~/tsr2/cluster_1/tsr2-assembly-1.0.0-SNAPSHOT/conf/redis/redis-18101.conf\u003c/code\u003e\u003c/pre\u003e\n\nFlashBase에 적재하기\n==============================================================================\n\n적재는 아래와 같이 FlashBase package에 포함된 tsr2-tools를 사용한다.\n사용 방법은 'tsr2-tools insert -d [적재할 데이터가 있는 파일 또는 directory] -s \"|\" -t [json file] -p 8 -c 1 -i'와 같다.\n이 경우 seperator로 '|'를 사용했다.\n\n``` console\ntsr2-tools insert -d ./load -s \"|\" -t ./json/test.json -p 8 -c 1 -i\n```\n\nTest data로 적재하기\n------------------------------------------------------------------------------\n아래와 같이 'tsr2-test' directory를 생성하고 관련 파일들을 복사한다.\n\u003cpre\u003e\u003ccode\u003emkdir ~/tsr2-test\u003c/code\u003e\u003c/pre\u003e\n\n적재할 데이터 파일들은 '~/tsr2-test/load'에 복사한다.\n\u003cpre\u003e\u003ccode\u003emkdir ~/tsr2-test/load\u003c/code\u003e\u003c/pre\u003e\n\n'~/tsr2-test/json'에 test.json 파일을 생성한다.\n\u003cpre\u003e\u003ccode\u003emkdir ~/tsr2-test/json\u003c/code\u003e\u003c/pre\u003e\n\n``` console\nvi test.json\n\n  1   {\n  2       \"endpoint\": \"127.0.0.1:18101\",\n  3       \"id\": \"101\",\n  4       \"columns\": 10,\n  5       \"partitions\": [\n  6           0, 3, 4\n  7       ],\n  8       \"rowStore\" : true\n  9   }\n```\n\n적재는 아래와 같이 실행한다.\n``` console\ntsr2-tools insert -d ~/tsr2-test/load -s \"|\" -t test.json -p 8 -c 1 -i\n```\n\n아래와 같이 입력하면 적재 상황을 모니터링할 수 있다.\n\u003cpre\u003e\u003ccode\u003eflashbase cli monitor\u003c/code\u003e\u003c/pre\u003e\n\n적재한 결과는 아래와 같다.\n``` console\n1534483316.477876 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:event_date:4:country:5:id}\" \"3:0:3:4\" \"1\" \"0\" \"event_date\\x1dname\\x1dcompany\\x1dcountry\\x1did\\x1ddata1\\x1ddata2\\x1ddata3\\x1ddata4\\x1ddata5\"\n1534483316.478114 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:180818:4:Mexico:5:1681090140399}\" \"3:0:3:4\" \"1\" \"0\" \"180818\\x1dMaite, Christian, Tad, Illiana\\x1dUltrices Corp.\\x1dMexico\\x1d1681090140399\\x1dPUY52MAG4EK\\x1d6\\x1d5\\x1dut quam\\x1dFusce feugiat. Lorem\"\n1534483316.478257 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:180320:4:American Samoa:5:1680111178699}\" \"3:0:3:4\" \"1\" \"0\" \"180320\\x1dTatum, Eliana, Iola, Colby\\x1dMi Eleifend Egestas Institute\\x1dAmerican Samoa\\x1d1680111178699\\x1dQYV83PRB0JW\\x1d12\\x1d8\\x1dtortor at risus. Nunc ac sem ut dolor\\x1dnon, lobortis quis, pede. Suspendisse dui. Fusce diam\"\n1534483316.478315 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:171102:4:Chile:5:1617010935199}\" \"3:0:3:4\" \"1\" \"0\" \"171102\\x1dTheodore, Holly, Carter, Fulton\\x1dNisi Nibh Lacinia Industries\\x1dChile\\x1d1617010935199\\x1dZEK46GWB7HN\\x1d14\\x1d5\\x1dlacus. Quisque purus sapien,\\x1dtempor lorem, eget mollis lectus pede et risus. Quisque libero\"\n1534483316.478390 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:190718:4:Saint Martin:5:1655052520899}\" \"3:0:3:4\" \"1\" \"0\" \"190718\\x1dKenyon, Jeremy, Hedda, Wayne\\x1dSit Consulting\\x1dSaint Martin\\x1d1655052520899\\x1dZKV28BVO2UJ\\x1d18\\x1d1\\x1dsed pede nec ante blandit viverra. Donec tempus, lorem\\x1da purus. Duis elementum, dui quis accumsan convallis, ante lectus\"\n1534483316.478481 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:181125:4:Djibouti:5:1662041755899}\" \"3:0:3:4\" \"1\" \"0\" \"181125\\x1dMarvin, Berk, Connor, Britanney\\x1dDonec Dignissim Magna Company\\x1dDjibouti\\x1d1662041755899\\x1dTOY60WWP1BV\\x1d22\\x1d10\\x1dMorbi metus. Vivamus euismod urna. Nullam\\x1dmollis vitae, posuere at, velit. Cras lorem lorem, luctus\"\n1534483316.478565 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:190808:4:Peru:5:1633051811499}\" \"3:0:3:4\" \"1\" \"0\" \"190808\\x1dAlisa, Vernon, Gregory, Dale\\x1dUltrices Foundation\\x1dPeru\\x1d1633051811499\\x1dDKY94BEN1QF\\x1d28\\x1d6\\x1dvitae erat\\x1dipsum. Phasellus vitae\"\n1534483316.478632 [0 127.0.0.1:49837] \"NVWRITE\" \"D:{101:1:180726:4:Chad:5:1637081315399}\" \"3:0:3:4\" \"1\" \"0\" \"180726\\x1dDrew, Adrienne, Blaze, Jade\\x1dAt Nisi PC\\x1dChad\\x1d1637081315399\\x1dMKT50ZSU1QN\\x1d32\\x1d10\\x1dsapien. Cras dolor dolor,\\x1dNulla eget metus eu erat semper rutrum. Fusce dolor\"\n...\n...\n\n```\n\n적재한 데이터 확인\n``` console \n$fb cli\n127.0.0.1:18100\u003e metakeys *\n 1) \"M:{101:1:170807:4:Tuvalu:5:1612091856899}\"\n 2) \"M:{101:1:171102:4:Chile:5:1617010935199}\"\n 3) \"M:{101:1:171214:4:Singapore:5:1668021703999}\"\n 4) \"M:{101:1:171221:4:Korea, South:5:1693112634099}\"\n 5) \"M:{101:1:180320:4:American Samoa:5:1680111178699}\"\n 6) \"M:{101:1:180415:4:Montenegro:5:1636031140599}\"\n 7) \"M:{101:1:180504:4:Dominican Republic:5:1610011098499}\"\n 8) \"M:{101:1:180726:4:Chad:5:1637081315399}\"\n 9) \"M:{101:1:180810:4:Tunisia:5:1608020722999}\"\n10) \"M:{101:1:180818:4:Mexico:5:1681090140399}\"\n11) \"M:{101:1:181020:4:Belgium:5:1648100961899}\"\n12) \"M:{101:1:181125:4:Djibouti:5:1662041755899}\"\n13) \"M:{101:1:181224:4:Zambia:5:1662070325099}\"\n14) \"M:{101:1:190312:4:Chile:5:1652011207799}\"\n15) \"M:{101:1:190323:4:Brazil:5:1624050541999}\"\n16) \"M:{101:1:190401:4:Ecuador:5:1638120251599}\"\n17) \"M:{101:1:190714:4:Nicaragua:5:1642072229899}\"\n18) \"M:{101:1:190718:4:Saint Martin:5:1655052520899}\"\n19) \"M:{101:1:190808:4:Peru:5:1633051811499}\"\n20) \"M:{101:1:event_date:4:country:5:id}\"\n127.0.0.1:18100\u003e info keyspace\n# Keyspace\ndb0:keys=40,memKeys=20,flashKeys=0,expires=20,avg_ttl=0\n127.0.0.1:18100\u003e\n\n```\n\n적재한 데이터를 'thriftserver beeline'으로 질의\n아래처럼 Spark executor가 정상적으로 떴으면 thriftserver beeline을 사용해서 질의를 한다.\n ``` console\n2500 ResourceManager\n6294 CoarseGrainedExecutorBackend\n2839 SparkSubmit\n2586 NodeManager\n6266 CoarseGrainedExecutorBackend\n6219 ExecutorLauncher\n ```\n\n'ddl_fb_test_101.sql'를 사용해서 아래와 같이 table을 생성한다.\n``` console\nthriftserver beeline -f ddl_fb_test_101.sql\n```\n\n수행 결과\n``` console\nCREATE TABLE `fb_test` (`user_id` STRING, `name` STRING, `company` STRING, `country` STRING, `event_date` STRING, `data1` STRING, `data2` STRING, `data3` STRING, `data4` STRING, `data5` STRING)\nUSING r2\nOPTIONS (\n  `query_result_partition_cnt_limit` '40000',\n  `query_result_task_row_cnt_limit` '10000',\n  `host` 'localhost',\n  `serialization.format` '1',\n  `query_result_total_row_cnt_limit` '100000000',\n  `group_size` '10',\n  `port` '18102',\n  `mode` 'nvkvs',\n  `partitions` 'user_id country event_date',\n  `table` '101'\n)\n```\n\ntable(fb_test)을 생성하였으므로, 아래와 같이 질의를 하면 된다.\n``` console\nselect event_date, company from fb_test  where event_date \u003e '0';\n\nor\n\nselect count(*) from fb_test;\n\nor\n\nselect event_date, company, count(*) from fb_test  where event_date \u003e '0' group by event_date, company;\n```\n\nFlashBase Commands\n==============================================================================\n\n기본적으로 아래와 같이 입력하여 client를 생성한다.\n\u003cpre\u003e\u003ccode\u003eflashbase cli -h [host] -p [port]\u003c/code\u003e\u003c/pre\u003e\n\n'flashbase cli'만 입력하면 첫번째 host의 첫번째 port로 생성한다.\n\u003cpre\u003e\u003ccode\u003eflashbase cli\u003c/code\u003e\u003c/pre\u003e\n\nclient에 들어온 후에, 'info'를 입력하면 전체 상태를 확인할 수 있다.\n\n``` console\n$flashbase cli -h localhost -p 18500\nlocalhost:18500\u003e info\n# Server\nredis_version:3.0.7\nredis_git_sha1:2d30588f\nredis_git_dirty:0\nredis_build_id:990d7b12314566d7\nredis_mode:cluster\nos:Darwin 17.3.0 x86_64\narch_bits:64\nmultiplexing_api:kqueue\ngcc_version:4.2.1\nprocess_id:52490\nrun_id:7f5dbe653c947acfef05a2ea9b32068fdf1c36b3\ntcp_port:18500\nuptime_in_seconds:237167\nuptime_in_days:2\nhz:10\nlru_clock:3547335\nconfig_file:/Users/admin/dev/cluster_5/tsr2-assembly-1.0.0-SNAPSHOT/conf/redis/redis-18500.conf\n\n# Clients\nconnected_clients:1\nclient_longest_output_list:0\nclient_biggest_input_buf:0\nblocked_clients:0\n\n# Alert Warnings\n[EVICTION_WARNING] partition setting is inefficient, 00Rowgroups ratio is 100\n\n# Memory\nisOOM:false\nused_memory:75585072\nused_memory_human:72.08M\nused_memory_rss:5165056\nused_memory_peak:83956368\nused_memory_peak_human:80.07M\nused_memory_lua:36864\nused_memory_rocksdb_total:114453440\nused_memory_rocksdb_block_cache:100663296\nused_memory_rocksdb_mem_table:13790144\nused_memory_rocksdb_table_readers:0\nused_memory_rocksdb_pinned_block:0\nmeta_data_memory:851760\npercent_of_meta_data_memory:1\nused_memory_client_buffer_peak:0\nmem_fragmentation_ratio:0.07\nmem_allocator:libc\n\n# Persistence\nloading:0\nrdb_changes_since_last_save:288641\nrdb_bgsave_in_progress:0\nrdb_last_save_time:1534242338\nrdb_last_bgsave_status:ok\nrdb_last_bgsave_time_sec:1\nrdb_current_bgsave_time_sec:-1\naof_enabled:1\naof_rewrite_in_progress:0\naof_rewrite_scheduled:0\naof_last_rewrite_time_sec:-1\naof_current_rewrite_time_sec:-1\naof_last_bgrewrite_status:ok\naof_last_write_status:ok\naof_current_size:101291191\naof_base_size:1\naof_pending_rewrite:0\naof_buffer_length:0\naof_rewrite_buffer_length:0\naof_pending_bio_fsync:0\naof_delayed_fsync:0\n\n# Stats\ntotal_connections_received:49\ntotal_commands_processed:2626085\ninstantaneous_ops_per_sec:0\ntotal_net_input_bytes:2727904217\ntotal_net_output_bytes:2935287875\ninstantaneous_input_kbps:0.03\ninstantaneous_output_kbps:0.00\nrejected_connections:0\nsync_full:1\nsync_partial_ok:9\nsync_partial_err:0\nexpired_keys:0\nevicted_keys:0\nkeyspace_hits:0\nkeyspace_misses:0\npubsub_channels:0\npubsub_patterns:0\nlatest_fork_usec:9228\nmigrate_cached_sockets:0\n\n# Replication\nrole:master\nconnected_slaves:1\nslave0:ip=127.0.0.1,port=18600,state=online,offset=2731230897,lag=0\nmaster_repl_offset:2731230897\nrepl_backlog_active:1\nrepl_backlog_size:1048576\nrepl_backlog_first_byte_offset:2730182322\nrepl_backlog_histlen:1048576\n\n# CPU\nused_cpu_sys:124.00\nused_cpu_user:170.14\nused_cpu_sys_children:1.15\nused_cpu_user_children:5.19\n\n# Cluster\ncluster_enabled:1\n\n# Keyspace\ndb0:keys=8884,memKeys=1451,flashKeys=0,expires=7433,avg_ttl=0\nlocalhost:18500\u003e\n```\n\n'info keyspace'와 같이 '#' 뒤에 오는 이름을 입력하면 해당 항목에 대해서만 확인할 수 있다.\n``` console\nlocalhost:18500\u003e info keyspace\n# Keyspace\ndb0:keys=8884,memKeys=1451,flashKeys=0,expires=7433,avg_ttl=0\nlocalhost:18500\u003e\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnms%2Fshare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmnms%2Fshare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnms%2Fshare/lists"}