{"id":15971688,"url":"https://github.com/vkuznet/dbconsistencycheck","last_synced_at":"2025-04-04T15:44:58.553Z","repository":{"id":136816870,"uuid":"56597587","full_name":"vkuznet/DBConsistencyCheck","owner":"vkuznet","description":"CMS DBS vs PhEDEx database consistency check","archived":false,"fork":false,"pushed_at":"2016-09-05T14:52:54.000Z","size":4300,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-10T01:41:25.844Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PLSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vkuznet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-19T13:08:41.000Z","updated_at":"2016-07-04T19:40:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"bdb5aba0-262a-4746-bab2-d55c5c8c0539","html_url":"https://github.com/vkuznet/DBConsistencyCheck","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2FDBConsistencyCheck","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2FDBConsistencyCheck/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2FDBConsistencyCheck/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2FDBConsistencyCheck/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vkuznet","download_url":"https://codeload.github.com/vkuznet/DBConsistencyCheck/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208066,"owners_count":20901568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T20:40:20.569Z","updated_at":"2025-04-04T15:44:58.526Z","avatar_url":"https://github.com/vkuznet.png","language":"PLSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DBConsistencyCheck\nCMS DBS vs PhEDEx database consistency check\n\n## Project description\nCMS data are recorded in files, which are organized in datasets and blocks.\nDatasets are set of files with common physics content, and have a size ranging from \na few files and few GBs to several hundred thousand files and hundred TBs depending on the \nphysics definition. Datasets are divided into groups of files called blocks to simplify data management;\neach block has a typical size of 100-1000 files and 100 GB-1 TB.\nThe Data bookkeeping system (DBS) and data location system (PhEDEx)\nare two most populated databases in CMS experiment. Both contains\na meta-data information about CMS data, such as dataset, block, file information.\nThe former is a meta-data catalog of the data produced in CMS, the latter\nis a catalog of data placement. The data are stored in those DBs\nby different data operation teams using various tools. Therefore it is\npossible that data may be inconsistent between the two databases.\nYou'll develop tools which will allow to check the content of two databases\nand make a visual (web-based) representation of such differences.\nWe need to compare the following information:\n\n- dataset exists both in PhEDEx and DBS\n- dataset has same blocks in PhEDEx and DBS\n- For each block in dataset:\n  - Block has same files (LFNs) in PhEDEx and DBS\n  - Block is in same open/closed status in PhEDEx and DBS\n  - For each file in block:\n    - File has same size in PhEDEx and DBS\n    - File has same checksums in PhEDEx and DBS\n\nThings to be careful about in checking:\n- Check only datasets/blocks/files older than 1 day\n- Ignore test datasets which may be allowed to be inconsistent (need to define list of test datasets...)\n- We have only closed blocks in DBS; so in practice we just need to \ncheck that blocks are closed in PhEDEx in a reasonable time (less than 1\nday from end of file injections)\n\nThe data from DBS will be fetched via DBS APIs, while the data from PhEDEx\nis already available on HDFS. We'll need to write a python script to\nextract DBS data and store it to HDFS. Then we need to write an additional\nscript to compare data on HDFS for DBS and PhEDEx. Finally, your task will be\nto develop simple web pages to visualize the difference.\n\nThe tasks should be organized to run as cronjobs that we'll be able to\nperform periodic checks.\n\nTools to review:\n- python language\n- HDFS, spark\n\n### PhEDEx data\nPhedex data can be found on HDFS:/project/awg/cms/phedex/catalog/csv\n```\n  hadoop fs -ls /project/awg/cms/phedex/catalog/csv\n```\nThe schema description is here:\nhttp://awg-virtual.cern.ch/data-sources-index/projects/#phedex-file-catalog\nand in short it is:\n```\ndataset_name, dataset_id, dataset_is_open, dataset_time_create,\nblock_name, block_id, block_time_create, block_is_open, file_lfn, file_id,\nfilesize, checksum, file_time_create\n```\n\nHere is an example of rows from HDFS\n```\n/QCD/v1/AODSIM,845055,y,1459821572.83263,/QCD/v1/AODSIM#123,6371058,1459881399.10162,n,/store/lfn.root,87425713,2852194134,\"adler32:3aa28be9,cksum:808021722\",1459894243.63832\n```\n\n### DBS data\nDBS data are available via [DBS APIs](https://cms-http-group.web.cern.ch/cms-http-group/apidoc/dbs3-client/current/dbs.apis.html)\n\nrecently we migrated DBS to HDFS:/project/awg/cms/dbs3verify/CMS_DBS3_PROD_GLOBAL\n\n#### How to run DBS client on lxplus:\n```\n  scramv1 project CMSSW CMSSW_8_0_4\n  cd CMSSW_8_0_4/\n  cmsenv\n  source /cvmfs/cms.cern.ch/crab3/crab.sh\n  voms-proxy-init -rfc -voms cms\n```\n\nExample of dbs_client.py script, we'll use the following procedure:\n\n1. fetch all datasets from DBS (it should yield around 100k datasets)\n2. loop over all datasets and get list of blocks and their attributes (should retrieve closed blocks ~O(1M) in about 5-6h)\n3. loop over all blocks and get list of files (and their attribuets) for individual block (should retrieve\n  around O(10M) files in about 27hours)\n\nThis will create an initial copy of the data and we'll store it to HDFS.\nThen we'll use a timestamp to fetch newer datasets and repeat steps #2 and #3\nto get newer list of blocks/files on periodic basis.\n\nExample of dbs client script:\n``` \n#!/usr/bin/env python\nfrom __future__ import print_function, division\n\n# system modules\nimport os\nimport sys\nimport time\n\n# DBS modules\nfrom dbs.apis.dbsClient import DbsApi\n\ndef client():\n    url = 'https://cmsweb.cern.ch/dbs/prod/global/DBSReader'\n    dbsclient = DbsApi(url)\n    min_cdate = 1460000000 # some time stamp\n    time0 = time.time()\n    datasets = dbsclient.listDatasets(min_cdate=min_cdate)\n    print(\"listDatasets time:\", time.time()-time0)\n    print(\"# datasets\", len(datasets))\n\n    dataset = '/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM'\n    time0 = time.time()\n    blocks = dbsclient.listBlocks(dataset=dataset, detail=True)\n    for blk in blocks:\n        print(blk['block_name'],blk['block_size'],blk['file_count'])\n    print(\"listBlocks time:\", time.time()-time0)\n\n    # get files for given block\n    time0 = time.time()\n    blk = '/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM#f7d604e6-5e87-11e1-a77d-003048f0e7dc'\n    files = dbsclient.listFiles(block_name=blk, validFileOnly=1, detail=True)\n    for lfn in files:\n        print(lfn['logical_file_name'],lfn['check_sum'],lfn['file_size'])\n    print(\"listBlocks time:\", time.time()-time0)\n\ndef main():\n    \"Main function\"\n    client()\n\nif __name__ == '__main__':\n    main()\n```\n\nExample of DBS data:\n\n```\n./dbs_client.py\nlistDatasets time: 0.532593011856\n# datasets 2272\n/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM#f7d604e6-5e87-11e1-a77d-003048f0e7dc 2488909525 1\n/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM#d702c5de-5f53-11e1-a77d-003048f0e7dc 1936668367 1\n/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM#ccf74e52-61c4-11e1-a77d-003048f0e7dc 628209051 1\n/ZMM_14TeV/Summer12-DESIGN42_V17_SLHCTk-v1/GEN-SIM#86fb3df4-60c1-11e1-a77d-003048f0e7dc 1217339580 1\nlistBlocks time: 0.0101478099823\n/store/mc/Summer12/ZMM_14TeV/GEN-SIM/DESIGN42_V17_SLHCTk-v1/0000/EE8A7288-865E-E111-A0BC-00266CFAE684.root 1025833106 2488909525\nlistBlocks time: 0.0118079185486\n```\n\n## References\nThe CMS Data Management System \nM. Giffels, Y. Guo, V. Kuznetsov, N. Magini and T. Wildish J. Phys.: Conf. Ser. 513 042052 doi:10.1088/1742-6596/513/4/042052, http://iopscience.iop.org/article/10.1088/1742-6596/513/4/042052/pdf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvkuznet%2Fdbconsistencycheck","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvkuznet%2Fdbconsistencycheck","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvkuznet%2Fdbconsistencycheck/lists"}