{"id":15690944,"url":"https://github.com/vkuznet/transfer2go","last_synced_at":"2025-05-08T00:32:35.788Z","repository":{"id":57608866,"uuid":"80159126","full_name":"vkuznet/transfer2go","owner":"vkuznet","description":"Distributed, loosely couple agent-based transferring system","archived":false,"fork":false,"pushed_at":"2019-10-14T14:38:26.000Z","size":4475,"stargazers_count":9,"open_issues_count":9,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-26T06:11:25.478Z","etag":null,"topics":["agent-request","cms","phedex","priority","tfc","transfer-request"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vkuznet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-26T21:51:15.000Z","updated_at":"2024-11-19T18:46:26.000Z","dependencies_parsed_at":"2022-08-30T10:30:18.836Z","dependency_job_id":null,"html_url":"https://github.com/vkuznet/transfer2go","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2Ftransfer2go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2Ftransfer2go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2Ftransfer2go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vkuznet%2Ftransfer2go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vkuznet","download_url":"https://codeload.github.com/vkuznet/transfer2go/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243059757,"owners_count":20229628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-request","cms","phedex","priority","tfc","transfer-request"],"created_at":"2024-10-03T18:17:58.000Z","updated_at":"2025-03-11T15:32:07.038Z","avatar_url":"https://github.com/vkuznet.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# transfer2go\n\n[![Build Status](https://travis-ci.org/vkuznet/transfer2go.svg?branch=master)](https://travis-ci.org/vkuznet/transfer2go)\n[![Go Report Card](https://goreportcard.com/badge/github.com/vkuznet/transfer2go)](https://goreportcard.com/report/github.com/vkuznet/transfer2go)\n[![GoDoc](https://godoc.org/github.com/vkuznet/transfer2go?status.svg)](https://godoc.org/github.com/vkuznet/transfer2go)\n[![DOI](https://zenodo.org/badge/80159126.svg)](https://zenodo.org/badge/latestdoi/80159126)\n\nGo implementation of CMS\n[PhEDEx](https://www.researchgate.net/publication/228732867_Data_transfer_infrastructure_for_CMS_data_taking)\ndistributed, loosly coupled agents for CMS transfering data.\n\n### Description\nThe [CMS](http://cms.web.cern.ch/) experiment at the LHC proton-proton collider\ndeveloped PhEDEx (Physics Experiment Data Export) service as reliable and\nscalable data management system to meet experiment requirements in Run I/II.\nDesigned in 2004, and written mainly in Perl, it is still used today for\nmanaging multi-PB transfer loads per week, across a complex topology of dozen\nof Grid-compliant computing centres.\n\nIts architecture, instead of having a central brain making global decisions on\nall CMS replica allocation, has a data management layer composed of a set of\nloosely coupled and stateless software agents - each managing highly specific\nparts of replication operations at each site in the distribution network -\nwhich communicate asynchronously through a central blackboard architecture.\nThe system is resilient and robust against a large variety of possible failure\nreasons, and it has been designed by assuming a transfer will fail (thus\nimplementing fail-over tactics) and being completely agnostic on the\nlower-level file transfer mechanism (thus focusing on full dataset management\nand detailed transfer bookkeeping). Collision data and derived data collected\nat LHC that allowed to achieve the Higgs boson discovery by ATLAS and CMS\nexperiments at LHC were transferred in the CMS worldwide domain using this\ntoolkit.\n\nThe aim of this project is to extend basic PhEDEX functionality\nto address up-coming challenges in exa-byte HL-HLC era via implementation of modern\nGo programming language.\n\nThe motivation for the effort is many fold:\n- eliminate central blackboard system and necessity to rely on ORACLE backend\n  via fully distributed nature of the agents, self-discovery and task\n  delegation;\n- even though current system is working well it lacks of support and expertise\n  in perl programming language. We would like to leverage modern language such as\n  Go to enhance concurrency model via native support in a language, and\n  dependency free deployment;\n- the data volume expected in HL-HLC era will grow significantly to exa-byte\n  level and we need to explore elasticity approach to handle variety of\n  opportunistic resources;\n- extend file access and transfer patterns to event streaming, or individual objects, etc.;\n- implement support for user produced data in addition to centrally produced and manager by the system\n- take advantage of built-in concurrency model of the Go language and explore the scalability boundaries.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvkuznet%2Ftransfer2go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvkuznet%2Ftransfer2go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvkuznet%2Ftransfer2go/lists"}