{"id":20294171,"url":"https://github.com/grab/symphony","last_synced_at":"2025-04-11T11:42:42.515Z","repository":{"id":97825909,"uuid":"206272795","full_name":"grab/symphony","owner":"grab","description":null,"archived":false,"fork":false,"pushed_at":"2021-05-31T06:55:31.000Z","size":17,"stargazers_count":32,"open_issues_count":0,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-25T08:03:15.340Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-04T08:41:55.000Z","updated_at":"2025-02-18T23:40:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"d4a8271b-1004-48ab-b627-9d1a0b8040c5","html_url":"https://github.com/grab/symphony","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fsymphony","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fsymphony/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fsymphony/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fsymphony/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grab","download_url":"https://codeload.github.com/grab/symphony/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248385879,"owners_count":21094964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T15:28:07.259Z","updated_at":"2025-04-11T11:42:42.485Z","avatar_url":"https://github.com/grab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Symphony\n`Symphony` is a Golang library to allow devs to declare dependencies of tasks/functions, \nand `symphony` automatically resolves run all tasks based on the dependencies.\n\nThis library significantly eases the flow-based/graph-based data fetching for Machine Learning prediction, used inside Grab where models require 200+ data in some cases. \n\nTraditionally, when different data comes with dependency(e.g. phone number based features requires calling user service to get phone number first), developers need to explicitly allocate dependent data fetching (usually different service/DB call) at different layers (layer-1: calling user-service to get phone number, layer-2: calling phone-number feature DB by phone number in layer-1). With `Symphony`, Developers can declare the direct dependency between different data, and `Symphony` will build a DAG internally and fetch data based on it, so no need for a developer to specify the global layers.\n\n\n## Quick Usage\n![](https://user-images.githubusercontent.com/1205083/120143502-cbd70b80-c212-11eb-997d-76ea694b01cc.png)\n\n\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"errors\"\n    \"fmt\"\n\n    \"github.com/grab/symphony\"\n)\n\ntype User struct {\n    ID int64\n    Weight float64\n    Height float64\n    BMI float64\n\n}\n\n\nfunc main() {\n    // create a dummy user with ID=1\n    u := \u0026User{ID: 1}\n\n    // init symphony\n    s := symphony.New()\n\n    // Define Task fetchWeight, fetchHeight, calculateBMI\n    // Task can be declare in any order\n    s.Add(\"fetchWeight\", nil, func(res map[string]*symphony.TaskState) (interface{}, error)){\n        // assume a remote WeightService call  \n        u.Weight = callWeightService(u.ID)\n        return \"weight fetched \", nil\n    }).Add(\"fetchHeight\", nil, func(res map[string]*symphony.TaskState) (interface{}, error){\n        // assume a remote HeightService call  \n        u.Height = callHeightService(u.ID)\n        return \"height fetched\", nil\n    }).Add(\"calculateBMI\", []string{\"fetchWeight\", \"fetchHeight\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        // Note that we declare the dependency in the above line, so this block will be called only after fetchWeight and fetchHeight tasks are done\n        u.BMI = callBMICalculator(u.Weight, u.Height)\n        return \"BMI calculated\", nil\n    })\n\n    // shutdown the call if it exceeds 1500ms \n    result, err := s.Do(context.Background(), 1500)\n\n    BMI := result[\"calculateBMI\"]\n    fmt.Printf(\"BMI: %d, err: %s\", BMI, err)\n }\n```\n\n## Basic Usage with Latency measure\nThe task depency graph is the same with Quick usage. This Usage just adds the way to log the latency of tasks.\nTo add the latency check, you just need to create a func like func(statRecord *symphony.TaskRunTimeStat), and then call SetTaskRuntimeStatFunc with the Symphony object. BTW, it is optional.\n\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"errors\"\n    \"fmt\"\n\n    \"github.com/grab/symphony\"\n)\n\n// an example func to log the latency. This is just to print the latency in console, but you can call other log utils too.\n// this function will be called after the symphony finished a task.\nvar symphonyLatencyFunc = func(statRecord *symphony.TaskRunTimeStat) {\n    //// add your latency log function here, like replacing the logLatency to the method your system supported.\n    //// statRecord.StartTime is start time of the task including the time to wait all dependents finishing.\n    //// statRecord.StartTimeForTaskFn is start time of the task's fun, after all dependencies finish.\n    //// statRecord.EndTime is end time of the task's fun, after all dependencies finish.\n    taskName := statRecord.Name\n    startTime := *statRecord.StartTime\n    startTimeForTaskFn := *statRecord.StartTimeForTaskFn\n    endTime := *statRecord.EndTime\n    fmt.Printf(\"taskName:%v, task begin time with depency waiting time: %v, end time: %v, latency: %v\\n\", taskName, startTime, endTime, endTime.Sub(startTime).Milliseconds())\n    fmt.Printf(\"taskName:%v, task begin time without depency waiting time: %v, end time: %v, latency: %v\\n\", taskName, startTimeForTaskFn, endTime, endTime.Sub(startTimeForTaskFn).Milliseconds())\n}\n\nfunc main() {\n     // create a dummy user with ID=1\n    u := \u0026User{ID: 1}\n\n    s := symphony.New()\n\n    // Define Task fetchWeight, fetchHeight, calculateBMI\n    // Task can be declare in any order\n    s.Add(\"fetchWeight\", nil, func(res map[string]*symphony.TaskState) (interface{}, error)){\n        // assume a remote WeightService call  \n        u.Weight = callWeightService(u.ID)\n        return \"weight fetched \", nil\n    }).Add(\"fetchHeight\", nil, func(res map[string]*symphony.TaskState) (interface{}, error){\n        // assume a remote HeightService call  \n        u.Height = callHeightService(u.ID)\n        return \"height fetched\", nil\n    }).Add(\"calculateBMI\", []string{\"fetchWeight\", \"fetchHeight\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        // Note that we declare the dependency in the above line, so this block will be called only after fetchWeight and fetchHeight tasks are done\n        u.BMI = callBMICalculator(u.Weight, u.Height)\n        return \"BMI calculated\", nil\n    })\n\n    // you could set the latency check func here, and it is optional.\n    // the function will be called after the symphony finished a task.\n    s.SetTaskRuntimeStatFunc(symphonyLatencyFunc)\n\n    // shutdown the call if it exceeds 1500ms \n    result, err := s.Do(context.Background(), 1500)\n\n    BMI := result[\"calculateBMI\"]\n    fmt.Printf(\"BMI: %d, err: %s\", BMI, err)\n }\n```\n\n## Advanced Usage\n![](https://user-images.githubusercontent.com/1205083/120151745-37bf7100-c21f-11eb-91c7-6d19f28c501b.png)\n\nAssuming the task dependency graph is the above one, \nthe following code will resolve and run based on this\n\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"errors\"\n    \"fmt\"\n\n    \"github.com/grab/symphony\"\n)\n\nfunc main() {\n    s := symphony.New()\n    \n    // Define Task f1, f2, f3, f4, f5, f6, f7\n    // Task can be declare in any order\n    s.Add(\"f2\", []string{\"f1\", \"f4\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f2==\")\n        return \"f2 result\", nil\n    }).Add(\"f3\", []string{\"f2\", \"f4\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f3==\")\n        return fmt.Sprintf(\"%s|%s|%s\", res[\"f2\"].R, res[\"f4\"].R, \"f3 result\"), nil\n    }).Add(\"f4\", []string{\"f5\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f4==\")\n        return \"f4 result\", nil\n    }).Add(\"f5\", nil, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f5==\")\n        return \"f5 result\", nil\n    }).Add(\"f6\", []string{\"f3\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f6==\")\n        return \"f6 result\", nil\n    }).Add(\"f7\", []string{\"f4\"}, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f7==\")\n        return \"f7 result\", nil\n    }).Add(\"f1\", nil, func(res map[string]*symphony.TaskState) (interface{}, error) {\n        fmt.Println(\"==starting f1==\")\n        return \"f1 result\", nil\n    })\n    // wait up to 1500ms\n    res, err := s.Do(context.Background(), 1500)\n    \n\n    f3r, _ := res[\"f3\"]\n    f6r, _ := res[\"f6\"]\n    fmt.Printf(\"err=%s\", err) // err=nil\n    fmt.Printf(\"f3=%s\", f3r.R) // f3=f2 result|f4 result|f3 result\n    fmt.Printf(\"f6=%s\", f6r.R) // f6=f2 result|f4 result|f3 result|f6 result\n }\n```\n## Features\n+ Automatic trigger dependent task\n  + once all dependencies of a task are solved, the task would start running immediately\n  \n+ Dependency Safety check\n  + Self Dependency\n  + Non-existent dependency\n  + Cyclic Dependency\n  \n+ Short-circuit\n  + If any task returns error, will short-circuit all dependent tasks\n  + In the above code, if `f3` returns error, f6 will not run (and return same error to all its dependencies if any)\n  \n + Timeout\n   + at `s.Do(context.Background(), 1500)`, `1500` defines the max running time in millisecond for, and will timeout by returning error\n   if it exceeds this\n   \nPlease see `symphony_test.go` for more use cases\n\n## Maintainers\n* [Muqi Li](https://www.linkedin.com/in/muqili/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrab%2Fsymphony","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrab%2Fsymphony","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrab%2Fsymphony/lists"}