{"id":19969855,"url":"https://github.com/naupio/pical","last_synced_at":"2025-10-19T09:58:35.541Z","repository":{"id":108483139,"uuid":"132692771","full_name":"Naupio/pical","owner":"Naupio","description":"(Work In Process) pita is a general distributed computation system with Erlang language base on DAG model. This project is inspired by DouBan 's DPark and Apache Spark.","archived":false,"fork":false,"pushed_at":"2021-12-22T07:06:55.000Z","size":36,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-08T00:41:38.582Z","etag":null,"topics":["big-data","bigdata","dag","data","distributed","distributed-computing","distributed-systems","erlang","erlang-otp","flink","spark"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Naupio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-09T02:48:41.000Z","updated_at":"2025-01-11T18:26:34.000Z","dependencies_parsed_at":"2023-04-26T03:17:39.412Z","dependency_job_id":null,"html_url":"https://github.com/Naupio/pical","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Naupio%2Fpical","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Naupio%2Fpical/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Naupio%2Fpical/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Naupio%2Fpical/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Naupio","download_url":"https://codeload.github.com/Naupio/pical/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252276955,"owners_count":21722447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","bigdata","dag","data","distributed","distributed-computing","distributed-systems","erlang","erlang-otp","flink","spark"],"created_at":"2024-11-13T02:51:53.840Z","updated_at":"2025-10-19T09:58:35.462Z","avatar_url":"https://github.com/Naupio.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PiCal\n\n(Work In Process) **PiCal** is a general distributed computation system with **Elixir** language base on DAG model. This project is inspired by [DouBan 's DPark](https://github.com/douban/dpark) and [Apache Spark](https://github.com/apache/spark).  \n\n# LICENSE\n- The [MIT License](./LICENSE)  \n- Copyright (c) 2016-2022 [Naupio Z.Y Huang](https://github.com/Naupio)  \n\n# WARNING\nThis project is **not finish** (yet).\n\n---\n---\n\n# **DAG Engine TODO LIST**\n# RDD  \n\n* getPartition  \n* compute  \n* Dependency  \n* Partitioner for K-V RDDs (Optional)  \n* preferredLocations (Optional)  \n\n# BaseRDD  \n```\nParallelCollectionRDD\nMappedRDD\nFlatMappedRDD\nMapPartitionsRDD\nMappedValuesRDD\nFlatMappedValuesRDD\nFilteredRDD\nShuffledRDD\nTextFileRDD\nOutputTextFileRDD\nUnionRDD\nCoGroupedRDD\nCartesianRDD\nCoalescedRDD\nSampleRDD\nCheckpointRDD\n\n```\n**`PipedRDD`**\n\n# DataSource\n* parallelize :\u003e ParallelCollectionRDD  \n* textFile :\u003e TextFileRDD  \n\n# Transformation  \n## simpleTransformation\n* map(func) :\u003e MappedRDD  \ncompute:\u003e iterator(split).map(f)  \n\n* filter(func) :\u003e FilteredRDD  \ncompute:\u003e iterator(split).filter(f)  \n\n* flatMap(func) :\u003e FlatMappedRDD  \ncompute:\u003e iterator(split).flatMap(f)  \n\n* mapPartitions(func) :\u003e MapPartitionsRDD  \ncompute:\u003e f(iterator(split))  \n\n* mapPartitionsIndex(func) :\u003e MapPartitionsRDD  \ncompute:\u003e f(split.index, iterator(split))  \n\n* sample(withReplacement, fraction, seed) :\u003e PartitionwiseSampledRDD  \ncompute:\u003e  \nPoissonSampler.sample(iterator(split))\nBernoulliSampler.sample(iterator(split))\n\n## complexThansformation\n* union(otherDataset) :\u003e (RDD a, RDD b) =\u003e UnionRDD  \n* groupByKey([numTasks]) :\u003e  RDD a =\u003e ShuffledRDD =\u003e MapPartitionsRDD  \n* reduceByKey(func, [numTasks]) :\u003e RDD a =\u003e MapPartitionsRDD =\u003e ShuffledRDD =\u003e MapPartitionsRDD  \n* distinct([numTasks])) :\u003e RDD a =\u003e MappedRDD =\u003e MapPartitionsRDD =\u003e ShuffledRDD =\u003e MapPartitionsRDD =\u003e MappedRDD  \n* cogroup(otherDataset, [numTasks]) :\u003e (RDD a, RDD b) =\u003e CoGroupedRDD =\u003e MappedValuesRDD  \n* intersection(otherDataset) :\u003e (RDD a, RDD b) =\u003e (MappedRDD a, MappedRDD b) =\u003e CoGroupedRDD =\u003e MappedValuesRDD =\u003e FilteredRDD =\u003e MappedRDD  \n* join(otherDataset, [numTasks]) :\u003e (RDD a, RDD b) =\u003e CoGroupedRDD =\u003e MappedValuesRDD =\u003e FlatMappedValuesRDD  \n* sortByKey([ascending], [numTasks]) :\u003e RDD a =\u003e ShuffledRDD =\u003e MapPartitionsRDD\n* cartesian(otherDataset) :\u003e (RDD a, RDD b) =\u003e  CartesianRDD\n* coalesce(numPartitions,shuffle=false) :\u003e RDD a =\u003e CoalescedRDD\n* repartition(numPartitions) == coalesce(numPartitions,shuffle=true) :\u003e RDD a =\u003e MapPartitionsRDD =\u003e ShuffledRDD =\u003e CoalescedRDD =\u003e MappedRDD  \n* combineByKey() :\u003e \taggregate and compute()\n```\ncombineByKey(createCombiner:\tV\t=\u003e\tC,\n\t\t\t\t\t\tmergeValue:\t(C,\tV)\t=\u003e\tC,\n\t\t\t\t\t\tmergeCombiners:\t(C,\tC)\t=\u003e\tC,\n\t\t\t\t\t\tpartitioner:\tPartitioner,\n\t\t\t\t\t\tmapSideCombine:\tBoolean\t=\ttrue,\n\t\t\t\t\t\tserializer:\tSerializer\t=\tnull):\tRDD[(K,\tC)])\n```\n* **pipe(command,\t[envVars])  :\u003e PipedRDD**  \n\n# Action  \n* reduce(func) :\u003e (record1, record2) =\u003e  result, (result, record i) =\u003e result  \ncompute(results) :\u003e (result1,\tresult2)\t=\u003e\tresult,\t(result, result i) =\u003e result  \n\n* collect() :\u003e Array[records] =\u003e result  \ncompute(results) :\u003e Array[result]  \n\n* count() :\u003e count(records) =\u003e result  \ncompute(results) :\u003e sum(result)  \n\n* foreach(f) :\u003e f(records) =\u003e result  \ncompute(results) :\u003e Array[result]  \n\n* take(n) :\u003e record(i\u003cn) =\u003e result  \ncompute(results) :\u003e Array[result]  \n\n* frist() :\u003e record 1 =\u003e result  \ncompute(results) :\u003e Array[result]  \n\n* takeSample() :\u003e selectd records =\u003e result  \ncompute(results) :\u003e Array[result]  \n\n* takeOrdered(n,[ordering]) :\u003e TopN(records) =\u003e result  \ncompute(results) :\u003e TopN(results)  \n\n* saveAsFile(path) :\u003e records =\u003e write(records)  \ncompute(results) :\u003e null  \n\n* countByKey() :\u003e (K, V) =\u003e Map(K, count(K))  \ncompute(results) :\u003e (Map,\tMap)\t=\u003e\tMap(K,\tcount(K))  \n\n---\n\n# Partitioner\n* HashPartitioner  \n* RangePartitioner  \n\n# Aggregator\n* createCombiner  \n* mergeValue  \n* mergeCombiner  \n\n---\n\n# Dependency\n\n## NarrowDenpendency  \n* OneToOneDependency (1:1)    \n* RangeDependency  \n* NarrowDenpendency (N:1)\n\n## WideDenpendency  \n* ShuffleDependency (M:N)  \n\n\n---\n\n# Scheduler\n## DAGScheduler\n* one ShuffleDependency one stage  \n\n## TaskScheduler\n* one finalRDD-partition one task  \n\n## Job  \n* runJob(rdd,\tprocessPartition,\tresultHandler)  \n* runJob(rdd,\tcleanedFunc,\tpartitions,\tallowLocal,\tresultHandler)  \n* submitJob(rdd,\tfunc,\tpartitions,\tallowLocal,\tresultHandler)  \n* handleJobSubmmitted()  \n\n## Stage\n* noParentStage computeSoon  \n* haveParentStage waitParentComputeFinish  \n* newStage()  \n* submitStage(finalStage)  \n* submitWaitingStages()  \n\n**ShuffleMapStage**  \n**ResultStage**  \n\n## Task  \n* ShuffleMapTask  \n* ResultTask  \n* TaskSet\n* submitTasks(taskSet)\n* LaunchTask(new\tSerializableBuffer(serializedTask))\n\n---\n\n# Shuffle  \n## Shuffle\twrite  \nShuffleBlockFile/FileSegment :\u003e record =\u003e partition =\u003e persist in bucket\nFileConsolidation\t:\u003e cores *\tR  \n\n## Shuffle\tread  \nfetch and combine (aggregate in HashMap)\n\n---\n# RTS  \n* masterNode  \n* workerNode  \n* driverNode  \n* executorBackend  \n* executorRunner  \n\n---\n# Persist\n* Cache  \n* Checkpoint\n\n---\n# Accumulator\n* value  \n* list  \n* set  \n* dict  \n\n---\n# Broadcast\n* BroadcastManager\n* P2PBroadcastManager\n\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaupio%2Fpical","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnaupio%2Fpical","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaupio%2Fpical/lists"}