{"id":15678963,"url":"https://github.com/billyrrr/onto","last_synced_at":"2025-05-07T09:07:28.611Z","repository":{"id":44336864,"uuid":"201345943","full_name":"billyrrr/onto","owner":"billyrrr","description":"Idealistic and yet usable framework for event-driven backend ","archived":false,"fork":false,"pushed_at":"2022-10-06T09:48:15.000Z","size":1054,"stargazers_count":12,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-07T09:07:22.812Z","etag":null,"topics":["architecture","backend","backend-for-frontend","firestore","flasgger","framework","gcloud","mobile-backend","openapi","python3","reactive-programming","reactive-services","stream-processing","websocket"],"latest_commit_sha":null,"homepage":"https://flask-boiler.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/billyrrr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-08T22:21:57.000Z","updated_at":"2024-01-08T16:36:25.000Z","dependencies_parsed_at":"2022-08-26T02:43:01.363Z","dependency_job_id":null,"html_url":"https://github.com/billyrrr/onto","commit_stats":null,"previous_names":["billyrrr/flask-boiler"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billyrrr%2Fonto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billyrrr%2Fonto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billyrrr%2Fonto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billyrrr%2Fonto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/billyrrr","download_url":"https://codeload.github.com/billyrrr/onto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252847490,"owners_count":21813453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["architecture","backend","backend-for-frontend","firestore","flasgger","framework","gcloud","mobile-backend","openapi","python3","reactive-programming","reactive-services","stream-processing","websocket"],"created_at":"2024-10-03T16:25:37.359Z","updated_at":"2025-05-07T09:07:28.592Z","avatar_url":"https://github.com/billyrrr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# onto\n\n[![Build Status](https://travis-ci.com/billyrrr/flask-boiler.svg?branch=master)](https://travis-ci.com/billyrrr/flask-boiler)\n[![Coverage Status](https://coveralls.io/repos/github/billyrrr/flask-boiler/badge.svg?branch=master)](https://coveralls.io/github/billyrrr/flask-boiler?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/flask-boiler/badge/?version=latest)](https://flask-boiler.readthedocs.io/en/latest/?badge=latest)\n\n\nDemo: \n\nWhen you change the attendance status of one of the participants \nin the meeting, all other participants receive an updated version \nof the list of people attending the meeting. \n\n![Untitled_2](https://user-images.githubusercontent.com/24789156/71137341-be0e1000-2242-11ea-98cb-53ad237cac43.gif)\n\nSome reasons that you may want to use this framework or architectual\npractice:\n- You want to build a reactive system and not just a reactive view. \n- You want to build a scalable app that is native to distributed \n    systems. \n- You want a framework with a higher level of abstraction, so you can \n    exchange components such as transportation protocols \n- You want your code to be readable and clear and written mostly \n    in python, while maintaining compatibility to different APIs. \n- You have constantly-shifting requirements, and want to have \n    the flexibility to migrate different layers, for example, \n    switch from REST API to WebSocket to serve a resource. \n\nThis framework is at ***beta testing stage***. \nAPI is not guaranteed and ***may*** change. \n\nDocumentations: [readthedocs](https://flask-boiler.readthedocs.io/)\n\nQuickstart: [Quickstart](https://flask-boiler.readthedocs.io/en/latest/quickstart_link.html)\n\nAPI Documentations: [API Docs](https://flask-boiler.readthedocs.io/en/latest/apidoc/flask_boiler.html)\n\nExample of a Project using onto (now named onto): [gravitate-backend](https://github.com/billyrrr/gravitate-backend)\n\n[Related Technologies](https://medium.baqend.com/real-time-databases-explained-why-meteor-rethinkdb-parse-and-firebase-dont-scale-822ff87d2f87)\n\n## Connectors supported \n\nImplemented: \n- REST API (Flask and Flasgger)\n- GraphQL (Starlette)\n- Firestore\n- Firebase Functions\n- JsonRPC (flask-jsonrpc)\n- Leancloud Engine\n- WebSocket (flask socketio)\n\nTo be supported: \n- Flink Table API\n- Kafka\n\n## What I am currently trying to build\n\nFront end creates mutations in graphql. \n\"Onto\" receives the view model, and triggers \naction on domain model. A method in domain model \nis called (which lives in Flink Stateful Functions\nruntime). Different domain models communicate to \npersist a change, and save the output view into Kafka. \nAnother set of system statically interprets \"view \nmodel definition code\" as SQL, \nand submit jobs with Flink SQL to assemble \"view model\". \nEventually, the 1NF view of the data is sent to Kafka, \nand eventually delivered to front end in forms of \nGraphQL Subscription. \n\n(Write side has Serializable-level consistency, \nand read side has eventual consistency) \n\n## What it already does  \n- Serialization and deserialization\n- GraphQL/Flask server \n- Multiple table join\n- ... \n\n## Installation\nIn your project directory, \n\n```\npip install onto\n```\n\nSee more in [Quickstart](https://onto.readthedocs.io/en/latest/quickstart_link.html). \n\n\u003c!--## Usage--\u003e\n\n\u003c!--### Business Properties Binding--\u003e\n\u003c!--You can bind a view model to its business properties (underlying domain model).--\u003e\n\u003c!--See `examples/binding_example.py`. (Currently breaking)--\u003e\n\n\u003c!--```python--\u003e\n\n\u003c!--vm: Luggages = Luggages.new(vm_ref)--\u003e\n\n\u003c!--vm.bind_to(key=id_a, obj_type=\"LuggageItem\", doc_id=id_a)--\u003e\n\u003c!--vm.bind_to(key=id_b, obj_type=\"LuggageItem\", doc_id=id_b)--\u003e\n\u003c!--vm.register_listener()--\u003e\n\n\u003c!--```--\u003e\n\n### State Management\n\nYou can combine information gathered in domain models and serve them in Firestore, so \nthat front end can read all data required from a single document or collection, \nwithout client-side queries and excessive server roundtrip time. \n\nThere is a medium [article](https://medium.com/resolvejs/resolve-redux-backend-ebcfc79bbbea) \n that explains a similar architecture called \"reSolve\" architecture. \n\nSee ```examples/meeting_room/view_models``` on how to use onto \nto expose a \"view model\" in firestore that can be queried directly \nby front end without aggregation.  \n\n### Processor Modes\n\n`onto` is essentially a framework for source-sink operations: \n\n```\nSource(s) -\u003e Processor -\u003e Sink(s)\n```\n\nTake query as an example,  \n\n- Boiler\n- NoSQL\n- Flink\n    - staticmethods: converts to UDF\n    - classmethods: converts to operators and aggregator's \n    \n\n### Declare View Model\n\n```python\nfrom onto.attrs import attrs\n\nclass CityView(ViewModel):\n\n    name: str = attrs.nothing\n    country: str = attrs.nothing\n\n    @classmethod\n    def new(cls, snapshot):\n        store = CityStore()\n        store.add_snapshot(\"city\", dm_cls=City, snapshot=snapshot)\n        store.refresh()\n        return cls(store=store)\n\n    @name.getter\n    def name(self):\n        return self.store.city.city_name\n\n    @country.getter\n    def country(self):\n        return self.store.city.country\n\n    @property\n    def doc_ref(self):\n        return CTX.db.document(f\"cityView/{self.store.city.doc_id}\")\n```\n\n### Document View\n\n``` python\n\nclass MeetingSessionGet(Mediator):\n\n    from onto import source, sink\n\n    source = source.domain_model(Meeting)\n    sink = sink.firestore()  # TODO: check variable resolution order\n\n    @source.triggers.on_update\n    @source.triggers.on_create\n    def materialize_meeting_session(self, obj):\n        meeting = obj\n        assert isinstance(meeting, Meeting)\n\n        def notify(obj):\n            for ref in obj._view_refs:\n                self.sink.emit(reference=ref, snapshot=obj.to_snapshot())\n\n        _ = MeetingSession.get(\n            doc_id=meeting.doc_id,\n            once=False,\n            f_notify=notify\n        )\n        # mediator.notify(obj=obj)\n\n    @classmethod\n    def start(cls):\n        cls.source.start()\n\n```\n\n### Create Flask View\nYou can use a RestMediator to create a REST API. OpenAPI3 docs will be \nautomatically generated in ```\u003csite_url\u003e/apidocs``` when you run ```_ = Swagger(app)```. \n\n```python\napp = Flask(__name__)\n\nclass MeetingSessionRest(Mediator):\n\n    # from onto import source, sink\n\n    view_model_cls = MeetingSessionC\n\n    rest = RestViewModelSource()\n\n    @rest.route('/\u003cdoc_id\u003e', methods=('GET',))\n    def materialize_meeting_session(self, doc_id):\n\n        meeting = Meeting.get(doc_id=doc_id)\n\n        def notify(obj):\n            d = obj.to_snapshot().to_dict()\n            content = jsonify(d)\n            self.rest.emit(content)\n\n        _ = MeetingSessionC.get(\n            doc_id=meeting.doc_id,\n            once=False,\n            f_notify=notify\n        )\n\n    # @rest.route('/', methods=('GET',))\n    # def list_meeting_ids(self):\n    #     return [meeting.to_snapshot().to_dict() for meeting in Meeting.all()]\n\n    @classmethod\n    def start(cls, app):\n        cls.rest.start(app)\n\nswagger = Swagger(app)\n\napp.run(debug=True)\n```\n\n(currently under implementation) \n\n## Object Lifecycle\n\n### Once\n\nObject created with ```cls.new``` -\u003e \nObject exported with ```obj.to_view_dict```. \n\n### Multi\n\nObject created when a new domain model is created in database -\u003e \nObject changed when underlying datasource changes -\u003e \nObject calls ```self.notify``` \n\n## Typical ViewMediator Use Cases \n\nData flow direction is described as Source -\u003e Sink. \n\"Read\" describes the flow of data where front end would find data in Sink useful. \n\"Write\" describes the flow of data where the Sink is the single source \nof truth. \n\n### Rest \n\nRead: Request -\u003e Response \\\nWrite: Request -\u003e Document\n\n1. Front end sends HTTP request to Server  \n2. Server queries datastore\n3. Server returns response\n\n### Query\n\nRead: Document -\u003e Document \\\nWrite: Document -\u003e Document\n\n1. Datastore triggers update function \n2. Server rebuilds ViewModel that may be changed as a result \n3. Server saves newly built ViewModel to datastore \n\n### Query+Task\n\nRead: Document -\u003e Document \\\nWrite: Document -\u003e Document\n\n1. Datastore triggers update function for document `d` at time `t`\n2. Server starts a transaction\n3. Server sets write_option to only allow commit if documents are last updated at time `t` (still under design)\n3. Server builds ViewModel with transaction \n5. Server saves ViewModel with transaction\n7. Server marks document `d` as processed (remove document or update a field)\n7. Server retries up to MAX_RETRIES from step 2 if precondition failed \n\n### WebSocket\n\nRead: Document -\u003e WebSocket Event \\\nWrite: WebSocket Event -\u003e Document\n\n1. Front end subscribes to a ViewModel by sending a WebSocket event to server \n2. Server attaches listener to the result of the query\n3. Every time the result of the query is changed and consistent:\n    1. Server rebuilds ViewModel that may be changed as a result \n    2. Server publishes newly built ViewModel\n4. Front end ends the session\n5. Document listeners are released \n\n### Document\n\nRead: Document -\u003e Document \\\nWrite: Document -\u003e Document\n\n### Comparisons \n\n|                 \t| Rest \t            | Query \t     | Query+Task                   | WebSocket \t    | Document |\n|-----------------\t|------         \t|-------\t|------------\t|-----------\t|----------\t|\n| Guarantees      \t|    ≤1   (At-Most-Once)         \t| ≥ 1 (At-Least-Once)          |  =1[^1] (Exactly-Once)    |   ≤1   (At-Most-Once)  \t|       ≥ 1 (At-Least-Once) \t|\n| Idempotence      \t| If Implemented    | No            | Yes, with transaction[^1]    \t| If Implemented  \t| No    |\n| Designed For      | Stateless Lambda  |  Stateful Container   | Stateless Lambda      | Stateless Lambda  | Stateful Container |\n| Latency         \t| Higher            | Higher \t|   Higher     |  Lower           \t|     Higher     \t|\n| Throughput      \t| Higher when Scaled| Lower[^2]       \t| Lower          \t|   Higher when Scaled\t|   Lower[^2]      \t|\n| Stateful        \t| No   \t            | If Implemented    | If Implemented   \t| Yes        \t| Yes         \t|\n| Reactive        \t| No   \t        | Yes    | Yes   \t| Yes        \t| Yes         \t|\n\n\u003c!---\nGaurantees\n| Back Pressure   \t|      \t|       \t|            \t|           \t|          \t|\nLatency\nThroughput\n| Fault Tolerance \t|      \t|       \t|            \t|           \t|          \t|\nStateful\n--\u003e\n\n[^1]:  A message may be received and processed by multiple consumer, but only one \nconsumer can successfully commit change and mark the event as processed. \n[^2]:  Scalability is limited by the number of listeners you can attach to the datastore.\n\n## Comparisons \n\n### GraphQL\n\nIn GraphQL, the fields are evaluated with each query, but \nonto evaluates the fields if and only if the \nunderlying data source changes. This leads to faster \nread for data that has not changed for a while. Also, \nthe data source is expected to be consistent, as the \nfield evaluation are triggered after all changes made in \none transaction to firestore is read. \n\nGraphQL, however, lets front-end customize the return. You \nmust define the exact structure you want to return in onto. \nThis nevertheless has its advantage as most documentations \nof the request and response can be done the same way as REST API. \n\n### REST API / Flask\n\nREST API does not cache or store the response. When \na view model is evaluated by onto, the response \nis stored in firestore forever until update or manual removal. \n\nonto controls role-based access with security rules \nintegrated with Firestore. REST API usually controls these \naccess with a JWT token. \n\n### Redux\n\nRedux is implemented mostly in front end. onto targets \nback end and is more scalable, since all data are communicated \nwith Firestore, a infinitely scalable NoSQL datastore. \n\nonto is declarative, and Redux is imperative. \nThe design pattern of REDUX requires you to write functional programming \nin domain models, but onto favors a different approach: \nViewModel reads and calculates data from domain models \nand exposes the attribute as a property getter. (When writing \nto DomainModel, the view model changes domain model and \nexposes the operation as a property setter). \nNevertheless, you can still add function callbacks that are \ntriggered after a domain model is updated, but this \nmay introduce concurrency issues and is not perfectly supported \ndue to the design tradeoff in onto. \n\n\n### Architecture Diagram: \n\n![Architecture Diagram](https://user-images.githubusercontent.com/24789156/70380617-06e4d100-18f3-11ea-9111-4398ed0e865c.png)\n\n## Contributing\nPull requests are welcome. \n\nPlease make sure to update tests as appropriate.\n\n## License\n[MIT](https://choosealicense.com/licenses/mit/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbillyrrr%2Fonto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbillyrrr%2Fonto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbillyrrr%2Fonto/lists"}