{"id":16096766,"url":"https://github.com/futtetennista/lambdawerk-backend-test","last_synced_at":"2026-04-16T14:05:26.942Z","repository":{"id":141715823,"uuid":"111792188","full_name":"futtetennista/lambdawerk-backend-test","owner":"futtetennista","description":"My proposed solution to the Lambdawerk backend test","archived":false,"fork":false,"pushed_at":"2017-11-23T10:54:50.000Z","size":731,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-05T21:15:18.329Z","etag":null,"topics":["aeson","conduit","docker","haskell","postgres","postgresql","postgrest","stm","xml-conduit"],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/futtetennista.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-23T09:53:09.000Z","updated_at":"2017-11-24T21:21:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"d4501baf-99b7-4134-b25a-54de75a3b185","html_url":"https://github.com/futtetennista/lambdawerk-backend-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/futtetennista/lambdawerk-backend-test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futtetennista%2Flambdawerk-backend-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futtetennista%2Flambdawerk-backend-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futtetennista%2Flambdawerk-backend-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futtetennista%2Flambdawerk-backend-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/futtetennista","download_url":"https://codeload.github.com/futtetennista/lambdawerk-backend-test/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/futtetennista%2Flambdawerk-backend-test/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262685617,"owners_count":23348447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aeson","conduit","docker","haskell","postgres","postgresql","postgrest","stm","xml-conduit"],"created_at":"2024-10-09T17:24:55.146Z","updated_at":"2026-04-16T14:05:26.911Z","avatar_url":"https://github.com/futtetennista.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LambdaWerk backend developer test\n\n## Problem assessment\n\n### Ubiquitous Language\n\n- person: an entity that has a first name `fname`, last name\n`lname` and date of birth `dob` and telephone number `phone`.\nA person is uniquely identified by their `fname`, `lname`, `dob`.\n- database: a PostgreSQL instance\n- persons table: a table in the database containing person records\n- person record: a person stored in the persons table\n- XML input file: a file on disk that contains the data to be merged\n- entry: an XML element contained in the XML input file that represents a\nperson with the following form:\n```xml\n\u003cmember\u003e\n \u003cfirstname\u003eJOHN\u003c/firstname\u003e\n \u003clastname\u003eDOE\u003c/lastname\u003e\n \u003cdate-of-birth\u003e2002-02-01\u003c/date-of-birth\u003e\n \u003cphone\u003e9548938821\u003c/phone\u003e\n\u003c/member\u003e\n```\n- well-formed (referred to an entry or aperson record): that has `fname`, `lname`,\nand `dob` not empty and and of the correct type\n- importer: a program that takes the entries in the XML input file and asks the\ndatabase to merge them with the person records in the persons table\n- merge process: update the persons table in such a way that, for each entry:\n  1. if the `phone` in the persons table is equal to the `phone` in the entry\n     nothing should be changed in the persons table\n  2. if the `phone` in the persons table is not equal to the `phone` in the entry\n     it should be changed in the persons table\n  3. if the entry is not stored in the persons table, a new person record needs\n     to be created\n- merge job: a sub-unit consisting of a given number of entries in which the\nmerge process is split for efficiency reasons\n\n### Task\n\nWrite a piece of software that executes the merge process in a way such that:\n- it must be *clean*\n- it must be *correct*\n- it should provide basic loading statistics at the end of the merge process\n- it should process the XML input file efficiently\n- it should minimize the overall run-time of the merge process\n\nBesides that, reason about performance and memory usage of the merge process\n\n## Running the project\n\n```bash\n# create and start database\ndocker-compose up\n# compile the importer\nstack build\n# run the importer\nstack exec importer -- update-file.xml 10000\n```\n\nThe importer right now is meant to be manually run from the command-line. It'd\nbe not hard to put it in a Docker container and run it in an automated fashion.\nThe [db dump](https://www.dropbox.com/s/kf5dv2way7nalik/person.sql.gz?dl=0) and \n[update file](https://www.dropbox.com/s/11p83a4bp3ze73d/update-file.xml?dl=0) \nneed to be downloaded separately as they are too big. Copy the db dump in the \ntop-level folder of the project since that's where the `Dockerfile` expects it.\n\n### Technologies\n\n- database: [PostgreSQL 10](https://www.postgresql.org/). I use `PL/pgSQL`\nmostly to provide stats on person records created or updated during the merge process.\n- importer: [Haskell](https://haskell-lang.org/). It's a purely functional\nand strongly, statically typed language that allowed me to have high assurances\non correctness and cleanness of my code and develop my code in a type-driven fashion.\n- database communication: [PostgREST](http://postgrest.com/) is a library\nthat provides an easy and convenient vai to interface with a PostgreSQL database\ninstance using a REST API over HTTP. It also takes care of authentication using\nJWT (authorization is left to the database)\n- Docker: PostgreSQL, postgREST run inside [Docker](https://www.docker.com/)\ncontainers using Docker Compose. The importer can be run from any machine\nmanually or also packaged in a Docker container.\n\n#### PROS\n\n- Haskell ecosystem is big and provided a lot of nice tools for handling concurrency\nsuch as Software Transactional Memory or the Async library that made expressing\nrelatively complex logic quite straight-forward and enjoyable\n- The vast Haskell ecosystem made it also easy to write code to efficiently\nread entries from in XML input file in and parse them writing a mimimum amount\nof code, and to communicate to the database over REST API\n- Haskell profiling tools are really enjoyable to work with and allow to easily\npin-point performance bottlenecks\n- Working with the DB as a REST API was also enjoyable and helped with separation\nof concerns since PostgREST forces consumers to write store procedures in the\ndatabase to define user-defined workflows. This is also good for performance\n\n#### CONS\n\n- Haskell is not currently used at Lambdawerk as far as I understood and this\ncould make it harder to get deep feedback or for the reviewer to easily understand\nthe code I've written\n- PostgREST is a relatively simple tool that can be used as a black-box but\ninternally it depends on other tools and frameworks - i.e. ngnix - and these become\ntransitive dependencies\n\n### Installation\n\n- [Docker](http://docker.com/) to have a sandboxed and reproduceable infrastructure\n(tested on Version 17.09.0-ce-mac35 (19611) Channel: stable a98b7c1b7c)\n- [Stack](https://docs.haskellstack.org/) build tool to have a sandboxed and\nreproduceable Haskell environment and install all needed dependencies (tested on\nVersion 1.5.1 x86_64 hpack-0.17.1)\n\n### Configuration\n\nComponents need some configurations parameters that need to be supplied via\nenvironment variables, specifically:\n\n- to set the environment variables needed for the PostgreSQL instance, you could\ncreate a file named `postgres.env` and provide values for the following variables:\n  * `POSTGRES_DB`\n  * `POSTGRES_USER`\n  * `POSTGRES_PASSWORD`\n  have a look at the sample file `postgres.env.sample`\n- to set the environment variables needed for the PostgRESTL instance, you could\ncreate a file named `postgREST.env` and provide values for the following variables:\n  * `PGRST_DB_URI` of the form:\n  `postgres://[POSTGRES_USER]:[POSTGRES_PASSWORD]@postgres_container_alias:5432/[POSTGRES_DB]`\n  * `PGRST_DB_SCHEMA`\n  * `PGRST_DB_ANON_ROLE`\n  * `PGRST_SERVER_PROXY_URI`\n  * `PGRST_JWT_SECRET`\n  have a look at the sample file `postgREST.env.sample` and `rsa.jwk.pub.sample`.\n  The latter contains a sample secret that can be set as `PGRST_JWT_SECRET`\n  and to generate JWT tokens to use as `API_TOKEN`\n- the environment variables needed for the importer are:\n  * `API_ENDPOINT`\n  * `API_TOKEN`\n\n## Implementation considerations\n\nWhile pondering and evaluating a solution, I identified three sub-problems:\n1. reading and parsing the XML input file\n2. serialising the content of the XML input file\n3. merging entries respecting the given invariants\n\nTo achieve a *clean* architecture my design decision early on has been that\nthe importer will be responsable for 1. and the database for 3.\nThe following sequence diagram aims to give a high level view of the architecture\nof the software I implemented\n\n![Merge process sequence diagram](images/merge-process-seqdiag.png)\n\n### 1. Reading and parsing the XML input file\n\nIt was clear from reading the problem definition that the XML input file was\npotentially to large to be read and parsed in memory. So the alternative was to\nstream it. Because lazy I/O is quite tricky and has subtleties that makes it easy\nto do something wrong, I used the [conduit](https://www.stackage.org/package/conduit-combinators)\nlibrary that provides a safer (when it come to releasing resources) and more\nconvenient way of streaming content. The importer reads and parses `batchSize`\nentries at a time, where `batchSize` is a parameter that must be configured when\ninvoking the importer to tell it how many entries should be parsed in one\n\"read \u0026 parse\" pass to be submitted to the database.\nWhen choosing a `batchSize` one should consider at least the following factors:\n- how much memory is available on the machine where the importer runs\n- how much memory is needed to represent a person (I wrote down details about\nthat in `Persons.hs:22`)\n- the optimal batch size for multi-row `INSERT`s in PostgreSQL\n\n### 3. Merging entries respecting the given invariants\n\nThe database is responsanble for merging entries supplied by the importer in a\ncorrect and efficient way. I explored various possibilities:\n* `SELECT`ing person records one-by-one and checking the invariants is hopelessly slow\n* conditional `UPDATE`s don't cover the case of new person records\n* `INSERT`ing using the `ON CONFLICT` clause looked promising since it's basically\nwhat the invariants describe. The only catch here is that the `conflict_target`\nmust be unique. That meant adding a `PRIMARY KEY` constraint to the persons table.\n\nUltimately I decided to implement the last option: when the database is created\nand populated the `person` table is `ALTER`ed to add a `PRIMARY KEY` but that lead\nto another issue: 257 person records are not well-formed. The \"Assumptions\" and\n\"Sanitising data\" sections talk about how I decided to tackle this.\nThe final SQL statement to merge entries looks like the following:\n\n``` SQL\nDECLARE\n  row_stats integer;\nBEGIN\n  INSERT INTO person AS p\n  SELECT * FROM json_populate_recordset(null::person,entries)\n  ON CONFLICT (fname,lname,dob) DO UPDATE\n  SET phone = EXCLUDED.phone\n  WHERE p.phone != EXCLUDED.phone OR p.phone IS null;\n\n  GET DIAGNOSTICS row_stats = ROW_COUNT;\n  RETURN json_build_object('row_stats',row_stats);\nEND;\n```\n\nIt uses `PL/pgSQL` to be able to return basic statistics about the affected\nperson records in the database back to the importer.\n\n### Profiling and benchmarks\n\nI run these benchmarks on my laptop: MacBook Pro (Retina, 15-inch, Mid 2015),\n2,5 GHz Intel Core i7, 16 GB 1600 MHz DDR3. First I did some general tests\nplaying around with different batch size to check how that impacted performance:\n\n* `batchSize=1000, max_wal_size=1GB` =\u003e 69.265649s (lots of connection errors:\npostgREST can't keep up apparently)\n* `batchSize=10000, max_wal_size=1GB` =\u003e 62.356193s (2-3 req/sec)\n* `batchSize=100000, max_wal_size=1GB` =\u003e 65.404345s (1 req/~5secs)\n\nLooking at PostgreSQL logs I noticed lots of warnings like\n`checkpoints are occurring too frequently (29 seconds apart)` so I configured\nPostgreSQL to use a bigger `max_wal_size`:\n\n* `batchSize=2000, max_wal_size=2GB` =\u003e 61.797248s (20+ req/sec)\n* `batchSize=5000, max_wal_size=2GB` =\u003e 64.429382s (4-5 req/sec)\n* `batchSize=10000, max_wal_size=2GB` =\u003e 61.140297s (2-3 req/sec)\n\nThe warnings were gone but the running time of the merge process wasn't\nchanged.\n\nChanging the batch size didn't seem to affect the total running time of the merge\nprocess. In order to make it run quicker I could think about two strategies:\n\n1. make the SQL in the database more efficient\n2. make the parsing/serialising in the parser more efficient\n\nLet's start with the database:\n1. sending the entries to the database: there is going to be some I/O involved,\nI tried tuning that by changing the batch size already and didn't really get\nanything out of it\n2. deserialisation of the input: this is internal to the database and\nit cannot be optimised as far as I know\n3. optimising the SQL statement: the statement is a database function that is a\nstored procedure. It's already compiled and optimised by the database query planner.\nThe index on the `PRIMARY KEY` slows `INSERT`ions down but it cannot be removed\nsince the constraint is needed for `ON CONFLICT` to work. Tuning some database\nsettings is also an option, I explored changing `max_wal_size` but didn't get any\nsubstantial perfomance gains\n4. hardware: this is out of the scope of the assignment\n5. serialisation of the output: the database return a minimal response and I don't\nsee this as a bottleneck\n\nI didn't walk this road too long because from my tests it didn't seem like the\ndatabase was the bottleneck. So I moved my attention to the importer with the\ngoal of raising the number of requests per second that it could send to the\ndatabase focusing on:\n1. deserialisation of the entries in the XML input file\n2. serialisation of the entries into JSON\n3. deserialisation of the database response\n\nHere things got more interesting: while profiling the importer to optimise its\nrunning time I actually found a space leak that prevented it from running in\nconstant memory and fixed it. The following image shows how memory consumption\ngrows linearly with the number of entries (50000 in this test)\n\n![Importer taking linear memory](images/importer-linearmem.png)\n\nand the following shows how the memory consumption stays mostly constant\n\n![Importer taking constant memory](images/importer-constantmem.png)\n\nWhen testing the space leak-free version of the importer I noticed that the merge\nprocess running time increased dramatically. It turned out the importer wasn't\nrunning merge jobs fully in parallel, that is it was waiting for a merge job to complete\nbefore merging the next one. The way I fixed it it's also described in the commit\nmessage whose SHA-1 is 333086d.\n\n## General assumptions\n- The database is hosted on a remote machine therefore the importer must connect\nto it over the internet.\n- Creating the persons table is a one-time task and the one-time costs to pay\nupfront to sanitise the person records (see below) so that a `PRIMARY KEY`\nconstraint can be created are acceptable costs. A `PRIMARY KEY` constraint is added\nbecause is generally a good practice to have it and allows to implement the merge\nprocess conveniently and efficiently.\n- Removing the 267 entries that do not have either a last name or a birthdate is\nan unacceptable data loss.\n\n## Sanitising data\nThere are some not well-formed entries in the XML input file, i.e.:\n`\"ABD AL\", \"PIKULSKI\", \"\\\\N\", \"3158419207\"`.\nI decided to simply sanitise it this way:\n`\"ABD AL\", \"PIKULSKI\", \"-infinity\", \"3158419207\"`.\nOther options I considered where:\n\n- filtering out the entry\n- somehow reporting the entries that are not well-formed\n- rejecting the whole file\n\nThe focus of the assignment wasn't on this particular aspect and 267 person records\nare also not well-formed, I applied the easiest solution I could think of.\n\n### Default values\n- `dob`: `-infinity`\n- `lname`, `fname`: `_` (in Haskell `_` means: \"ignore this\")\n\n## Solutions exploration\n\nThis is a short analysis of the various solutions I pondered before implementing\nthe final one\n\n### 1. Reading the whole XML file\n\n#### High-level algorithm description\n\n- parse all entries in XML input file\n- `SELECT `all person records\n- for each person record check if their telephone number is different from the\none in the XML file: if so, `UPDATE` the person record otherwise do nothing\n- `INSERT` all remaining entries\n\n#### Performance evaluation\n\nThe algorithm needs O(P) memory, it would require O(1) `SELECT` and O(P) `UPDATE`s /\n`INSERT`s where P is the number of entries in the XML input file.\n\nThe memory requirement are undesirable, the ideal solution would require\nconstant memory\n\n### 2. Streamig the XML file\n\n#### High-level algorithm description\n\n- parse one entry in the XML input file\n- `SELECT` the person record corresponding to the entry from the database\n- if the result is empty `INSERT` the entry otherwise check if the phone\nnumber of the person record is different from the one of the wentry: if so,\n`UPDATE` the person record otherwise do nothing\n- repeat for all entrie in the XML input file\n\n#### Performance evaluation\n\nThe algorithm needs O(1) memory but would require O(P) `SELECT`s and O(P) `UPDATE`s /\n`INSERTION`s where P is the number of entries in the XML input file.\n\nConstant memory usage is good but O(P) `SELECT`s are required. That's slow and\nputs too much load on the database.\n\n### 3. Batching database operations\n\n#### High-level algorithm description\n\n- parse `batchSize` entries in the XML input file (`batchSize` might depend on\ndifferent factors, i.e. memory available)\n- `SELECT` `batchSize` person records\n- for each person record check if their telephone number is different from the\none in the XML file: if so, `UPDATE` the person record otherwise do nothing\n- `INSERT` all remaining entries\n- repeat for all entris in the XML input file\n\n#### Performance evaluation\n\nThe algorithm needs O(1) memory and it would require O(P / M) `SELECT`s and O(P)\n`UPDATE`s / `INSERTION`s where P is the number of entries in the XML input file\nand `batchSize` is the fixed batch size - this might depend on different factors,\ni.e. memory available.\n\nConstant memory usage is good but again O(P / N) `SELECT`s are required. That's\nbetter than the previous algorithm but still potentially slow. Is it possible to\navoid this step completely?\n\n\n### 4. Batch-upserting\n\n#### Prerequisites\n\n- `UPDATE` the persons table and add default `lname`s and `dob`s where missing\nin order to be able to add a primary key constraint\n- `ALTER` the persons table and add a `PRIMARY KEY` constraint needed by the\n`ON CONFLICT` statement.\n\n#### High-level algorithm description\n\n- parse `batchSize` entries from XML input file (`batchSize` might depend on\ndifferent factors, i.e. memory available)\n- `UPSERT` `batchSize` entries leveraging the database own merge capabilities\n- repeat for all entries in the XML input file\n\nMerging would look like this in SQL:\n``` sql\nINSERT INTO person AS p VALUES\n('JIARA','HERTZEL','1935-06-05','5859012134'),\n('RONJARVIOU','COMELLO','1932-09-27','7702713416')\nON CONFLICT (fname,lname,dob) DO UPDATE\nSET phone = EXCLUDED.phone\nWHERE p.phone != EXCLUDED.phone OR p.phone IS null;\n```\n\n#### Performance evaluation\n\nThe algorithm needs O(1) memory and O(P) `UPSERT`ions where P is the number of\npersons in the XML file.\n\n##### PROs\n\n- client runs in constant memory\n- leverage native merging capabilities of the database to:\n  - simplify the importer logic\n  - avoid the need to do any (slow) I/O to `SELECT` person records first to apply\n  merging in the client\n  - achieve good separation of concerns:\n    - importer parses the XML input file and sends the entries to the database\n    - database handles the merge logic\n\n##### CONs\n\n- there is a one-time cost to pay to sanitise the data in the persons table\n- there is a one-time cost to pay to add a `PRIMARY KEY` constraint in the\npersons table\n\n\n## Not implemented\n\nThe following problems are not solved in satisfactory fashion, mostly because of\ntime constraints but I wanted to mention them since I did ponder them.\n\n### Backpressure\n\nThe importer doesn't have any short-circuiting when it come to temporarly halting\nor slowing down the merge process. This might overwhelm the database in case for\nsome reason it cannot keep up.\n\n### Error handling\n\nIf the database is down or the is any other issue that prevents the request from\nreaching the database, the importer should either store the failed entries - i.e.\nin an `update-failed.xml` file - to retry later or be manually re-submitted.\n\n### Tests\n- I'm not entirely sure how merge can be tested in an automated fashion.\nThat would include spinning up the infrastructure that holds the database and\ninvoke its `merge` function and then check the returned stats and query the database.\nI did do some manual tests to assure the correctness of the `merge` function.\n- Fuzzy testing on parsing / serialising entries in the importer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffuttetennista%2Flambdawerk-backend-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffuttetennista%2Flambdawerk-backend-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffuttetennista%2Flambdawerk-backend-test/lists"}