{"id":13603318,"url":"https://github.com/yesco/jml","last_synced_at":"2025-04-11T19:31:57.257Z","repository":{"id":148990845,"uuid":"55430096","full_name":"yesco/jml","owner":"yesco","description":"jml - a useful web/cloud minimal unikernel distributed operating system","archived":false,"fork":false,"pushed_at":"2024-05-22T21:11:05.000Z","size":628,"stargazers_count":12,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-22T21:30:16.918Z","etag":null,"topics":["eval","functional","functional-language","html","lambda-calculus","language","operating-system","unikernel","urbit","web","webserver"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yesco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-04T17:16:18.000Z","updated_at":"2024-08-01T19:37:24.651Z","dependencies_parsed_at":"2024-05-19T16:29:34.015Z","dependency_job_id":"1b103bf6-c8ce-4767-a486-bb0595131594","html_url":"https://github.com/yesco/jml","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yesco%2Fjml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yesco%2Fjml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yesco%2Fjml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yesco%2Fjml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yesco","download_url":"https://codeload.github.com/yesco/jml/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223476773,"owners_count":17151515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eval","functional","functional-language","html","lambda-calculus","language","operating-system","unikernel","urbit","web","webserver"],"created_at":"2024-08-01T19:00:17.940Z","updated_at":"2024-11-07T07:30:56.963Z","avatar_url":"https://github.com/yesco.png","language":"C","funding_links":[],"categories":["Projects"],"sub_categories":["Generic"],"readme":"# jml - a useful web/cloud minimal unikernel distributed operating system\n\nTODO: rename to [MagniOS](https://en.wikipedia.org/wiki/M%C3%B3%C3%B0i_and_Magni)\n\n## Goal\n\nSimple online-programmable virtual physical computer with built in\npersistent \"memory\" extending the IoT to simple internet/cloud\ncomputers, that are disposable scalable online reconfigurable minimal\nprocessing instances. This is an instance of a \"single (minimalish)\ncomputer\".\n\nBut as we know the future is distributed and we need to provide\ninfrastructure \"above a single computer\". This system provides such\nservices implemented utilizing the said single computers.\n\n## Woa!?\n\nSo crassly seen it's a html-generating programmable virtual computer\nwith high-level lambda style \"instructions\". As we know, lambda\ncalculus is \"described\" by subsitution, here we take the approach that\nactual in-place substitution constitutes a viable computer. We provide\na minimal of basic services without getting stuck in the trap of\ngenerality of an operating system. All the services/functions are\n\"not\" mapped to or dependent on a UNIX computer as such. Therefore, it\ncan be hosted on containerized physical hardware SOAC (System On A\nChip) like the ESP-8266.\n\n### Services provided\n\n(Things in parenthesis in progress..)\n- Lambda style computational engine\n- (Reliable) Messaging to remote entity\n- Dispatch of incoming messages to user define API\n- Persistent local event data store\n- Online extensible program\n- Encryption secured remote execution, and program upgrade\n- (Transferable state, movability)\n- CHORD-style distributed storage and addressing\n- (distributed service directory)\n- (distributed files and directories)\n\nCurrent limitations:\n- messaging currently limited to \"REST\" calls, could/should use UDP for small messages, otherwise TCP/IP\n- messaging not reliable/transactionable\n- persistence not atomic per triggering event/message\n\nIt's not clear what would be missing as a minimized viable portable\ncomputing instance. If it's hardware connected to physical sensors\nmeasurement naturally it would be limited to such capable instances,\noften even to specific locations. Other functionality is portable, and\ntransferable.\n\nHandy features:\n- built in / handler\n- built in mini-wiki!\n- built in reflection\n- built in display of public /api handler and generating form/buttons for easy interaction\n- simple portable symmetric encryption (XXTEA)\n\n## First Principles\n\nThis is a [First Principle's](https://en.wikipedia.org/wiki/First_principle) project.\nIt means it starts from \"scratch\" with minimal requirements. Elon Musk\nuses this as his [innovation\nprinciple](http://99u.com/workbook/20482/how-elon-musk-thinks-the-first-principles-method).\n\nFor example: \"In physics, a calculation is said to be from first\nprinciples, or ab initio, if it starts directly at the level of\nestablished laws of physics and does not make assumptions such as\nempirical model and fitting parameters.\" - wikipedia\n\n## Requirements\n\nC (gcc) compiler, and simple unix style standard library calls.\nFor \"security\" no external libraries are used.\n\n[HowTo Compile a 32-bit Application Using gcc On the 64-bit Linux Version](https://www.cyberciti.biz/tips/compile-32bit-application-using-gcc-64-bit-linux.html)\n\n## Why\n\nThe idea of a minimal OS is coming up again, maybe recently more (now\nis 2016) than the last 30 years. We have docker, and various\nvirtualization environments. It's funny to simulate an actual once\nexisting hardware computer - mostly just a IBM PC compatible. However,\nthere are other approaches, more minimal which are single language\nbased. These are sometimes called a library OS implementations. For\nexample RTOS for ESP8266\n[esp-open-rtos](https://github.com/SuperHouse/esp-open-rtos) can be\nseen as such. It provides a small implementation of most of the\nlibraries and system calls you'd expect on a POSIX/linux style\ncomputer. It is limited to a single process, so no fork/popen or\nsystem cals. It does provide it's own threading/tasks and\nsemaphore/messaging system. However, it doesn't provide infrastructure\nto think \"above a single computer\".\n\n## What is it?\n\nThis project is an attempt to provide a small relatively simple C\nprogram implementing a *portable small language* onto such an\ninvironment. Of course, for more complicated tasks a multitude of\nsuch distributed instances would be able to cooperate in order to solve\na bigger problem.\n\n## Status\n\nStatus: More \"useless\" than [Urbit](http://urbit.org/)!\n\n## Outline of actions/TODO/DONE\n\n- DONE: small simple small language interpreter in C\n- NONMINAL: what's the minimal set of functions needed?\n- connectivity, receive requests/send requests (messaging)\n- logical + physical addressing something like [Kademlia](https://en.wikipedia.org/wiki/Kademlia)\n- callback verification [Magic cookie](https://en.wikipedia.org/wiki/Magic_cookie)\n- https://en.wikipedia.org/wiki/Merkle_tree\n- cheap hashes https://en.wikipedia.org/wiki/Tiger_(cryptography)\n\n## Extentions?\n\nIt's not mean to be extended on the C-level. However, some way of\naccessing hardware/sensors is needed.  Possibly, the most portable way\nto approach this is to provide the service as a minimal\nwebrequest/mqtt interface. This would allow the sensors to be directly\naccessible, if secure, on external interfaces too,\ntransparently. Indistinguishable from other mqtt/webservices devices.\nThe \"strict\" goal is to keep all the code/native functions, *exactly*\nthe *same* on all devices. In princple, after a while, we should be\nable to freeze them, and only work on extension on the soft-layer.\n\n## How to run\n\n### command line interactive\n\n    ./run\n\n### batch\n\nThis allows a single command, or several to be run and output captured.\n\n    echo \"[concat a b c]\" | ./run 2\u003e/dev/null\n\n### web\n\nTo run it as a webserve on port 1111, or as given use the command below.\nConnect to it as [localhost:1111/](localhost:1111/).\n\n    ./run -w [PORT]\n\n### simulated group of servers\n\n    ./start-servers\n\n### debugging using tracing\n\n    unix\u003e echo \"[+ [map inc 1 2 3]]\" | ./run -t\n    \u003e\u003e\u003e[+ [map inc 1 2 3]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[+ [inc 1] [inc 2] [inc 3] ]\u003c\u003c\u003c\n    \u003e\u003e\u003e[+ 2 3 4 ]\u003c\u003c\u003c\n    \u003e\u003e\u003e9\u003c\u003c\u003c\n    9\n    \n\n    unix\u003e echo '[macro fac $n][* [iota 1 $n]][/macro]' | ./run\n      [appended fac to file]\n    \n    unix\u003e echo \"[fac 5]\" | ./run -t\n    \u003e\u003e\u003e[fac 5]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* [iota 1 5]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 1 2 3 4 5 ]\u003c\u003c\u003c\n    \u003e\u003e\u003e120\u003c\u003c\u003c\n    120\n\n### options for performance/info\n\n    -q       == quiet mode, only writes %%error messages\n    -w       == normal mode, start webserver\n    -v       == log timing and reductions info\n    -v -v    == log reallocs\n    -v -v -v == log allocs too\n\n# Language\n\nThe language is a bastard variant of lambda calculus where the\nevalution is eager, and evaluation is performed by plain textual\nsubstitution. There are no variables as such, no stack, no closure, no\ncircular data structures, no data structures. Semantics of our\nlangauges is specified by legal substitutions. We find any \"innermost\"\nexpression [FUN ARGS...] where FUN and ARGS do not contain any other\nexpression (i.e. '[' or ']'). These are directly substitutable. Any\nnumber of such expressions may be substituted in any order as\ndetermined by the evaluator. Keep it functional, and it's safe! For\nachieving specific order of substitutions, create a \"data dependency\"\nwhere the dependent calculations wraps the calculation that are needed\nto be performed before the dependent calculation.\n\n## simple string based evaluation\n\n    foo =\u003e foo\n    foo [+ 3 4] bar =\u003e foo 7 bar\n    foo [upper bar] fie =\u003e foo BAR fie\n    [[concat up per] fie] = FIE\n\nWhat is an if-statement, or any choice of execution path, if not just\nthat: a choice of execution path!  In this language, we provide an\n'if' that will return which named function to invoke on the rest of\nthe data;\n\n    [if 0 1 2] =\u003e 2\n    [if 1 1 2] =\u003e 1\n    [if [\u003c 3 4] smaller bigger] =\u003e smaller\n\nFrom this it can be seen that it is NOT a special form or have any\nspecific rules, it only provides a choice of functions to invoke \non given arguments.\n\nFor example, to implement fac recursively:\n\n    [macro fac $n][* $n [[if [\u003c= $n 1] ignore fac] [- $n 1]]][/macro]\n\n    \u003e [fac 6]\n    \u003e\u003e\u003e[fac 6]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [[if [\u003c= 6 1] ignore fac] [- 6 1]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [[if 0 ignore fac] 5]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [fac 5]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [[if [\u003c= 5 1] ignore fac] [- 5 1]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [[if 0 ignore fac] 4]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [fac 4]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [[if [\u003c= 4 1] ignore fac] [- 4 1]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [[if 0 ignore fac] 3]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [fac 3]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [[if [\u003c= 3 1] ignore fac] [- 3 1]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [[if 0 ignore fac] 2]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [fac 2]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [[if [\u003c= 2 1] ignore fac] [- 2 1]]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [[if 0 ignore fac] 1]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [fac 1]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [* 1 [[if [\u003c= 1 1] ignore fac] [- 1 1]]]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [* 1 [[if 1 ignore fac] 0]]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [* 1 [ignore 0]]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 [* 1 ]]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 [* 2 1]]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 [* 3 2]]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 [* 4 6]]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 [* 5 24]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 6 120]\u003c\u003c\u003c\n    \u003e\u003e\u003e720\u003c\u003c\u003c\n    720\n\nObviously a more efficient implementation is:\n\n    [macro fac $n][* [iota 1 $n]][/macro]\n\n    \u003e [fac 6]\n    \u003e\u003e\u003e[fac 6]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* [iota 1 6]]\u003c\u003c\u003c\n    \u003e\u003e\u003e[* 1 2 3 4 5 6 ]\u003c\u003c\u003c\n    \u003e\u003e\u003e720\u003c\u003c\u003c\n    720\n\nIt can be used and misunderstood easily, the world be dammned!\n\n    [macro foo]11 22[/macro]\n    [macro bar]2 1[/macro]\n\n    [if 0 [foo] [bar]] =\u003e 22\n\n    ???\n\n    [if 0 [foo] [bar]] =\u003e\n    [if 0 11 22 [bar]] =\u003e\n    [if 0 11 22 2 1]] =\u003e\n    22 !!! second argument returned\n\nHow to do IFFFF THEN?\n\n    [[if 0 foo bar]] =\u003e 2 1\n    [[if 1 foo bar]] =\u003e 11 22\n\nNow, gettin into the gritty, this can be used to implement choice/switch/case-statements!\n\n    [macro en-0]zero[/macro]\n    [macro en-1]one[/macro]\n    [macro en-2]two[/macro]\n    [macro en-3]three[/macro]\n    [macro en-4]four[/macro]\n    [macro en-5]five[/macro]\n    [macro en-6]six[/macro]\n    [macro en-7]seven[/macro]\n    [macro en-8]eight[/macro]\n    [macro en-9]nine[/macro]\n    [macro en-10]ten[/macro]\n    [macro en-][/macro]\n    \n    [macro en $digit][en-$digit] [/macro]\n\n    [macro english @digits][map en @digits][/macro]\n\n    [macro english-rec $digit @digits][en-$digit] [[if [empty @digits] ignore english-rec] @digits][/macro]\n\n## help\n\nfrom command line:\n\n    [help]\n\nfrom web browser\n\n    unix\u003e ./jml -w\n\nGoto:\n\n    http:localhost:1111/help\n\n## super eager evaluation\n\nThe interpreter is very simple: it just eagerly evaluates any inner\n[fun params...] expression where fun and params themselves have no\nfunction calls. I.e., the innermost [ ] expressions are replaced\nsuccessively, the result is achieved by replacing the expression with\nthe body where the formal parameter names have been substituted by the\nactual values. Parameters are space delimited.\n\n## efficiency\n\nBah... computers are fast! Seriously, we're generating HTML and serving\nit on the net, so what's fast anyway?\n\n## features\n\n- active messaging\n- reactive, single user space, no address sharing\n- no processes, or tasks, single threaded\n- persistent data/program/functions once \"uploaded\"\n- simple evaluation mechanism\n- http communication (TODO)\n\n## built-in functions\n\nMath\n- % inc dec \u003e \u003c \u003e= \u003c= = !=\n- [+ 1 2 3 4] =\u003e 10\n- [* 1 2 3 4] =\u003e 24\n- [iota 1 3] =\u003e 1 2 3\n- [iota 0 10 2] \n\nLogic\n- [and 1 1 ...] =\u003e 1\n- [and 1 0 ...] =\u003e 0\n- [not 0] =\u003e 1\n- [not 7] =\u003e 0\n- [xor 0 0] = [xor 1 1] =\u003e 0\n- [xor 0 1] = [xor 1 0] =\u003e 1\n\n    val  X: empty  has  is  not number alpha\n    ========================================\n    [X   ]    1     0    0    1    0     0\n    [X  0]    0     1    0    1    1     0 \n    [X  1]    0     1    1    0    1     0\n    [X  a]    0     1    1    0    0     1\n\nTest\n- [if 0 THIS not] =\u003e THIS\n- [if X not THAT] =\u003e THAT\n- [empty] =\u003e1\n- [empty ...] =\u003e 0\n- [empty    ] =\u003e 1\n\n- [equal A B] =\u003e 0\n- [cmp A B] =\u003e 1\n- [cmp B A] =\u003e -1\n- [cmp A A] =\u003e 0\n- lower upper\n\n- [length] =\u003e 0\n- [length 1 2 3] =\u003e 3\n- [length 11 22 33 44] =\u003e 4\n\n- [bytes] =\u003e 0\n- [bytes 1 2 3] =\u003e 5\n- [bytes 1 2 3 ] =\u003e 6\n- [bytes  1 2 3 ] =\u003e 7\n\nLists\n- [ignore X] =\u003e\n- [identity X ...] =\u003e X ...\n\n- [map F a b c ...] =\u003e [F a] [F b] [F c] [map F ...]\n- [filter P a b c ...] =\u003e [[if [P a] identity ignore] a] ...\n- [filter-do P F a b c ...] =\u003e [[if [P a] F ignore] a] ...\n\n- [nth 2 11 22 33] =\u003e 22\n- [first A B] =\u003e A\n- [second A B C] =\u003e B\n- [third A B C D] =\u003e C\n- [rest A B C D] =\u003e B C D\n\nStrings\n- [after X abcXzy] =\u003e zy\n- [before X abcXzy] = abc\n- [prefix abcdef abc abcd ab] =\u003e ab\n- [split a aAaBBaAa] =\u003e A BB A\n- [split-do inc a1a22a3a] =\u003e 2 23 4\n- [xml name ksajf; sadflk dsaflk \u0026lt;name c='foo'\u0026gt;FISH\u0026lt;/name\u0026gt; sdfl sadf asdfdsa] =\u003e FISH\n\n- [substr FIRST LEN abcXzy] get substrings out\n- [substr 0 3 abcXzy] =\u003e abc, this is essentially \"left\"\n- [substr 3 2 abcXzy] =\u003e Xz, this is essentially \"mid\"\n- [substr 4 1 abcXzy] =\u003e z\n- [substr -2 -2 abcXzy] =\u003e zy get the last N character, this is \"right\"\n\n- [concat A B C D ...] =\u003e ABCD...\n- [concat A\\ B C D ...] =\u003e A BCD...\n- [concat [concat a\\ b c]] =\u003e A BCD...\n\n- [match a(b*)(cd*)e(.*)f abbbcexxxxxfff] =\u003e  bbb c xxxxx\n- [match-do F a(b*)(cd*)e(.*)f abbbcexxxxxfff] =\u003e [F bbb c xxxxx]\n- [match-do F \u003c(a*)\u003e aa\u003caaa\u003e\u003caaaa\u003eaa\u003ca\u003efish\u003ca\u003ex] =\u003e [F aaa] [F aaaa] [F a] [f a]\n- [subst-do F \u003c(a*)\u003e aa\u003caaa\u003e\u003caaaa\u003eaa\u003ca\u003efish\u003ca\u003ex] =\u003e aa[F aaa][F aaaa]aa[F a]fish[f a]x\n\nWEB, decode URL\n- [decode foo+bar%2b%25] =\u003e foo bar+%\n- [wget URL] =\u003e BODY (quoted for safety)\n- [eval/FUN1/Fun2 ...] - see below!\n\nStorage/persistent/database\n- [data firstname Peter] .. [data-firstname] =\u003e Peter\n- [datas] =? firstname ...\n- [funcs] =\u003e user defined func names\n- [funcs prefix] =\u003e user defined func names starting with prefix\n- [funcs start end] =\u003e user defined func names in [start, end[\n- [fargs macroname] -\u003e $a $b @foo\n- [fbody macroname] -\u003e foo $a fie $b fum: @foo\n\nFailure\n- [xyz sadfasdf] =\u003e [FAIL xyz sadfasdf]\n- [FAIL $id @params] =\u003e %(FAIL:xyz sadfasdf) - you can override this!\n\n### Content Addressable Network\n\nWe also add the capability of (CAN)[https://en.wikipedia.org/wiki/Content_addressable_network].\n\nFor now we just implement Content Hashing by using the encrypt\nfunction with a fixed non-secret key.  It may not qualify as\ncryptographically secure hashing function, but serves the purpose of a\ndecent hash function, as it's already have it...\n\nTODO: ...\n\n#### Node Joining\nA joining node must:\n- Find a node already in the overlay network\n- Identify a zone that can be split\n- Update the routing tables of nodes neighbouring the newly split node\n\n#### Node Departing\nTo handle node departing:\n- Identify a node departing\n- have node's zone merged or taken over by a neighbouring node\n- update the routing tables across the network\n\nheartbeat to neighbours\n\n- TODO: Consider (implementing) mDNS\n- TODO: Consider if we should be using https://en.wikipedia.org/wiki/Kademlia\n- p2p filesystem w some redundancy - [PAST](https://en.wikipedia.org/wiki/Pastry_(DHT))\n- p2p pubsub - [SCRIBE](https://en.wikipedia.org/wiki/Pastry_(DHT))\n- http://stackoverflow.com/questions/3076222/top-hashing-and-encryption-algorithms\n\n#### hotspots\n\nTODO: maybe we're doing (implementing) this???\n\nWhen searching for NAMED data, we should search for:\n\n    [ NAME-HASH    '/' YOUR-ID-HASH '/' ]\n    [ CHUNK-HASH   '/' YOUR-ID-HASH '/' ]\n\nFor example when a file name is hashed for filesystem implementations there may be many\nhosters/locations storing the same file and registering the same file, in order not to\noverload a single \"bucket\" we need to be able to split on a \"longer\" key so, key for\ninsertions is:\n\n    [ NAME-HASH    '/' HOSTER-ID-HASH '/' LIST-HASH ':name' ]  =\u003e NAME\n    [ LIST-HASH    '/' HOSTER-ID-HASH ':file']                 =\u003e LIST of (offset, CHUNK)\n    [ CHUNK-HASH   '/' HOSTER-ID-HASH ':chunk']                =\u003e CONTENT\n\nThis way the normal splitting would be able to handle hotspots by just splitting the range.\nWhen searching you identify yourself and will be routed to the bucket for the same CONTENT-HASH\nthat is closest to your ID.\n\nExample:\n\nfile content hash\n    \u003e echo \"[content-hash \u003cdata...tatatatatat\u003e]\" | ./run -q\n    0A543F0365179C05\n\n    \u003e echo \"[content-hash \u003cdata...other\u003e]\" | ./run -q\n    C0458858B721576B\n\nfile name component \"StarWarsIV\" probably also will have \"star\" \"wars\" \"iv\"...\n    \u003e echo \"[content-hash StarWarsIV]\" | ./run -q -t\n    A2065D7267A72D3D\n\n    star =\u003e 015BA7A2250FA4D1\n    wars =\u003e 6181303D2E3EEB6C\n    iv   =\u003e 63AB646145DFFEA9\n\nanother file is \"star trek\"\n\n    star =\u003e 015BA7A2250FA4D1\n    trek =\u003e EC8DA41B66A78863\n\nthis is your node ID:\n\n    \u003e uuid\n    5aaf5bb2-aca3-11e6-b0d5-cb6044163794\n    \u003e echo \"[content-hash 5aaf5bb2-aca3-11e6-b0d5-cb6044163794]\" | ./run -q -t\n    DA623E175D65C3C0\n\nwe have 2 more nodes storing this file\n    \n    64CDE172E66BA52D\n    FD017067ECF5E177\n\nthe second file is stored by you and another person:\n\n    4B3722E05881C728\n\n3 nodes have the file thus we will store 3 pointers in the extended key\nhere is the full table\n\n    015BA7A2250FA4D1/4B3722E05881C728/C0458858B721576B =\u003e star\n    015BA7A2250FA4D1/64CDE172E66BA52D/0A543F0365179C05 =\u003e star\n    015BA7A2250FA4D1/DA623E175D65C3C0/0A543F0365179C05 =\u003e star\n    015BA7A2250FA4D1/DA623E175D65C3C0/C0458858B721576B =\u003e star\n    015BA7A2250FA4D1/FD017067ECF5E177/0A543F0365179C05 =\u003e star\n\n    6181303D2E3EEB6C/64CDE172E66BA52D/0A543F0365179C05 =\u003e wars\n    6181303D2E3EEB6C/DA623E175D65C3C0/0A543F0365179C05 =\u003e wars\n    6181303D2E3EEB6C/FD017067ECF5E177/0A543F0365179C05 =\u003e wars\n\n    A2065D7267A72D3D/64CDE172E66BA52D/0A543F0365179C05 =\u003e StarWarsIV\n    A2065D7267A72D3D/DA623E175D65C3C0/0A543F0365179C05 =\u003e StarWarsIV\n    A2065D7267A72D3D/FD017067ECF5E177/0A543F0365179C05 =\u003e StarWarsIV\n\n    EC8DA41B66A78863/4B3722E05881C728/C0458858B721576B =\u003e trek\n    EC8DA41B66A78863/DA623E175D65C3C0/C0458858B721576B =\u003e trek\n\n#### example commands\n\n    echo \"[wget [route-data [content-hash Hello. This is a message!]]/id]\" | ./run -q -t\n\nThis hashes by content, finds the route, connects to that server, retrieves /id URL\nthat should contain its hash.\n\n#### We need a distributed filesystem\n\nsee above?\n\nTwo variants\n- CA, Content Addressable, hash the content, store on the node as owner, and \"k nearest\".\n- by URL name, like PAST: hash by URL to find (date,size,CA-hashes)\n\n### Note on quoting\n\nSince everything works by substitution (close to lambda calculus?), the need for\nquoting is low. This is normally called interpolation in languages as Perl. In JML\nit's a little different as the interpolated string is continously interpolated by\nsubstitutions of [fun ...] \"calls\". However, as it's uses '[', ']', and ' ' as significant\nthese now instead need to be quoted. LOL. However, for the domain of email. This is \"convenient\".\n\nThis is how to quote these characters\n- A\\ B is interpreted as one parameter in calling, see concat above\n- \\[+ 3 4\\] will print [+ 3 4] and not 7\n\n#### Safety, \"SQL Injection\"\n\nIn order to not allow injection, certain \"web\" characters should be\nquoted on \"input\" from the web or external sources. Examples include:\n\u0026lt; \u0026gt; \\[ \\] \u0026amp; ' \"\n\n### eval/fun1/fun2/fun3...\n\nHaving an eval is flexible, however, a big security risk. We take the\nmiddle-ground and provide a \"sandboxed\" eval. When eval is invoked it\nalso requires an enumeration of all functions that are allowed to be\ncalled.  Essentially this defines an API. For example to define a\ncalculator with only plus minus times divide you specify\n[eval/plus/minus/times/divide ...]  Eval will change then change each\noccurance of {FUN...} to [FUN...] if fun is listed. If {} doesn't\nbalance, empty string is returned, same if unlisted FUN is\nmentioned. This doesn't limit further eval/substituations.\n\nThis, for example, can be used to implement a forward request in\nCHORD/DHT by having the server return updates as well as a new\nrequest in case the route is wrong. If the routing is right\nthe contacted server would return it's own ID. Otherwise it\nreturns a \"textual-continuation\" (program with data filled in),\nwhich coincidentially is exactly what a jml-program is!\n\n    [route-resolve $HASH]\n\nIf this doesn't resolve to the server you're at, it'll wget call that server and\nit will either confirm, or return \"forwarding instructions\". Essentially,\nsome updates (route-add) and then a new route-resolve that depends on those updates.\n\n    {route-resolve $HASH {route-add 0375F9AB200504E9 2016-... http:...} {route-add DB348B3005A37278 2016-... http:...}}\n\nThese are performed at/by the originating server, which will use:\n\n    [eval/route-resolve/route-add ...]\n\nto evaluate, if it again doesn't resolve locally, we'll get another\nforward and the new remote server will conform that it is the\nreceipient by returning it's ID:\n\n    $HOST_ID\n    \n#### Unicode\n\nTODO: Haha, come again? Don't you know the world consists of bits and bytes?\n\nSeriously, the web is multilingular, and I travel in china, use swedish so UTF-8 is a reasonable requirement.\n\n[UTF-8](https://en.wikipedia.org/wiki/UTF-8):\n\n\"And ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, making UTF-8 safe to use within most programming and document languages that interpret certain ASCII characters in a special way, e.g. as end of string.\"\n\n### Security/Encryption/(TEA)[https://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm]\n\n- [encrypt FOOBAR] =\u003e {39CE0CD92EEBACC8}\n- [decrypt {39CE0CD92EEBACC8}] =\u003e FOOBAR\n- [encrypt-eval {+ 3 4}] =\u003e {D9403DC570A74AF1}\n- [decrypt {D9403DC570A74AF1}] =\u003e {+ 3 4}\n- [decrypt-eval {D9403DC570A74AF1}] =\u003e [+ 3 4] =\u003e 7\n\n#### using custom keys\n\nThe default key is \"1234123412341234\", maximum lenght is 16, ascii, no\n\"/\" or \" \" character allowed.  The key is part of the \"function\" name\n(funny hack) thus the name to use is [encrypt/MYSECURITYKEY data to\nencrypt], similarly for decrypt/MYSECURITYKEY,\nencrypt-eval/decrypt-eval...\n\n*Note* on \"security\", it's just XXTEA which can only provide nominal\nsecurity in access. They keys aren't protected and if one has access\nto the physical device would be able to extract the key. However, each\ninstance should have it's own key and the person communicating with it\nneed it too.  This means; It's protected against middle-man\nattack. Each unikernel/device should have it's own key. It may be\npossible to use the key to only allow the owner of that key to modify\nand run any code. Others would only be allowed to interact using the\npublic defined macro functions, like [macro /foo]...\n\nEncrypted data becomes HEX coded, to keep it ascii, thus it will at\nuse at least the double amount of bytes.\n\nTODO: enable setting/changing keys\n\n#### Termination\n\nCurrently, if you loop forever by recursion there is no limit. One could add\na max-substitution count, or time that would terminate the current running request.\n\n### Alternative Universe Inspired Readings \n\n- (unikernel listings)[http://unikernel.org/projects/]\n  TODO: add myself?\n  \n- [immutable operating system](https://hn.algolia.com/story/7166173/an-immutable-operating-system?query=\u0026sort=byPopularity\u0026prefix\u0026page=0\u0026dateRange=all\u0026type=all)\n\n- HackerNews: [Alan Kay has agreed to do an AMA today](https://news.ycombinator.com/item?id=11939851)\n  This is an amazing discussion. \n  \n- https://en.wikipedia.org/wiki/Amorphous_computing\n\n- Phatom_OS - a managed code on object level rather than process level\n  (no pointer arithmetic). Cheap IPC. Persistence at core, live forever, no file system needed,\n  (link)[https://en.wikipedia.org/wiki/Phantom_OS]\n\n- Active Messaging, where is a messaging passing system where messages\n  not only contain data but also the name of a userspace handler to be\n  executed upon arrival (i.e. a function call!), sometimes it can\n  carry the actual code in the message...  (active\n  messaging)[https://en.wikipedia.org/wiki/Active_message]\n\n- Urbit OS; clean slate OS, packet protocol, pure functional language, deterministic OS, ACID database,\n  versioned filesystem, a webserver, global PKI, runs a frozen combinator interpreter, opaque computing\n  and communcation layer ontop of unix and internet. The user owns his own general purpose server or \"planet\".\n  Repeatable computing. \n  (link)[http://urbit.org/docs/theory/whitepaper]\n\n- http://vpri.org/html/work/ifnct.htm\n\n- http://www.otonomos.com/\n\n### Readings about x86 style OSes\n\n- how to build and bootstrap your own os (on PC arch) [link](http://www.brokenthorn.com/Resources/)\n- http://www.menuetos.net/index.htm\n- http://www.jamesmolloy.co.uk/tutorial_html/index.html\n- http://www.osdever.net/FreeVGA/home.htm\n- https://github.com/ReturnInfinity/BareMetal-OS\n- http://wiki.osdev.org/Main_Page\n- http://home.in.tum.de/~hauffa/slides.pdf\n- crochet [squeak/3d collaboration space](http://worrydream.com/refs/Smith%20-%20Croquet%20-%20A%20Collaboration%20System%20Architecture.pdf)\n\n## Urbit summary (and features I'd like have?)\n\n[urbit intro by third party](http://alexkrupp.typepad.com/sensemaking/2013/12/a-brief-introduction-to-urbit.html)\n\n- decentralized computing platform\n- clean-slate OS\n- personal server\n- persistent virtual computer\n- you own it, trust and control\n- \"Urbit is perhaps how you'd do computing in a post-singularity world, where computational speed and bandwidth are infinite, and what's valuable is security, trust, creativity, and collaboration. It's essentially a combination of a programming language, OS, virtual machine, social network, and digital identity platform.\"\n- UDP\n- reputation based social network (?)\n- event logging\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyesco%2Fjml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyesco%2Fjml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyesco%2Fjml/lists"}