{"id":13398655,"url":"https://github.com/JuniperKernel/JuniperKernel","last_synced_at":"2025-03-14T02:31:31.880Z","repository":{"id":60721650,"uuid":"100770242","full_name":"JuniperKernel/JuniperKernel","owner":"JuniperKernel","description":"R Kernel for Jupyter","archived":false,"fork":false,"pushed_at":"2018-12-09T22:35:24.000Z","size":6528,"stargazers_count":73,"open_issues_count":7,"forks_count":8,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-07-31T19:16:43.374Z","etag":null,"topics":["bloomberg","jupyter","jupyter-kernels","jupyter-notebook","r","rcpp","rlang","xeus","zmq"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JuniperKernel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-19T04:39:05.000Z","updated_at":"2024-07-26T09:15:33.000Z","dependencies_parsed_at":"2022-10-03T20:03:39.384Z","dependency_job_id":null,"html_url":"https://github.com/JuniperKernel/JuniperKernel","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuniperKernel%2FJuniperKernel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuniperKernel%2FJuniperKernel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuniperKernel%2FJuniperKernel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuniperKernel%2FJuniperKernel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JuniperKernel","download_url":"https://codeload.github.com/JuniperKernel/JuniperKernel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243511660,"owners_count":20302595,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloomberg","jupyter","jupyter-kernels","jupyter-notebook","r","rcpp","rlang","xeus","zmq"],"created_at":"2024-07-30T19:00:30.133Z","updated_at":"2025-03-14T02:31:31.267Z","avatar_url":"https://github.com/JuniperKernel.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"Juniper Kernel \u003cimg src=\"./extras/juniper_logo.png\" width=120, align=\"right\" /\u003e\n==============================================================================\n\n[![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/JuniperKernel)](https://CRAN.R-project.org/package=JuniperKernel)\n[![Travis](https://travis-ci.org/JuniperKernel/JuniperKernel.svg?branch=master)](https://travis-ci.org/JuniperKernel/JuniperKernel)\n[![Appveyor Build status](https://ci.appveyor.com/api/projects/status/gah4rq18ml06d6xp?svg=true)](https://ci.appveyor.com/project/spennihana/juniperkernel)\n[![Downloads](http://cranlogs.r-pkg.org/badges/JuniperKernel)](https://cran.rstudio.com/package=JuniperKernel)\n\nAn [R](https://cran.r-project.org/) kernel for [Jupyter](https://jupyter.org) built with [Xeus](https://github.com/QuantStack/xeus) and [Rcpp](http://www.rcpp.org/).\n\nThe development of JuniperKernel is sponsored by [Bloomberg](https://www.techatbloomberg.com/).\n\n## Install from CRAN\n\n```\n\u003e install.packages(\"JuniperKernel\")\n\u003e library(JuniperKernel)\n\u003e installJuniper(useJupyterDefault = TRUE)  # install into default Jupyter kernel location\n```\n\n## Install via Conda\n\nAvailability is architecture dependent, [see](https://anaconda.org/conda-forge/r-juniperkernel/files):\n* `conda install -c conda-forge r-juniperkernel` or\n* `conda install -c conda-forge/label/gcc7 r-juniperkernel`\n\n## Building\n\n##### Requirements\n\n- R \u003e= 3.4.0 \n- [Rtools34.exe](https://cran.r-project.org/bin/windows/Rtools/) for Windows\n\nIf you're going to build from scratch, then these packages will be necessary to fetch:\n \n- R packages: BH,  Rcpp (\u003e= 0.11.0), gdtools (\u003e= 0.1.6), pbdZMQ (\u003e= 0.3-0), roxygen2, jsonlite, repr, data.table\n\nOther necessary bacon bits (fetched automatically--c.f. Makevars/Makevars.win):\n\n- zeromq (4.4.2)\n- xeus (0.7.0)\n- xtl (0.3.1)\n\n#### Devtools Build and Install from Github\n\nInstalling from github is probably the easiest option.\n\nLoop through the requirements, and then do:\n\n```\ndevtools::install_github(\"JuniperKernel/JuniperKernel\")\n```\n\n\n#### Building on Windows\n\nSupported compilers:\n - mingw32/64 (bundled with Rtools)\n - no other compilers are officially supported (though you may have success via cygwin)\n\nSupported architectures:\n - 32/64 bit arch\n\nTo build and install the Juniper kernel run the following from a `cmd` prompt:\n\n```\ncmd /c mk.bat\n```\n\nAlternatively, you can run the build from RStudio. \n\nYou should see the following compilation output:\n\n```\nC:/Rtools/mingw_64/bin/g++  -std=gnu++11 -I\"C:/PROGRA~1/R/R-34~1.1/include\" -DNDEBUG -I../inst/include -I. -Wno-conversion-null -I\"C:/Program Files/R/R-3.4.1/library/Rcpp/include\" -I\"C:/Program Files/R/R-3.4.1/library/gdtools/include\" -I\"C:/Program Files/R/R-3.4.1/library/BH/include\"   -I\"d:/Compiler/gcc-4.9.3/local330/include\"     -O2 -Wall  -mtune=core2 -c RcppExports.cpp -o RcppExports.o\nC:/Rtools/mingw_64/bin/g++  -std=gnu++11 -I\"C:/PROGRA~1/R/R-34~1.1/include\" -DNDEBUG -I../inst/include -I. -Wno-conversion-null -I\"C:/Program Files/R/R-3.4.1/library/Rcpp/include\" -I\"C:/Program Files/R/R-3.4.1/library/gdtools/include\" -I\"C:/Program Files/R/R-3.4.1/library/BH/include\"   -I\"d:/Compiler/gcc-4.9.3/local330/include\"     -O2 -Wall  -mtune=core2 -c juniper.cpp -o juniper.o\nC:/Rtools/mingw_64/bin/g++ -shared -s -static-libgcc -o JuniperKernel.dll tmp.def RcppExports.o juniper.o -lzmq -Lx64 -Ld:/Compiler/gcc-4.9.3/local330/lib/x64 -Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-34~1.1/bin/x64 -lR\ninstalling to C:/Program Files/R/R-3.4.1/library/JuniperKernel/libs/x64\n** R\n** inst\n** preparing package for lazy loading\nNo man pages found in package  'JuniperKernel' \n** help\n*** installing help indices\n** building package indices\n** testing if installed package can be loaded\n* DONE (JuniperKernel)\n```\n\n#### Building on macOS\n\nYou may install via RStudio, or via terminal with\n\n```\n$ make\n$ make install\n```\n\nwhich produces the following output:\n\n```\n* installing to library ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’\n* installing *source* package ‘JuniperKernel’ ...\n** libs\nclang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I../inst/include/ -I../inst/include/xeus -I. -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include\" -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/gdtools/include\" -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include\" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o\nclang++ -std=gnu++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I../inst/include/ -I../inst/include/xeus -I. -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include\" -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/gdtools/include\" -I\"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include\" -I/usr/local/include   -fPIC  -Wall -g -O2 -c juniper.cpp -o juniper.o\nclang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o JuniperKernel.so RcppExports.o juniper.o -lzmq -L../inst/zmq -Wl,-rpath,/Library/Frameworks/R.framework/Resources/library/JuniperKernel/zmq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation\ninstalling to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/JuniperKernel/libs\n** R\n** inst\n** preparing package for lazy loading\n** help\n*** installing help indices\n** building package indices\n** testing if installed package can be loaded\n* creating tarball\npackaged installation of ‘JuniperKernel’ as ‘JuniperKernel_0.0.0.1.tgz’\n* DONE (JuniperKernel)\n```\n\n### Installing the Juniper Kernel\n\n##### Jupyter Requirements\n\nThese are the notebook versions that are compatible with the Juniper kernel:\n```\nnotebook version: 5.0.0\njupyter_client version: 5.1.0\njupyter_core version: 4.3.0\n```\n\nIf you want to also enable widgets, this is the compatible version:\n\n```\nwidgetsnbextension version: 3.0.2\n```\n\nNote also that Juniper installation depends on `jupyter kernelspec` being available:\n\n```\n$ jupyter kernelspec --version\n5.1.0\n```\n\n##### Juniper Kernel Installation\n\nThe Juniper kernel is installed via the R package that was just built and installed:\n\n```\n$ R\n\u003e library(JuniperKernel)\n\u003e installJuniper()\n[InstallKernelSpec] Installed kernelspec juniper_r3.4.1 in /PATH/TO/jupyter/kernels/juniper_r3.4.1\n```\n\nYou may also install into a virtual environment, provide your own names, and install under different R versions.\nTo list all of the available kernels you may run:\n```\n$ jupyter kernelspec list\n\n```\n\nor you can do it from the R package:\n\n```\n\u003e JuniperKernel::listKernels()\n```\n\n\n#### Juniper In Action\n\nJuniper Screenshot:\n\n![](./extras/jnote.png)\n\n\n#### xwidgets demo:\n\nxwidgets integration screenshot:\n![](./extras/xwidgets_demo.png)\n\n\n# Architecture Overview\n\nJuniper is a from-scratch Jupyter kernel implementing the 5.2 Jupyter messaging protocol and extending xeus\nfor custom messaging. The zeromq architecture handles R stdout/stderr in a novel \nway by listening on regular TCP sockets that R scribbles in via `socketConnection`s. \n\nHere's a diagram of the architecture:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./extras/zmq_arch.png\" width=500\u003e\n\u003c/p\u003e\n\nDespite the number of arrows and boxes in the diagram, the architecture is about as straightfoward \nas you'd expect it to be after a read-through of the docs. There's a client that connects to five \nsockets exposed by the kernel (`heartbeat`, `iopub`, `shell`, `ctrl`, `stdin`), a request handler, and two \ninternal sockets to handle ipc. Green boxes represent independent threads: heartbeat gets its own \nthread; IOPub gets its own thread; and shell, ctrl, stdin all share the same thread. The need for \nthe signaller and internal publisher may not be obvious; but a read-through of both the Jupyter \nand ZeroMQ docs make it evident that if you're going to have a multi-threaded zmq architecture, \nyou'll need a mechanism of ipc (with the additional wrinkle of portability). Practically speaking you'll \nneed some way of sending shutdowns to socket pollers (the signaller's job); \nand you'll need a way of ack'ing client requests with a 'busy' signal over IOPub or sending partial \nresults (since you can't just magically reach into that thread and say `iopub_sock.send('busy')`, \nand so you have the internal publisher pass the message along).\n\nIn order to start the kernel, a client forks off a new process with some JSON in a `connection_file`. \nPrimarily this file instructs the kernel about the transport mechanism (tcp, ipc, etc.), ip address, and ports for all sockets:\n\n```\n // sample connection_file\n{\n \"iopub_port\": 59992,\n \"stdin_port\": 59993,\n \"key\": \"8de88e2a-05ef-4ac8-9af0-000d6389cbc8\",\n \"shell_port\": 59991,\n \"transport\": \"tcp\",\n \"hb_port\": 59995,\n \"ip\": \"127.0.0.1\",\n \"control_port\": 59994,\n \"signature_scheme\": \"hmac-sha256\"\n}\n```\n\nMore details in the docs. The kernel boot sequence goes like this:\n\n1. Read connection_file\n2. Bind the internal pub/sig sockets\n3. Start the heartbeat/iopub threads\n4. Bind the ctrl/shell sockets\n5. Poll the ctrl/shell sockets forever\n\nThe heartbeat and iopub threads do exactly what the main thread does in binding sockets to the \nright ports, connecting to the necessary inproc topics, and properly breaking out of an infinite \npoll loop when signalled. By decree of the ZMQ docs, each thread is responsible for its own socket \ncreation/teardown; therefore, it should strike you that since all of the logic is the same, all of \nthe startup/poll/shutdown code should be shared. And to a large degree it is. You do need some \nindividuality so that messages can be ushered to the right handlers, but a generic solution is helpful \nfor maintenance.\n\n#### Request Handling and Execution\n`ctrl` and `shell` receive the preponderance of client requests, and these are all handled \nin a single-threaded fashion. Heartbeats are implemented with a plain-old echo socket that \npongs back whatever binary data the client pings the kernel with. It's worth noting that if \nyour kernel doesn't support a multithreaded model, then your kernel will run into hot water \nwhen launching code-execution requests that require execution times on the order of the elapsed \ntime between heartbeats; from the client's perspective the kernel looks non-responsive since your \nkernel has no way of saying it's not just stuck in 'busy' and doing real work.\n\nWhen a request comes in over a channel, there's some boilerplate handling to decode the message, \ndo validation, read off its type bits, create a reply, and finally ship the reply back over the \nchannel to the client. Reply content is created by doing some work in the language-specific engine \n(e.g., `plot(x,y)` or `hist(data)`) and then handing back any results to be packaged for client \nconsumption. The request handler, therefore, is the most appropriate place to draw a boundary where \nthe zeromq architecture can be abstracted away from the code-exec'ing one.\n\n\n# Version Notes\nRelease versions will be of the form `\u003codd/even\u003e.\u003ceven\u003e.\u003cpatch\u003e.0`\n\nDev versions will be of the form `\u003codd/even\u003e.\u003codd/even\u003e.\u003cpatch\u003e.\u003cdev\u003e`\n\nThe major version always increments.\n\n`\u003cpatch\u003e` numbers increment always and dev versions will match patched versions.\n\nFor example:  \n  1.0.0.0 is a release version\n  1.1.0.0 is a dev version\nnext minor:\n  1.2.0.0 is the next minor release version\n  1.3.0.0 is the next minor dev version\nnext major:\n  2.0.0.0 is the next major release version\n  2.1.0.0 is the next major dev version\n         \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJuniperKernel%2FJuniperKernel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJuniperKernel%2FJuniperKernel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJuniperKernel%2FJuniperKernel/lists"}