{"id":18315407,"url":"https://github.com/lefticus/embeddedscripting","last_synced_at":"2025-08-25T18:45:43.432Z","repository":{"id":137250562,"uuid":"51970232","full_name":"lefticus/EmbeddedScripting","owner":"lefticus","description":null,"archived":false,"fork":false,"pushed_at":"2016-03-21T15:14:48.000Z","size":32,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-15T06:45:48.333Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lefticus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-18T01:53:02.000Z","updated_at":"2023-02-16T19:24:33.000Z","dependencies_parsed_at":"2024-06-11T21:47:58.116Z","dependency_job_id":null,"html_url":"https://github.com/lefticus/EmbeddedScripting","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefticus%2FEmbeddedScripting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefticus%2FEmbeddedScripting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefticus%2FEmbeddedScripting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lefticus%2FEmbeddedScripting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lefticus","download_url":"https://codeload.github.com/lefticus/EmbeddedScripting/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045266,"owners_count":21038555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T16:39:32.956Z","updated_at":"2025-04-09T13:11:57.652Z","avatar_url":"https://github.com/lefticus.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"See also the [Feasibility Report](/Feasibility.md).\n\n# Goals\n\n * Provide a single executable that contains an embedded ruby interpreter and all requirements\n * Expose C++ functions using [SWIG](http://swig.org/) that defined inside of this executable to ruby\n * Support all major (MacOS, Windows, Linux) platforms equally\n * Attempt to reduce the number of temporary files created during program execution\n \n# Challenges\n\n * Embedding ruby into an executable\n * Loading SWIG exposed functions into embedded ruby\n * Embedding files into a C++ application\n * Loading embedded files as ruby modules\n \n# Solutions\n\n## Embedding Ruby Into An Executable\n\nThere are many nuances for properly initializing an embedding ruby. Fortunately, these nuances have been worked out previously\nin OpenStudio applications, so we have been able to borrow those solutions for this project. Specifically, the \n[RubyInterpreter.hpp](/RubyInterpreter.hpp) file has been cleaned up and adapted for use.\n\n## Loading SWIG Exposed Functions Into Embedded Ruby\n\nSWIG does not generate a header designed for manually initializing a generated module. However, the function name that is\ncreated by the generator is well defined because it is the name that is expected by Ruby.\n\nTo avoid `#include`ing large files or having to manipulate generated files, we simply declare and call the function\nthat we expect to exist.\n\nThat is:\n\n```cpp\nextern \"C\" {\n// declare expected initialization function\nvoid Init_EmbeddedScripting(void);\n}\n\nint main()\n{\n  // create and initialize ruby iterpreter \n  \n  // ...\n  \n  Init_EmbeddedScripting();\n}\n```\n\n(Note that the name 'EmbeddedScripting' is the name of the module as defined in the SWIG.i file.\n See [main.cpp](/main.cpp) and [EmbeddedScripting.i](/EmbeddedScripting.i#L1) for full details)\n\nThe declared function is found by the linker at link time when the SWIG generated module is compiled and linked\ninto this executable as any other .cpp file would be.\n\n## Embedding Files Into A C++ Application\n\n### Options\n\n#### Qt Resource System\n\nThe [Qt Resource System](http://doc.qt.io/qt-5/resources.html) is a common tool for this situation. However, Qt is a large\ndependency and inappropriate for this situation.\n\n#### xxd\n\n[xxd](http://linux.die.net/man/1/xxd) is a UNIX command line tool that can be used to dump binary data as hex strings designed for \nembedding into a C application (using the `-i` parameter). This tool is also available on recent releases of the git \nshell installer on Windows. Since [Git for Windows](http://www.git-scm.com/download/win) is largely a requirement for building the OpenStudio suite already,\nthis was the presumed solution initially. \n\nHowever, xxd's output does *not* give us:\n  \n  * Easy capability for embedding multiple files\n  * Storage or customization of the saved file's name\n  \n#### Pure CMake Solution\n\nUsing the [file](https://cmake.org/cmake/help/v3.0/command/file.html) function in CMake allows us to load a file as a string\nof hexidecimal bytes.\n\n```cmake\nfile(READ \"filename\" outputvariable HEX)\n```\n\nThis string can then be split up and dumped into a byte array in a generated cxx file. Allowing us to later access all of the\ndata and resulting location of the file.\n\n### Chosen Solution\n\nThe \"Pure CMake Solution\" was chosen as it provide the most flexibility and portability. The current implementation is not\nefficient because it does not generate an actual target, and needs to be run each time cmake is run. \n\nSee [CMakeLists.txt](/CMakeLists.txt) for the implementation details.\n\n## Loading Embedded Files as Ruby Modules\n\n### Options\n\n#### Implement Virtual Filesystem\n\n##### Concept\n\nIntercept C standard library function calls that are trying to access some predefined location (e.g. `myvirtualfs/`)\nand return back some well formed value that allows the application to access files stored inside of the current\nexecutable without having to actually extract any of those values.\n\n##### Implementation\n\nThis can be accomplished with a relatively simple concept of just defining our own versions of C library functions\nthen allowing the linker to choose our versions over others provided by the standard library.\n\nExample: \n\n```cpp\n#include \u003ciostream\u003e\n#include \u003cdlfcn.h\u003e\n\n// Note that there is no legal way in C++ to cast a `void*` to a function pointer,\n// so we must either use a c-style cast or a trick that involves \nint (*origopen)(const char *, int) = nullptr;\n\nextern \"C\" {\n  int open(const char *path, int flags) {\n    const auto id = origopen(path, flags);\n    std::cout \u003c\u003c \"(\" \u003c\u003c id \u003c\u003c \") open: \" \u003c\u003c path \u003c\u003c '\\n';\n    return id;\n  }\n}\n\nint main()\n{\n  origopen = (int (*)(const char *, int))(dlsym(RTLD_NEXT, \"open\"));\n}\n```\n\nor similarly on Winddows:\n\n```\n#pragma comment(linker, \"/FORCE\")\n\n#include \u003cstring\u003e\n#include \u003ciostream\u003e\n#include \u003cwindows.h\u003e\n\n#ifdef _DEBUG\nFILE *(*origfopen)(const char *, const char *) = (FILE *(*)(const char*, const char *))GetProcAddress(GetModuleHandle(TEXT(\"ucrtbased.dll\")), \"fopen\");\n#else\nFILE *(*origfopen)(const char *, const char *) = (FILE *(*)(const char*, const char *))GetProcAddress(GetModuleHandle(TEXT(\"ucrtbase.dll\")), \"fopen\");\n#endif\n\nFILE *fopen(const char *name, const char *mode)\n{\n  std::cout \u003c\u003c \"fopen called: \" \u003c\u003c name \u003c\u003c '\\n';\n  return origfopen(name, mode);\n}\n\nint main()\n{\n}\n```\n\n##### Issues\n\nWhile the UNIX method is a fairly normal way of intercepting C function calls, it's not entirely clear if the Windows version would\nwork as expected.\n\nParticularly note that we had to enable `/FORCE` linker flags to get it to link due to multiply defined symbols.\n\n*A more complete and general solution for intercepting function calls in Windows is available on [CodeProject](http://www.codeproject.com/Articles/34237/A-C-Style-of-Intercepting-Functions)*\n\nMore generally, however, we have the issue of needing to intercept many different function calls on two different\nplatforms (UNIX/Windows) to provide the consistent behavior that Ruby expects.\n\nThat is, we need to convince ruby:\n\n * Our virtual directory exists\n * Our virtual directory is readable with current permissions\n * Our virtual directory is in fact a directory (as opposed to a file)\n * The virtual files it wants to read exist\n * The virtual files are readable with current permissions\n\nWe also need to be able to indicate:\n \n * File size\n * End of file status\n * Error conditions\n   * Attempt to write to file\n   * Attempt to open file in write or append mode\n\nIt's estimated that approximately 10 system calls would need to be intercepted and our own thread safe `FILE *`, `DIR *` and \nfiledescriptor tracking system would need to be implemented.\n\n#### Intercept Ruby `require` Method\n\nThe concept is to implement our own method that replaces the ruby `Kernel::require` method. \n\n 1. Our version would intercept require calls that are looking in a path that exists in our virtual filesystem.\n 2. If the file exists in our internal filesystem the file will be read and then evaluated\n 3. If not, the require call is passed to the built in require method\n\n##### Implementation\n\nThe idea is sound, as it's exactly what rubygems does. A sample implementation is:\n\n```ruby\nmodule Kernel\n  if defined?(gem_original_require) then\n    # Ruby ships with a custom_require, override its require\n    remove_method :require\n  else\n    ##\n    alias gem_original_require require\n    private :gem_original_require\n  end\n\n  def require path\n    puts \"Requiring #{path}\"\n    # Actually do something here\n    return gem_original_require path\n  end\nend\n```\n\nThis version would likely need to be executed before the `rubygem` module is loaded so that the load for it\ncan be intercepted as well.\n\n##### Issues\n\n * What to do if the found file is a dynamic module? The implementation would need to somehow implement the mechanism of `LoadLibrary` on Windows and `dlopen` on UNIX to be able to load a shared object from inside of the existing executable.\n * This code can only execute after the ruby interpreter has been initialized, so it cannot be used to load the ruby gems modules, for instance\n\n\n#### Actually Mount Embedded Files As A Virtual FS\n\nThe concept is to tell the operating system to create embedded files as if they were a real filesystem so that they would be visible\nto all applications.\n\nOn Linux and MacOS this could be implemented using the [user mode filesystem](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) driver \n[FUSE](https://github.com/libfuse/libfuse). On Windows a similar approach could be taken with [dokan](http://dokan-dev.github.io/).\n\nUnfortunately using dokan on Windows would require the user to install an OS driver. So this option was ruled out and not fully explored.\n\n\n#### Extract Embedded Files To Actual Filesystem\n\nThis is the most straightforward and least error prone option\n\n 1. On start of executable, extract files to temporary location\n 2. Initialize ruby interpreter with temporary location as an inlude path\n 3. Perform work\n 4. Delete temporary location when exitting application\n\n### Chosen Solution\n\nAs outlined in the various possibilities, many of these solutions where experimented with. This codebase contains enough\nexamples and information to more fully explore most of them.\n\nThe recommended solution is the final option, extracting the embedded files to the local filesystem.\n\n * It's the most versatile, since it requires no special cross-platform considerations\n * It allows for possible compression of embedded files\n * It occurs before the Ruby interpreter is initialized, so core Ruby libraries can be stored\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flefticus%2Fembeddedscripting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flefticus%2Fembeddedscripting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flefticus%2Fembeddedscripting/lists"}