Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/bendudson/EuraliOS

A hobby x86-64 operating system written in Rust
https://github.com/bendudson/EuraliOS

Last synced: about 1 month ago
JSON representation

A hobby x86-64 operating system written in Rust

Awesome Lists containing this project

README 

        
* EuraliOS



A hobby x86-64 operating system written in [[http://rust-lang.org][Rust]]. Named for Euralia in

A.A. Milne's [[https://en.wikipedia.org/wiki/Once_on_a_Time][Once on a Time]] because I don't know who this is intended

for except, to borrow Milne's introduction, "For those, young or old,

who like the things which I like...  Either you will enjoy it, or you

won't. It is that sort of +book+ project."



#+CAPTION: EuraliOS demonstration

#+NAME: fig-demo

[[./doc/euralios.gif]]



*Disclaimer*: This was written by someone who doesn't (or didn't) know

anything about Rust or operating systems. The result is probably not

idiomatic Rust or a sensible OS design. If you are looking for a

usable operating system implemented in Rust then [[https://www.redox-os.org/][Redox OS]] is probably

what you want.



This started with [[https://os.phil-opp.com/][Philipp Oppermann's amazing blog]] on OS development

with Rust (second edition), and makes use of many of the crates that

Phil maintains (e.g. [[https://docs.rs/x86_64/latest/x86_64/][x86_64]] and [[https://docs.rs/bootloader/latest/bootloader/][bootloader]]). That blog has inspired

many other operating system siblings that I've found useful to study

including [[https://github.com/WartaPoirier-corp/ananos][ananos]], the [[https://github.com/sos-os/kernel][Stupid Operating System]], [[https://github.com/intermezzOS][intermezzOS]], and

especially [[https://github.com/vinc/moros][MOROS the Obscure Rust Operating System]]. The [[https://github.com/redox-os/kernel][Redox OS

kernel]] is very readable, and [[https://github.com/betrusted-io/xous-core][Xous]] has a similar focus on message

passing (This is a more comprehensive [[https://github.com/flosse/rust-os-comparison][comparison of Rust operating

systems]]).  Anything useful here was probably borrowed from one of

these projects; mistakes and omissions are my own.



** Features



There are many different directions to explore in building an OS, and

with EuraliOS I've focussed mainly on memory management and

pre-emptive multitasking so far. The result is an OS that can't do

much, but it can do it all at the same time.



- The "Merriwig" kernel

  - [X] Pre-emptive multitasking of user and kernel threads, with a round-robin scheduler

  - [X] Program memory isolation using paging

  - [X] On-demand memory allocation using page fault handler

  - [X] Stack and heap memory management

  - [X] Frame allocator based on a bitmap tree structure

  - [X] ELF loader using the [[https://crates.io/crates/object][object crate]]

  - [X] Rendezvous message passing for all inter-process communication.

        Messages can contain communication handles or transfer ownership of memory regions

        between processes and threads.

  - [X] 18 syscalls for memory and process/thread control and inter-process communication

  - [X] A [[https://man7.org/linux/man-pages/man7/vdso.7.html][vDSO]]-like fast mechanism for access to timing information from user programs

  - [X] Virtual file systems that can be shared between processes or per-process

        as a mechanism for capability-based security and user isolation



- Drivers as userspace programs (Ring 3)

  - [X] Keyboard driver using the [[https://docs.rs/pc-keyboard/latest/pc_keyboard/][pc-keyboard]] crate

  - [X] Virtual consoles and VGA text output using the [[https://crates.io/crates/vga][vga crate]]

  - [X] RTL8139 network card driver

  - [X] TCP network stack using [[https://docs.rs/smoltcp/latest/smoltcp/][smoltcp]] in user space, with DHCP and DNS

  - [ ] Virtio network driver



- User programs

  - [X] Login process and multiple users

  - [X] A basic shell

  - [X] A simple [[https://en.wikipedia.org/wiki/Gopher_(protocol)][Gopher]] browser

  - [X] A RAM disk for temporary files

  - [X] A text editor

  - [ ] Virtio 9P to access host filesystems



There is no disk driver yet, so user programs are compiled and then

the ELF files are included in the =init= binary using Rust's

[[https://doc.rust-lang.org/std/macro.include_bytes.html][include_bytes]] macro (another idea taken from [[https://github.com/vinc/moros][MOROS]]).



** Building and running



Euralios depends on the [[https://github.com/rust-osdev/vga][vga]] crate, with some modifications in [[https://github.com/bendudson/vga/tree/euralios][a branch

here]] as a git submodule.  Either clone with submodules (with e.g the

=--recurse-submodules= option) or run:

#+begin_src bash

  $ git submodule update --init --recursive

#+end_src



After setting up Rust, Cargo and QEMU (see [[https://os.phil-opp.com/minimal-rust-kernel/][phil-opp blog]]), running

#+begin_src bash

  $ make run

#+end_src

should download dependencies, build everything, and launch qemu.



*Notes*



1. You'll probably need to use the Rust nightly build. EuraliOS was

   originally developed using Rustc =1.61-nightly (6a7055661 2022-02-27)= and

   is known to work with =1.79.0-nightly (7f2fc33da 2024-04-22)=.

2. Rust needs to build =core= and =compiler_builtins= libraries for

   the =x86_64-euralios= target. To do that cargo needs the rust source code,

   which can be installed by running =rustup component add rust-src=.



** Documentation



I've tried to document the steps to build EuraliOS, starting from the

end of [[https://os.phil-opp.com/heap-allocation/][post 10 (heap allocation)]] of Phil Opp's blog (2nd Ed). At that

point the kernel had a heap and a bump frame allocator (so could

allocate but not free memory), some code to handle pages,

exceptions, and VGA output. [[https://os.phil-opp.com/async-await/][Post 11]] went on to add Async/Await

cooperative multitasking, but here we go in a slightly different

direction...



- [[file:doc/journal/01-interrupts-processes.org][1. Interrupts and processes]] in which kernel multi-threading is

  introduced by modifying the timer interrupt handler to capture and

  change the context (registers etc).

- [[file:doc/journal/02-userspace.org][2. Userspace]] in which a program is loaded into memory and run in

  Ring 3 (user space). Syscall/sysret is used to implement a simple

  way to print from user programs.

- [[file:doc/journal/03-memory.org][3. Memory management]] in which we create separate page tables for

  user processes and stacks for each thread.

- [[file:doc/journal/04-more-syscalls.org][4. Syscalls for thread control]] which adds syscalls to spawn new

  threads and to exit from a thread. We also learn how to use the

  [[https://www.felixcloutier.com/x86/swapgs][swapgs]] and [[https://www.felixcloutier.com/x86/wrmsr][wrmsr]] instructions to switch stacks inside syscalls.

- [[file:doc/journal/05-memory-returns.org][5. Memory returns]] which implements a better frame allocator that

  can keep track of available frames, so that memory can be freed when

  threads and processes exit.

- [[file:doc/journal/06-user-memory.org][6. User space memory management]] in which we create a heap for each

  user process so that user programs can use =Box=, =Vec= etc, using a

  linked list allocator. As a bonus this allows a threading API with

  closures as in the Rust stdlib. We also re-organise the repository

  into a Cargo Workspace, with separate crates for the kernel and user

  program.

- [[file:doc/journal/07-ipc.org][7. Inter-process communication (IPC)]] where a simple "[[https://en.wikipedia.org/wiki/Rendezvous_(Plan_9)][rendezvous]]"

  communication method is implemented, enabling a user program to

  get input from the keyboard.

- [[file:doc/journal/08-faster-ipc.org][8. Faster IPC]] in which we switch tasks in syscalls and keyboard

  interrupt handler, to minimise delays in the communication.

- [[file:doc/journal/09-message-sending.org][9. Sending messages]] which adds the =send= syscall, allowing the user

  program to send messages to a VGA output kernel thread.

- [[file:doc/journal/10-stdlib.org][10. A standard library]] in which we start a =euralios_std= "standard"

  library, moving code out of the user program into a separate crate.

- [[file:doc/journal/11-messages.org][11. More messages]]

- [[file:doc/journal/12-devices.org][12. PCI devices]] where we start working on input/output to PCI

  devices and discovering which devices are available.

- [[file:doc/journal/13-return-to-sender.org][13. Return to sender]] where we add the =send_receive= syscall to make

  remote procedure calls more reliable, and start to develop a Virtual

  File System (VFS) with a new =open= syscall.

- [[file:doc/journal/14-network.org][14. RTL8139 network card]] where we develop a basic driver for the

  [[https://wiki.osdev.org/RTL8139][RTL8139]] card, adapting [[https://github.com/vinc/moros/blob/trunk/src/sys/net/rtl8139.rs][the MOROS driver]]. In the process we wrap the

  =send_receive= system call into an =rcall= remote procedure call,

  and add frame allocation of consecutive physical frames for direct

  memory access.

- [[file:doc/journal/15-messages.org][15. Message error handling]] where we add error handling and send retries to

  make messaging more robust. We also add a =thread_yield= system call

  to yield control of the processor when waiting and recovering from errors.

- [[file:doc/journal/16-arp.org][16. Address Resolution Protocol]] implementation: Writing a simple program

  to send and receive an ARP packet through the network.

- [[file:doc/journal/17-tcp-stack.org][17. TCP stack]] where the [[https://docs.rs/smoltcp/latest/smoltcp/][smoltcp]] crate is used to provide a TCP stack

  in user-space which communicates with the network card driver by

  messaging.

- [[file:doc/journal/18-gopher.org][18. Gopher]]: Developing a simple Gopher protocol browser, and in the

  process improving the Virtual File System (VFS) to handle more

  complicated OPEN messages, and the TCP program to READ and WRITE

  sockets.

- [[./doc/journal/19-timing.org][19. Timing]]: Adding functions to get time since system start. To

  speed up access to timer calibration data we map a page read-only

  into every user program, as the [[https://man7.org/linux/man-pages/man7/vdso.7.html][Linux virtual dynamic shared object

  (vDSO)]] does.

- [[./doc/journal/20-dns.org][20. Domain Name System (DNS)]], adding the ability to look up IP addresses

  from host names.

- [[./doc/journal/21-vga.org][21. VGA driver and terminals]] in user space, using the [[https://crates.io/crates/vga][vga

  crate]]. Allows separate consoles for system programs and user

  programs like the Gopher browser.

- [[./doc/journal/22-ramdisk.org][22. RAMdisk driver]] to store files and develop the filesystem API,

  starting a basic interactive shell able to list files and run

  programs.

- [[./doc/journal/23-keyboard.org][23. Interrupts and a better keyboard]]: Enabling user programs to receive

  hardware interrupts, and moving the keyboard driver out of the kernel

  into a user-space driver.

- [[./doc/journal/24-directories.org][24. Directories]] to hierarchically organise files

- [[./doc/journal/25-multiple-users.org][25. Multiple users]] and login process, using separate virtual file systems

  to control user capabilities

- [[./doc/journal/26-servers.org][26. Servers]], generalising the RAMdisk server code

- [[./doc/journal/27-text-editor.org][27. Text editor]]. Writing a text editor to run on EuraliOS.



** Notes



- [[./doc/journal/xx-shell.org][A shell]]

- [[./doc/journal/xx-usb.org][USB]] (xHCI)

- [[./doc/journal/xx-wifi.org][WiFi]]



** Bibliography



Useful reference material includes:



- [[https://wiki.osdev.org/Expanded_Main_Page][The OSDev.org wiki]]

- [[https://www.amd.com/system/files/TechDocs/24593.pdf][AMD64 Architecture Programmer's Manual (Vol 2: System programming)]]

- [[https://0xax.gitbooks.io/linux-insides/content/][Linux insides]] by [[https://twitter.com/0xAX][@0xAX]]

- [[https://www.kernel.org/doc/html/latest/][The Linux Kernel documentation]]



Other relevant links



- [[https://osblog.stephenmarz.com/index.html][The Adventures of OS: Making a RISC-V Operating System using Rust]]

- [[https://github.com/nuta/resea][The resea micro-kernel based OS]] written in C

- [[https://github.com/nuta/kerla/][The kerla monolithic kernel]] written in Rust

- [[https://github.com/thepowersgang/rust_os][The "Tifflin" kernel]] written in Rust

- [[https://github.com/IsaacWoods/poplar][Poplar]] operating system in Rust