Introducing Mkos ================ Mkos is a real-mode, 16-bit operating system for x86/PC compatible hardware. I recently reached the first major project milestone: the smallest slice of functionality that's satisfying to me. Now's a good time for me to take a retrospective look and think about what to do next. What can Mkos do for a user today? Create, edit, list, and delete files. Here's a quick demo: .. raw:: html In this post and the next few following it, I'll show you some of the work behind this very short demo. Inspiration ----------- Mkos was inspired by my continued desire to work closer to electronic hardware and get more experience in programming for embedded systems. I had a couple false start attempts at OS development circa 2016. I made one attempt at starting one for x86/PC and one for ARM. I had very little experience with C, assembly language, and register CPU operation at the time. While small in scope, my efforts at creating breadboard and FPGA computers, along with my focus on programming applications in C, have prepared me better this time around. That's not to say this has been an easy task, or that I haven't made costly mistakes. I knew I wanted to make a new attempt at writing an OS, but I deliberated for some hours over which CPU and hardware platform to target first. I considered microcontrollers and single board computers. As a beginner, though, the temptation to go with PC compatible hardware (I'll just say "PC" from now on) was just too attractive. The platform is well-understood and well-documented. The BIOS is available as a "cheat" to quickly bootstrap video and I/O. There's an enormous number of interesting assembly language and C programs to potentially port to a hobby OS. PC it would be. While I've done projects around other CPUs like 6502, LR35902 (Game Boy), and 68000, I'm a newcomer to the x86 CPU line. I've recently taken an interest in this processor line, though, and done a bit of real mode assembly language programming for it. Beyond some research and `light journalism `__, I have no knowledge of how to use x86 in 32-bit protected mode. While I'm interested in later porting this project to protected mode systems, I knew I wanted to start with a real mode, 16-bit OS. This gives me the potential to run my OS on 8088- and 80286-based computers, and satisfies my penchant for resource-constrained development. Scope ----- When I first started the project, it was pretty clear what to work on next. As I started to get some things working, though, the list of possibilities grew. To help focus, I came up with the following *phase 1* scope for functionality of this operating system: - Boot from a floppy on a real computer and run completely from the floppy disk. - Support running kernel and user code in C and x86 assembly language. - Support minimal heap memory management. - Be built with automated testability in mind and carry reasonable test coverage. - Include a FAT filesystem driver and a floppy disk driver. - Implement a set of kernel syscalls allowing the user to execute programs and read and write from the floppy disk. - Implement enough of the C language to support the rest of the scope. Notably: - display I/O (printf) - memory management (malloc/free) - filesystem I/O (much of stdio.h) - Include a command shell to run any user program. - Include the following user programs: - ls: list the contents of the filesystem directory. - cat: display the contents of a file. - rm: delete a file. - edit: create and edit files in a full-screen editor. After I settled on that scope, the tasks fell naturally out of it. Design ------ I haven't done much design work for this project. As a newcomer to OS development, I don't feel like I know enough to do an up-front design, so much of the design process has been iterative. I did have two design goals from the beginning, though: - Create the simplest possible working system. - Work with automated testing in mind. From these goals, this general data design resulted: - To a reasonable degree, the components of this system shall be testable automatically and in isolation. - This is a single tasking operating system. - Kernel and user programs each occupy different address spaces. Each of kernel and user programs can occupy up to one physical memory segment (64 KiB). - If a driver manages a shared resource that must be managed by the kernel (e.g. filesystem), the driver shall be in kernel space. Otherwise, it shall be in user space. These goals also resulted in this basic design for control flow: - The computer power-on sequence transfers control to the boot loader, of course. - The boot loader loads the entire kernel into memory. - The kernel saves its state and loads the default user program into memory. - The user program invokes kernel functions using an interrupt. The interrupt handler saves the user program state, restores the kernel state, and transfers control to the kernel, which executes the desired function. The interrupt handler then restores the user state and transfers control back to the user program. The default user program is the command shell, where the user can type commands that run other programs. The `build system <#build-system>`__ also makes an automated test image. In this image, the test suite program is the default user program. Booting it immediately runs the tests. This graph shows the major components of Mkos and how they depend on each other. .. raw:: html Mkos dependency graph The source code is organized into these directories: - /kernel: kernel code and kernel interrupt handler. - /user: user programs. - /system: drivers and system services. - /syscall: the C interface that user code can use to interrupt the kernel with a syscall. - /libc: our implementation of the C standard library. - /test: automated tests for the project. As mentioned earlier, a particular driver may run in either kernel or user spaces. However, the source code for all drivers is in a ``system`` directory. This makes it easy to link either kernel code or user code, or both, with any particular driver. One example of where this is useful is for automated tests, which run in user space. While a driver may run in kernel space in the context of the running operating system, it doesn't have to be that way for an automated test to exercise the driver's code. A test program can link the driver and exercise it without invoking the kernel. In fact, it's easier to test a kernel driver outside of the running system, because that removes the system state and the entire interrupt handling layer from the situation. Development Environment ----------------------- Development Workstation OS ~~~~~~~~~~~~~~~~~~~~~~~~~~ I'm currently using `NixOS `__ as the distribution for all my GNU/Linux systems, including the workstation I use to develop Mkos. Toolchain ~~~~~~~~~ After I decided I wanted to use C in the project, I set to work looking for a 16-bit C toolchain and found the excellent `OpenWatcom `__. Mkos currently uses OpenWatcom 1.9, since that's the latest version that was in Nix packages at the start of the project. The `OpenWatcom v2 fork `__ has since been added since then, so I may try it later. OpenWatcom has worked well. A few of its many features that are useful in this project are: - wcc: 16-bit C compiler - wdis: x86 disassembler with support for `Relocatable Object Module `__ (OMF) format - wlink: object linker with support for OMF input, flat binary file output, and output memory map generation. Early in the project, I hadn't added C language support to its scope and was writing all code in x86 assembly. Being new to x86 assembly, I wasn't sure what assembler to use. I tried several assemblers, including `fasm `__, `NASM `__, and OpenWatcom's w86. Once I starting adding C language support and kernel interrupts, it became clear I needed to link C and assembly object files together. Since wlink has great support for the OMF file format, I'd need an assembler that supported OMF output. NASM supports it directly, so I settled on NASM as the assembler. Build System ------------ Mkos uses GNU Make and a combination of C programs and shell scripts to build the OS. All files produced in the standard build go into a ``build`` directory. Similarly, the automated test build goes into a ``build_test`` directory. It took some work to completely separate source and build directories, but there are some benefits I like. It makes it easy to see if the ``make clean`` operation is leaving behind any files. And if there's any suspicion that the build directory is in a problematic state or the Makefile is failing at incremental builds, it's easy to just blow away the whole build directory and test from a clean start. dd/mkfs/mtools ~~~~~~~~~~~~~~ The last stage of the build is to create an image file to be used in an emulator or written to physical floppy disk for use on actual hardware. We use dd to create an empty floppy image file and mkfs.fat to create the empty FAT filesystem on it. I started out using a `loopback device `__ to mount the image and copy the OS's files onto it. This had the disadvantage of making me enter my superuser password every build. I eventually switched to mtools, which can work with FAT filesystems in disk images without mounting them as devices, and that's an improvement. Emulated Hardware and Debugging ------------------------------- I do most development and testing of Mkos inside an emulated PC virtual machine. I did some early testing with QEMU. I quickly switched to `bochs `__, which is easy to configure for PC-specific work. bochs has a great debugger with a nice feature called "magic breakpoint"; if you place the useless ``xchg bx,bx`` assembly instruction anywhere in code, bochs invokes the debugger at that point. bochs also has an option to redirect the emulated serial port to a file. Mkos sends all its display output to the serial port, and bochs writes it to a file. I don't have to worry about important information scrolling away on the emulated display. Target hardware --------------- As a real mode operating system, one of the design goals of Mkos is that it run on hardware all the way back to the IBM 5150 PC and compatible machines. I don't have one of these systems, so I don't have a way to test performance against that goal right now. The earliest machine I have is a Compaq Presario 4784. This machine has a Pentium 1 100 MHz CPU and onboard VGA graphics. Coincidentally, I picked up a Lacie USB 3.5 in. floppy drive at a thrift store last year. Armed with it and some floppies from eBay, I can use dd to write the build image to a floppy disk, and then boot that disk on the Compaq machine. All automated and manual tests pass on the Presario. Next ---- I'm going to continue working on Mkos for the time being. I have several ideas for things I might like to include in *phase 2*: - Support for executing DOS COM/EXE files. - Try to port some real mode games. - Add support for hard disk drives/FAT16 filesystem. - Basic multi-tasking. I haven't decided what I want to do next, so I may try experimenting with several ideas and then plan. In the meantime, I want to write about Mkos as it is today. In the next few posts, we'll take a closer look at some of its components. Your comments and feedback are appreciated. If you'd like to comment, please reply to the threads for this post on Twitter and Mastodon, which serve as the comments section for this blog.