Smarter debugging with the Duckling virtual machine

In the first post from the "Meet the compiler" series my colleague wrote about the importance of compiler technology, focusing on its speed and how it influences many aspects of the development process, including debugging. Effective debugging tools directly influence productivity, code quality, and the ability to maintain complex systems. That’s why we’re committed to improving the debugging experience, making it more intuitive and powerful for developers.

In this post, we explore Duckling’s dual-architecture approach, which combines a high-performance native code compiler with a dedicated virtual machine for execution. This unique design unlocks advanced debugging capabilities that are difficult to achieve in traditional systems.

Traditional debugging methods and their limitations

Debugging is an inevitable part of programming, requiring developers to rely on a range of methods, from basic print statements to advanced debugging tools. Even before a programmer gets involved, the compiler's static analysis serves as the first line of defense against errors, making it crucial to use a capable compiler and enable optional error-checking flags. On top of that, following good programming practices can make potential issues more apparent, reducing the need for extensive debugging later.

However, if the compiled code doesn’t behave as expected, the first thing I usually do is employ a method I like to call "scan and look". This involves skimming through the code to spot typos, fragments that don’t align with the rest, or anything that seems unusual. The biggest advantage of this method is that it works everywhere and requires no additional tools, but it typically becomes less effective with larger projects, and leaves me stuck, if I can't spot the error.

After scanning the code, poeple often try to verify the program's state by inserting print statements to output variable values to the console. This "print debugging" method is widely used because it's quick to implement and doesn't require additional tools. Sadly, it comes with several limitations:

• It can take multiple attempts to narrow down the issue.
• Complex data structures may not be printable (that's why in Duckling we strive to have every type printable by default).
• It involves modifying the source code, which must be restored to its original state.

While traditional debugging methods like "scan and look" and "print debugging" are useful, they often fall short when dealing with complex or large-scale projects. This is where the Duckling's virtual machine comes into play.

The Duckling's virtual machine

We should start with answering the question: what exactly is a virtual machine, and what does it do?

In general, programming languages are divided into two types: compiled and interpreted, but in reality, the line between the two is blurred. In fact, many popular languages, such as Java, Python, and C#, are both compiled and interpreted. When compiling such languages, the source code is first translated into an extremely simple programming language, called intermediate representation or bytecode, which is then executed in a runtime environment.

The program responsible for executing this intermediate code is called a virtual machine or, sometimes, an interpreter. The intermediate representation often resembles native processor code in its structure to facilitate speedy execution. However, even the most efficient runtime environments come with some overhead¹ compared to running native code directly on the processor. To combine the best of both worlds, Duckling supports two compilation paths: one that compiles directly to native machine code for maximum performance, and another that targets an intermediate representation for enhanced debugging.

At the heart of Duckling’s virtual machine lies DuckBC, our custom intermediate representation. A snippet of DuckBC looks like this:

start:
  mov_r0_const 10;
  add_r0_r1;
  cmpeq_r0_const 20;
  jmp_rel_if_label label1:
  return 0;
label1:
  return 1;

The exact meaning of these instructions isn’t crucial at this point, but if you’re familiar with assembly language, you’ll immediately notice the similarities.

A good way to describe a virtual machine is as an "abstraction of a processor." The idea is that the virtual machine simulates the execution of instructions of the intermediate code. In doing so, it has full control over memory, register states, and the pointer to the currently executed instruction. In essence, the virtual machine isn’t limited to merely executing code - it can control and monitor the entire state of the program, offering deep insight and precise control for debugging purposes.

The state of debugging tools today

To see how the Duckling virtual machine will help with debugging, it’s worth examining the tools and techniques that developers rely on today. Among these, the interactive debugger is the most widely used. This tool allows the user to pause the program, inspect its state and execute code step by step. It's particularly useful when the debugger is integrated into the IDE and no additional configuration is needed.

Every mature programming language comes with its own debugger. However, many interactive debuggers are problematic in one way or another. A common frustration is the overshooting problem, which forces you to restart the entire program to pinpoint where the bug occurred. In addition, when working with languages that are compiled to machine code (e.g. C++), changing even a single line in a function often requires recompiling the entire compilation unit, re-linking the project, reloading it into the debugger and restarting execution.

However, by using a compiler that supports incremental compilation and a dedicated virtual machine we can address these limitations, while giving even more flexibility to developers. By leveraging a VM, we strive to eliminate many of the pain points associated with traditional approaches.

Benefits of using a virtual machine

We will now explore some of the key benefits of using an interpreter for debugging. The technical details of how this is achieved is beyond the scope of this post, but rest assured — we'll cover them in the future. For now, we focus on a few ideas which will make the Duckling VM a powerful debugging tool.

Control over memory bugs

Memory management is one of the most common sources of errors in software development. While garbage-collected languages avoid manual memory handling, they sacrifice control and performance. On the other hand, languages like C/C++ grant full control but leave developers struggling with memory leaks, use-after-free errors, and pointer arithmetic mistakes.

One of the most popular tools for memory error detection is Valgrind, which operates at the binary level to analyze and modify executables. By inserting its own functions between memory allocation calls, it marks memory regions as "poisoned" and monitors every access, reporting errors when poisoned memory is referenced.

Importantly, it can't detect a situation when a pointer is moved in such a way that it points to a valid region of memory, but to a different variable. This is also the case with another tool called Address Sanitizer, which allocates memory on the stack with gaps that are labeled as "poisoned".

The Duckling virtual machine introduces much more precise memory tracking through a system of "memory blocks". Each pointer consists of a block identifier and an offset within that block. When a pointer is shifted, the only value that changes in the VM is the offset, making it impossible to shift a pointer in a way that causes it to reference a different memory block. This approach, combined with full control over code execution, ensures better detection of memory errors without relying on heuristics or approximations.

Hot code replacement

Hot code replacement is a powerful feature that allows developers to modify code while the program is running, without needing to restart it. However, implementing hot code replacement is notoriously difficult for languages compiled directly to machine code, such as C or C++. In contrast, languages running on virtual machines, like Java or C#, have a natural advantage due to their runtime control.

The concept of hot code replacement gained popularity with C# and its "Edit and Continue" feature in Visual Studio. Inspired by its success, Microsoft developers attempted to implement a similar feature for C++ in Visual Studio. It relies on close cooperation between the project management software and the debugger. Updating a function while the program is running is done by mapping the compiled code of the new function to the same memory location where the old function is loaded. Naturally, problems arise when the new function is significantly larger than the old one, which is why this method is highly limited.

The Linux ecosystem, with GCC and GDB, does not provide dedicated support for hot code replacement. However, GDB includes a set of features known as "altering execution". Essentially, while the program is loaded into GDB and paused, users can call functions, assign values to local variables, or jump to different points in the code. While these features offer some flexibility, they fall short of true hot code replacement, as it's the compiler's job to be able to recompile functions on the fly.

In the world of interpreted and VM-based languages, hot code replacement is more straightforward. Java’s "HotSwap" feature, for example, allows developers to replace methods at runtime, though it has its own limitations. For example, it doesn't support adding or removing class members or modifying methods already on the call stack.

Duckling’s virtual machine architecture simplifies hot code replacement by design. Together with powerful, incremental compilation, this capability transforms debugging from a stop-and-restart cycle into a fluid, iterative process.

Time travel debugging

Traditional debuggers force developers to predict where a bug might occur, set breakpoints, and check if they didn't "overshoot". Time travel debugging flips this model by recording the entire execution history, allowing developers to rewind and inspect the program state at any point. This is particularly useful for programs where the bug occurs intermittently, where once the program hits an error the developer can go back in time to find its source.

"Time travel debugging" is not a new concept, and there are many projects that implement it. The simplest way to achieve this mechanism is to record the entire execution history of the program, line by line, into an external file. After that, another program loads this file and allows interactive playback of the execution history.

However, when it comes to rewinding to the last checkpoint during an interactive debugging session, a significant challenge lies in handling sources of randomness and interfacing with the operating system. These include random number generators, console input, system clock readings, or thread scheduling. The most popular software that records all sources of non-determinism is Undo.io and they value their product at about $1,000 per developer per year.²

Few people are aware that newer versions of GDB also include a built-in mechanism for reversing executed instructions. However, it’s not widely popular, partly because of its limited compatibility. As stated on GDB’s official page, the functionality heavily depends on the processor architecture and the operating system. ³

The core issue with GDB is that it debugs the machine code of a compiled application. With a virtual machine, which has full control over instruction execution, implementing "time travel debugging" becomes simpler and more universally applicable.

That said, it doesn’t solve everything. For instance, handling sources of randomness and system calls in the program is still a challenge, as the VM doesn’t have control over the host system.

However, implementing this technique for a limited subset of instructions can still bring significant benefits in specific use cases. For example, competitive programmers, who typically avoid multi-threading or system calls in their solutions, could greatly benefit from such a feature.

Summary

Duckling’s dual-architecture approach improves the debugging experience by integrating a high-performance compiler with a dedicated virtual machine. Compiled-only languages (the ones that do not require a runtime) often suffer from inconveniences, such as the need to restart programs while debugging, limited runtime introspection, and slow recompilation cycles. By leveraging a VM, Duckling may overcome these limitations, enabling advanced debugging features like precise memory tracking, hot code replacement, and time travel debugging.

We’ll dive deeper into the inner workings of Duckling’s virtual machine in future posts — stay tuned!