2024-12-12 08:00:00
Jujutsu is a new version control system that seems pretty nice!
The first few times I tried it I bounced off the docs, which to my taste has too much detail up front before I got the big picture. Someone else maybe had a similar experience and wrote an alternative tutorial but it's in a rambly bloggy style that is also too focused on the commands for me.
I suspect, much like writing a monad tutorial, the path of understanding is actually writing it down. So here's an attempt from me at an introduction / tutorial.
Perhaps unlike the others, my goal is that this is high-level enough to read and think about, without providing so much detail that it washes over you. Don't try to memorize the commands here or anything, they're just here to communicate the ideas. At the end if you're curious to try I recommend the docs found on their website.
Omitting details, you can think of Jujutsu (hereafter "jj") as a new Git frontend. The underlying data is still stored in Git. The difference is how you interact with your files locally, with a different conceptual model and a different set of commands.
Git quiz: are commits snapshots of file state or diffs? The technical answer is subtle — as a user you usually interact with them as diffs, while conceptually they are snapshots, but concretely they are stored as deltas. The more useful answer is that thinking about the details obfuscates the conceptual model. Similarly, to describe jj in terms of what happens in Git is tempting but I think ultimately clouds the explanation.
In practice what this means is try to put your knowledge of Git on hold, but also be aware you can use jj and continue to interoperate with the larger Git ecosystem, including e.g. pushing to GitHub.
The purpose of a version control system is to keep track of the history of your code. But interestingly in most, as soon as you edit a file locally in your working copy, that new history ("I have edited file X starting on version Y") is is in a kind of limbo state outside of the system and managed separately.
This is so pervasive it's almost difficult to see. But consider how a command
like git diff has one mode that takes two commits to diff, and then a bunch of
other modes and flags to operate on the other kinds of things it tracks. You can
get a diff against your working copy but there's no way to name "the working
copy" in the diff command. (Git in particular adds the additional
not-quite-a-commit state of the index, with even more flavors of attendant
commands. The ultimate Git quiz: what are the different soft/hard/mixed
behaviors of git reset?)
Another example: consider how if you have a working copy change and and want to
check out other code you either have to put it in a new place (git stash, a
fourth place separate from the others) or make a temporary commit. Or how if you
have a working copy change that you want to transplant elsewhere you
git checkout -m, but to move around committed changes it's git rebase.
In jj, in contrast, your working copy state is always a commit. When making a new change this is a new (descriptionless) commit. Any edit you make on disk is immediately reflected in the current commit.
So many things fall out of this simple decision!
jj diff shows a commit's diff. With no argument it's the current commit's
diff, i.e. your current working copy's diff; otherwise you can specify which
historical one you want. Many other jj commands similarly have a pleasing
symmetry about their behavior like this.
You can draft the commit message for a work in progress commit before you're
done, using the same command you'd use to edit any other commit message. There
is no final jj commit command, the commit is implicit. (Instead you jj new
to start a new empty commit when done.)
You never need to "stash" your current work to go do something else, it is already stored in the current commit, and easy to jump back to.
In Git, to fix a typo in an old commit, you might make a new commit then
git rebase -i to move the patch around. In jj you directly check out the old
commit (because working copy == commit) and edit the files with no further
commands.
(This blog post
walks through a real-world operation like this with Git and jj side by side.)
From a Git perspective, jj is very "rebasey". Editing a file is like a
git commit --amend, and in the "fix a typo" move above the edit implictly
rebases any downstream commits. To make that work out there are some other
conceptual leaps around conflict handling and branches that will come below
after the basics.
In a Git repo:
$ jj git init --colocate
This creates a .jj dir that works with the Git repo. Git commands will still
work but can be confusing.
The plain jj command runs jj log, showing recent commits. Here it is from
the repo for this blog:
$ jj
@ zyqszntn [email protected] 2024-12-12 11:58:52 21b06db8
│ (no description set)
○ pmnzyyru [email protected] 2024-12-12 11:58:48 86355427
│ unfinished drafts
◆ szzpmvlz [email protected] 2024-09-18 09:08:15 fcb1507d
│ syscalls
~
The leftmost letter string is the "change id", which is the identifier you use
to refer to the change in a command like diff. They are stable across edits,
unlike the Git hashes on the right. In the terminal the change ids are colored
to show the necessary prefix to uniquely refer to them (a single letter) in
commands.
The topmost commit zyqszntn is the current one, containing this blog post as I
write it. As you would expect, if I run jj status it shows me the list of
edited files, and if I run jj diff it shows me a diff.
I can give it a description now or when I'm done:
$ jj desc -m 'post about jujutsu'
And then create a new commit for the next change:
$ jj new
That's enough for trivial changes, but often I work on more significant changes where I might lose context across days. There are two ways you might do this depending on how you work.
The first is to just describe your change as above and keep on editing it,
without running jj new. Each subsequent edit will update the change as you go.
This is simple to operate but it means jj diff will always show the whole
diff. In Git this is similar to just keeping a lot of edits in your working
copy.
The other option is called
"the squash workflow"
in the tutorial book. In this, when you do new work you jj new to create a new
distinct commit from your existing work, and when you are happy with it (by e.g.
examining jj diff, which shows you just the working copy's new changes) you
run jj squash to flush these new changes into the previous commit. To me this
feels pretty analogous to using the Git index as a staging area for a complex
change, or perhaps repeatedly using git commit --amend.
These commands like jj diff and jj desc work on the current commit (or any
explicitly requested via the -r flag).
To switch the working copy to an existing change, it's jj edit <changeid>.
Again, any changes you make here, to the files or descriptions, or by making new
changes and squashing them, work directly on the historical commit you are
editing. I repeat this because it is both weird and obvious in retrospect.
Any operations on history cause implicit rebases that happen silently. Rebases can conflict. jj has interesting handling of how this works.
In Git, rebase resolution happens through your working copy, so there is again
extra state around "rebase in progress" and git rebase --continue. In jj
instead, conflicting commits are just recorded as conflicting and marked as such
in the history, so rebases always "succeed" even if they produce a string of
conflicting commits.
If you go to fix a conflicting commit (via jj edit as above), you edit the
files as usual and once the conflict markers are removed it's no longer
considered conflicting.
As usual, once you make a history edit, downstream changes are again rebased, possibly resolving their conflicted state after your edit. Again, the jj pattern of "all of the relevant information is modeled in the commits" without having a separate rebase mode with state etc. is a recurring powerful theme.
I don't have a lot of experience with this yet so I can't comment on how well it works, except that the times I've ran into it I was pleasantly surprised. The jj docs seem proud of the modeling and behavior here which makes me think it's plausibly sophisticated.
jj doesn't have named branches, but rather only keeps track of commits. Because of the way jj juggles commits, where it's trivial to start adding commits at random points in history, branch names are not as useful. In my experience so far having useful commit descriptions is enough to keep track of what I'm working on. Coming from Git the lack of named branches is surprising, but I believe this is comfortable for Mercurial users and Monotone worked similarly (I think?).
It's worth highlighting the absence of branches because in particular when
interoperating with Git you still do need branches, if only to push. There is
support in jj for this (where "bookmarks" are pointers to specific commits) but
it feels a little clunky. On the other hand, I probably have Stockholm syndrome
about the git push syntax.
Working with jj made me realize how much I rely on VSCode's Git support, for both viewing diffs and for merges.
When editing a given commit in jj, Git thinks all the files in the commit are in the working tree and not the index. In other words, in the VSCode UI the current diff shows up as pending changes just as they would in Git. This works pretty well and is about all I would expect. I haven't yet touched the buttons that interact with Git's index, for fear of what jj will do with it.
For technical reasons I do not quite understand — possibly VSCode only does three-way file merges and jj needs three-way directory merges? — the two do not quite cooperate for resolving conflicts. The jj docs recommend meld and I have used meld in the past but I hadn't quite realized how VSCode had hooked me until I missed using it for a merge.
The author of jj works at Google and is possibly making it for the Google internal version control system. (Above I wrote that jj is a Git frontend, but officially it has pluggable backends; I'm just unlikely to ever see a non-Git one.)
When I left Google three years ago I recall they were trying to figure out what to do about either making Git scale, or adopting Mercurial, or what. I remember talking to someone involved in this area and thinking "realistically your users have to use Git to work with the larger world, so anything else you do is pure cost". I found this post from a Mercurial fan about jj an interesting read in how it talks about Mercurial shortcomings it fixes. From that perspective it is pretty interesting: it can replace the places you currently use Git, while also providing a superior UI.
In all, jj seems pretty polished, has been around for years, and has a pretty simple exit strategy if things go wrong — just bail out to the Git repo. I aim to continue using it.
PS: every time I read the name "jujutsu" I kept thinking it was a misspelling of "jiu-jitsu", the martial art. But the Japanese word is じゅうじゅつ, it's actually it's jiu-jitsu that is misspelled! Read a longer article about it.
2024-09-18 08:00:00
This post is part of a series on retrowin32.
The "Windows emulator" half of retrowin32 is an implementation of the Windows API, which means it provides implementations of functions exposed by Windows. I've recently changed how calls from emulated code into these native implementations works and this post gives background on how and why.
Consider a function found in kernel32.dll as taken from MSDN (with some
arguments omitted for brevity):
BOOL WriteFile(
[in] HANDLE hFile,
[in] LPCVOID lpBuffer,
[in] DWORD nNumberOfBytesToWrite,
);
In retrowin32 we write a Rust implementation of it like this:
#[win32_derive::dllexport]
pub fn WriteFile(
machine: &mut Machine,
hFile: HFILE,
lpBuffer: &[u8],
) -> bool {
...
}
Suppose we have some 32-bit x86 executable that thinks it's calling the former. retrowin32's machinery connects it through to the latter — even though the implementation might be running on 64-bit x86, a different CPU architecture, or even the web platform.
Before we get into the lower-level x86 details, you might first notice that
those function prototypes don't even take the same arguments —
nNumberOfBytesToWrite disappeared.
On the caller's side, a function call like WriteFile(hFile, "hello", 5);
creates a stack that looks like:
return address ← stack pointer points here
hFile
pointer to "hello" buffer
5
The dllexport annotation in the Rust code above causes a code generator to
generate an interop function that knows how to read various Rust types from the
stack. For example for WriteFile the code it generates looks something like
the following, which maps the pointer+length pair into a Rust slice:
mod impls {
pub fn WriteFile(machine: &mut Machine, stack_pointer: u32) -> u32 {
let hFile = machine.mem.read(stack_pointer + 4); // hFile
let lpBuffer = machine.mem.read(stack_pointer + 8); // pointer to "hello"
let len = machine.mem.read(stack_pointer + 12); // 5
let slice = machine.mem.slice(lpBuffer, len);
super::WriteFile(machine, hFile, slice) // the human-authored function
}
}
There's a bit more detail around how the return value makes it into the eax
register and how the stack gets cleaned up, but that is most of it. With these
in place, the remaining question is how to intercept when the executable is
making a DLL call and to pass it through to this implementation.
Emulation aside, when an executable calls an internal function it knows where that function is in memory, so it can generate a call instruction targeting the direct address. When an executable wants to call a function in a DLL, it doesn't know where in memory the DLL is, so how does it know where to call?
In Windows this is handled by the "import address table" (IAT), which is an array of pointers at a known location within the exectable. Each call site generates code that calls by indirecting through the IAT: instead of calling WriteFile at specific address, it calls the address found in the WriteFile entry in the IAT. At load time, each needed DLL address is written into the IAT.
(In this picture, XXXX is a constant address pointing at the IAT, while CCCC is the address written into the IAT at load time that points to the implementation of WriteFile.)
Advanced readers might ask: given that we're writing address into the IAT at load time, why is there this indirection instead of just writing the correct addresses in to the call sites? The answer is there's a complex balance between making things fast in the common case as well as in sharing memory. Windows even has an optimization that attempts to make the (mutated after loading, so typically causing a copy on write) IATs amenable to being shared across process instances.
In retrowin32's case we just needed some easy way to identify these calls. So retrowin32 poked easy to spot magic numbers into the IAT.
The emulator's code then looks like:
const builtins = [..., WriteFile, ...]
fn emulate() {
if instruction_pointer looks like 0xFIAT_xxxx {
builtins[xxx].call()
} else {
...emulate code found at instruction_pointer...
}
}
This worked fine for a long time! But it had a few important problems that became apparent over time.
LoadLibrary() call gives you a pointer to the real base address
where a DLL was loaded. An executable I was trying to get to run was calling
LoadLibrary() and was attempting to traverse the various DLL file headers
found at that pointer. Because retrowin32 didn't actually load real
libraries, these didn't exist.msvcrt.dll exports plain variables like ints.WriteFile().The shared root cause of some of the above issues is that executables just expect real DLLs to be found in memory. (Pretty much every problem bottoms out as Hyrum's Law.) I could make retrowin32 attempt to construct all the appropriate layout at runtime, but ultimately we can match DLL semantics best by creating real DLLs for system libraries and loading them like any other DLL.
However, all these DLLs need to do is hand control off to handlers within retrowin32. For this I changed retrowin32's code generator to generate trivial snippets of x86 assembly which compile into real win32 DLLs as part of the retrowin32 build. (My various detours through cross-compiling finally paying off!) retrowin32 needs to be able to load DLLs anyway to load DLLs found out in the wild, and by using a real DLL toolchain to construct these I guarantee the system DLLs have all of the expected layout.
I can then bundle the literal bytes of the DLLs into the retrowin32 executable such that they can be loaded when needed without needing to shuffle files around; they are a few kilobytes each.
(Coincidentally, the DREAMM emulator, a much more competent Windows emulator, recently made a similar change for pretty similar reasons, which makes me more confident this is the right track.)
Each stub function needs to trigger the retrowin32 code that handles builtin
functions.
On x86-64 that code is especially tricky.
To make this swappable, these stub functions all call out to known shared
function, yet another symbol that retrowin32 can swap out for the different use
cases. They can find a reference to that symbol using exactly the DLL import
machinery again with a magic ambient retrowin32.dll.
Under emulation retrowin32_syscall can point at some code that invokes the x86
sysenter instruction, which is arbitrary but easy for an emulator to hook. On
native x86-64 we can instead implement it with the stack-switching far-calling
goop.
Finally, in this picture, how does retrowin32_syscall know which builtin was
invoked? At the time the sysenter is hit, the return address is at the top of
the stack, so with some bookkeeping we can derive that it was Writefile from
that address.
COM libraries like DirectDraw expose vtables to executables: when you call e.g.
DirectDrawCreate you get back a pointer to (a pointer to) a table of functions
within DirectDraw. For the executable to be able to read that pointer, it has to
exist within the emulated memory, which means previously I had some macro soup
that knew to allocate a bit of emulated memory to shove some function pointers
into it when the DLLs were used.
After this change, I can just expose the vtables as static data from the DLLs, like this:
.globl _IDirectDrawClipper
_IDirectDrawClipper:
.long _IDirectDrawClipper_QueryInterface
.long _IDirectDrawClipper_AddRef
.long _IDirectDrawClipper_Release
.long _IDirectDrawClipper_GetClipList
.long _IDirectDrawClipper_GetHWnd
...
To generate these stub DLLs, I currently generate assembly and run it through
the clang-cl toolchain, which seamlessly cross-compiles C/asm to win32. It
feels especially handy to be able to personally bug one of the authors of it for
my tech support, thanks again Nico!
It feels a bit disappointing to need to pull in a whole separate compiler
toolchain for this, especially given that Rust contains the same LLVM bits. I
spent a disproportionate amount of time tinkering with alternatives, like
perhaps using one of the other assemblers coupled with the lld-link embedded
in Rust, or even using Rust itself, but I ran into
a variety of small problems with that.
Probably something to revisit later.
Any time I tinker in this area (and also looking at Raymond Chen's post, also linked above) I always find myself thinking about how much dang machinery there is to make dynamic linking work and to try to fix up the performance of it.
It makes me think of a mini-rant I heard Rob Pike give about dynamic linking (it was clearly a canned speech for him, surely a pet peeve) and how much Go just attempts to not do it. I know the static/dynamic linking thing is a tired argument, I understand the two sides of it, and I understand that engineering is about choosing context specific tradeoffs, but much like with minimal version selection it is funny how clear it is to me that dynamic linking is just not the technology to choose unless forced. Introspecting, I guess this just reveals something to me about my own engineering values.
The above omitted a lot of details, sorry pedants! As penance here is one small detail I ratholed on that might be interesting for you; non-pedants, feel free to skip this section.
In the stub DLLs I first generated code like:
WriteFile:
call _retrowin32_syscall
ret 12
To make it all link, at link time I provided a retrowin32.lib file that lets
the linker know that _retrowin32_syscall comes from retrowin32.dll.
I expected that to generate a call via the IAT, but the code it generated went through one additional indirection. That is, the resulting machine code in the DLL looked like:
WriteFile:
call _retrowin32_syscall ; e8 rel32
...
_retrowin32_syscall:
jmp [IAT entry for retrowin32_syscall] ; ff25 addr
Why did it do this? At least in
this blog's explanation, when
compiling the initial assembly into an object file, the assembler doesn't know
whether retrowin32_syscall is a local symbol that will be resolved at link
time or if it's a symbol coming from a DLL. So it assumes the former and
generates a rel32 call to a local function.
Then, at link time, the linker sees that it actually needs this call to resolve
to an IAT reference, and so it generates this extra bit of code containing the
jmp (which ghidra labels a "thunk").
To eliminate this indirection I instead generate assembly that looks like:
WriteFile:
call [__imp__retrowin32_syscall]
where that __imp_-prefixed symbol is somehow understood to refer directly to
the IAT entry. At the level of C code this is related to the dllimport
attribute, but I am not sure whether that attribute exists at the assembly
level. Peeking at the LLVM source I see it actually has some special-casing
around symbols beginning with the literal string "__imp_", yikes.
2024-09-12 08:00:00
This post is part of a series on retrowin32.
It's been just about two years since I left my job. The Saturday after my last day I opened up my laptop and, without before even considering this idea up to that point, spontaneously decided to try writing a Windows emulator to run an old program I loved.
I didn't know much of anything about how emulators work. I could read the occasional x86 assembly when debugging things but I had never written much of it. I worked with programming Windows a bit a few times over my professional career but I am far from an expert. But overall, how hard could it be, right?
The truth is that there is kind of a lot of detail to all of it. But also, detail ultimately just means it is a slog. x86 has a scrillion opcodes to implement, win32 has scrillion APIs, but the path from zero to a scrillion starts with a step like any other.
I picked up this project with the explicit intent to just follow my interest wherever it led. I was inspired in part by this thread: "Good things happen when I try hard to chase my sense of excitement, ignoring impulses to produce legible outcomes." I think that observation about legibility really reached me. I went through a period in the past where I found I was only reading books that I felt like I ought to be reading and had ultimately been killing my enjoyment of reading, and I was trying to recover that feeling about programming.
I haven't been employed since, in part due to having a hard time as a parent. This is a larger topic for another post, but my son has been a lot; there was even a period where his preschool was having me come in for two hours a day to help manage him. I remain mystified how parents manage careers and children simultaneously, even with easy children. I also try to rationalize that, to the extent I can afford it, if there was ever a time to be there for him it is now.
Does it really take two years to make a not particularly complete emulator? Looking over my commit logs it seems like I only even made commits for about 85% of the weeks I've been working on it, and many other weeks there's only been a few random touches. I have had plenty of other distractions, including video games (I beat Factorio's Space Exploration with some friends, which either means nothing to you or which will impress you a lot), other projects on the side (like my build system experiment, which has since been forked by Android!), vacations, and so on.
Two years in, one thing I have been reflecting on is where I am going with this project. At some point I ought to call it complete, or abandon it. As long as it remains interesting to tinker with I intend to tinker, but I also am well aware of the feeling an unending project that starts to feel like a burden. I am not there yet but I am watching for the signs.
When I first started I imagined the project would be somehow tying my WebAssembly experience into x86 emulation. It turns out that for various reasons JITting into Wasm isn't the most important piece (though the cheerpx people do it) and I never got there. Similarly, my memory of the demoscene — the above program that spawned the whole project is from that world — is from the 90s and I had kind of imagined there would be a rich collection of interesting 2d Windows programs from there. This turned out to not be true. I discovered there are a lot of 2d DOS demos but once Windows came around the demoscene started adopting 3d APIs relatively quickly, and I am not particularly interested in 3d graphics, in part because the look of 90s 3d demos hasn't aged well to my eyes. In other words, the demoscene I had thought I started this project for may have existed only in my imagination.
But at least for now I can say I accomplished something — I got at least the first scene of that original demo working! (Looking now I haven't updated the website to show this; maybe something to do after this post.) The very last bug fix was that I had flipped some math backwards in my MMX implementation. Also, had I known I would need to implement some of MMX, would I have even started this project? Not even sure. I have seen it observed that sometimes not knowing how hard something will be is an important help to actually just starting to try.
The other surprising accomplishment is that after nearly two years of working on it solo (though to properly credit, I've had some really great advice from Dean and Nico!) recently I have had a few contributors show up. In particular somehow the group behind this project found me because they want to run old Windows compilers on Linux cloud machines, but because of the particular thing they do they are also comfortable with disassembly. I have sometimes thought about this: what are the chances of someone having both the low-level skill set needed to usefully contribute, and also the need to emulate an old Windows program? This is to me one of the best things about the internet, where even if such a person is a one in a billion chance, we have a few billion people around on here. And this includes someone else who for whatever reason got Win95's solitaire.exe to run.
