WebAssembly is a powerful tool for bringing command line utilities to the web and giving people the chance to tinker with tools. This tutorial is about building a standalone, fast WebAssembly based command line tool using the Emscripten tool chain.
This post is derived from the original source at OpenSource.
Another application of WebAssembly is command line playgrounds, where users are free to play with a simulated version of a command line tool. In this article, we’ll explore a concrete example of leveraging WebAssembly for this purpose, specifically to port the tool jq—which is normally confined to the command line—to run directly in the browser.
If you haven’t heard, jq is a very powerful command line tool for querying, modifying, and wrangling JSON objects on the command line.
Aside from WebAssembly, there are two other approaches we can take to build a jq playground:
- Set up a sandboxed environment on your server that executes queries and returns the result to the user via API calls. Although this means your users get to play with the real thing, the thought of hosting, securing, and sanitizing user inputs for such an application is worrisome. Aside from security, the other concern is responsiveness; the additional round trips to the server can introduce noticeable latencies and negatively impact the user experience.
These two solutions are not ideal because we have to choose between security and a meaningful learning experience. Ideally, we could simply run the command line tool directly in the browser, with no servers and no simulations. Lucky for us, WebAssembly is just the solution we need to achieve that.
Set up your environment
In this article, we’ll use the Emscripten tool to port jq from C to WebAssembly. Conveniently, it provides us with drop-in replacements for the most common C/C++ build tools, including gcc, make, and configure.
Instead of installing Emscripten from scratch (the build process can take a long time), we’ll use a Docker image I put together that comes prepackaged with everything you’ll need for this article (and beyond!).
Let’s start by pulling the image and creating a container from it: # Fetch docker image containing Emscripten
# Fetch docker image containing Emscripten docker pull robertaboukhalil/emsdk:1.38.26 # Create container from that image docker run -dt --name wasm robertaboukhalil/emsdk:1.38.26 # Enter the container docker exec -it wasm bash # Make sure we can run emcc, Emscripten's wrapper around gcc emcc --version
If you see the Emscripten version on the screen, you’re good to go!
Porting jq to WebAssembly
Next, let’s clone the jq repository:
git clone https://github.com/stedolan/jq.git cd jq git checkout 9fa2e51
Note that we’re checking out a specific commit, just in case the jq code changes significantly after this article is published.
Before we compile jq to WebAssembly, let’s first consider how we would normally compile jq to binary for use on the command line.
From the README file, here is what we need to build jq to binary (don’t type this in yet):
# Fetch jq dependencies git submodule update --init # Generate ./configure file autoreconf -fi # Run ./configure ./configure \ --with-oniguruma=builtin \ --disable-maintainer-mode # Build jq executable make LDFLAGS=-all-static
Instead, to compile jq to WebAssembly, we’ll leverage Emscripten’s drop-in replacements for the configure and make build tools (note the differences here from the previous entry: emconfigure and emmake in the Run and Build statements, respectively):
# Fetch jq dependencies git submodule update --init # Generate ./configure file autoreconf -fi # Run ./configure emconfigure ./configure \ --with-oniguruma=builtin \ --disable-maintainer-mode # Build jq executable emmake make LDFLAGS=-all-static
If you type the commands above inside the Wasm container we created earlier, you’ll notice that emconfigure and emmake will make sure jq is compiled using emcc instead of gcc (Emscripten also has a g++ replacement called em++).
So far, this was surprisingly easy: we just prepended a handful of commands with Emscripten tools and ported a codebase—comprising tens of thousands of lines—from C to WebAssembly. Note that it won’t always be this easy, especially for more complex codebases and graphical applications, but that’s for another article.
Let’s generate that glue code from the executable file jq that emmake outputs:
# But first, rename the jq executable to a .o file; otherwise, # emcc complains that the "file has an unknown suffix" mv jq jq.o # Generate .js and .wasm files from jq.o # Disable errors on undefined symbols to avoid warnings about llvm_fma_f64 emcc jq.o -o jq.js -s ERROR_ON_UNDEFINED_SYMBOLS=0
To make sure it works, let’s try an example from the jq tutorial directly on the command line:
# Output the description of the latest commit on the jq repo $ curl -s "https://api.github.com/repos/stedolan/jq/commits?per_page=5" | node jq.js '..commit.message' "Restore cfunction arity in builtins/0nnCount arguments up-front at definition/invocation instead of doing it atnbind time, which comes after generating builtins/0 since e843a4f"
And just like that, we are now ready to run jq in the browser!
Although there are similar web apps that let you execute arbitrary jq queries in the browser, they are generally implemented as server-side applications that execute user queries in a sandbox (option #1 above).
Instead, by compiling jq from C to WebAssembly, we get the best of both worlds: the flexibility of the server and the security of the browser. Specifically, the benefits are:
- Flexibility: Users can “choose their own adventure” and use the app with fewer limitations
- Speed: Once the Wasm module is loaded, executing queries is extremely fast because all the magic happens in the browser
- Security: No backend means we don’t have to worry about our servers being compromised or used to mine Bitcoins
- Convenience: Since we don’t need a backend, jqkungfu is simply hosted as static files on a cloud storage platform
WebAssembly is a powerful tool for bringing existing command line utilities to the web. When included as part of a tutorial, such playgrounds can become powerful teaching tools. They can even allow your users to test-drive your tool before they bother installing it.
If you want to dive further into WebAssembly and learn how to build applications like jqkungfu (or games like Pacman!), check out the book Level up with WebAssembly.