Felix Geisendörfer

Blazingly Fast Shadow Stacks for Go

Tue, 28 May 2024 10:19:46 +0000

tl;dr: Software shadow stacks could deliver up to 8x faster stack trace capturing in the Go runtime when compared to the frame pointer unwinding that landed in go1.21. This doesn’t mean that this idea should escape from the laboratory right away, but it offers a fun glimpse into a potential future of hardware accelerated stack trace capturing via shadow stacks.

Inspiration

A couple of days ago I came across an interesting comment from the author of the Bytehound profiler on Hacker News. The comment was in response to somebody asking if the fast stack trace capturing technique mentioned in the project’s README is based on frame pointers. The response was:

No. It’s DWARF based.

The main two tricks are: it preprocesses all of the DWARF info at startup for faster lookups, and it dynamically patches the return addresses of functions on the stack injecting an address to its own trampoline, which allows it to skip going through the whole stack trace every time it needs to dump a backtrace

Background

Before we get into the gnarly details, let’s briefly remember that capturing a stack trace for a native language such as C/C++/Go/Rust is essentially the act of traversing all call frames on the stack and collecting the return addresses that are stored inside of them.

The simplest way to do this is via a compiler (option) that adds a preamble to every function, causing it to push a pointer to the previous frame onto the stack. This effectively turns the stack frames into a linked list and allows for very fast stack trace capturing which I helped to implement for go1.21.

However, for legacy reasons going back to the days when we had only 32bit CPUs with limited registers, compilers other than Go generally omit frame pointers for optimized builds (-O1 or greater). The tide on this is currently turning, but for the time being, in many cases DWARF or other unwind tables have to be used in order to lookup the distance between the frames of the stack.

Since DWARF unwind tables are expressed in terms of opcodes for a Turing-complete VM (I wish I was making this up), it’s pretty common to preprocess this data for better lookup performance. In fact, there is a neat paper that goes as far as directly precompiling these unwind tables into x86_64 code. I’m not sure if that’s the kind of preprocessing the Bytehound profiler does, but it doesn’t really matter.

Because at the end of the day, no matter how well this kind of preprocessing is done, frame pointer unwinding tends to outperform these approaches by at least an order of magnitude. Pitting a table lookup against linked list traversal is simply like bringing a knife to a gun fight … or so I thought.

It turns out that the main advantage of a gun is to quickly attack a target over a distance. But, as we’ll now explore, the second trick mentioned by the Bytehound author is to shrink the distance between the knife and the gun, turning the situation into more of a close-quarter combat.

Shadow Stacks

So, what do I mean by shrinking the distance? Well, it turns out that the act of repeatedly capturing stack traces often involves a lot of duplicated work. That’s because most of the time only a few leaf frames have changed in between two stack traces being captured, yet we end up re-traversing the unchanged frames over and over again. This insight naturally leads to the idea of maintaining a cache of return addresses that allows us to avoid this duplicate work at the cost of maintaining the cache. Such a cache is also know as a “shadow stack”.

One way to maintain such a shadow stack is to push return addresses to both the normal as well as the shadow stack for every function call and to pop addresses from both places upon return. This could be managed by the compiler, but would add a fixed cost to all function calls and returns that is likely to be significantly larger than the act of pushing a frame pointer. Alternatively this could be done by hardware, and it turns out this has recently landed as a security feature for modern Intel CPUs in the Linux kernel.

But as far as I can tell, it’s not yet possible to access these hardware shadow stacks from user space, nor is the technology widely available at this point. So until I saw the comment from Bytehound’s author, I had assumed that shadow stacks would not be a feasible user space technique.

It turns out that I was wrong. An alternative approach is to populate a shadow stack when a stack trace is captured. This eliminates the overhead of having to update the shadow stack for every function call. However, it still leaves the problem of popping return addresses when the function that populated the cache returns. This is where the clever trampoline trick from the Bytehound author comes in. The idea is to exploit the very security problem hardware shadow stacks are supposed to prevent, and overwrite return addresses on the stack to inject a trampoline function that pops an address from the shadow stack before returning control to this address.

I was immediately intrigued by the potential of this idea. In theory it seems like it could amortize most of the costs associated with DWARF unwinding on the happy path. However, I was skeptical how well the approach would fare under what I imagined to be a worst-case workload.

Message from an internal slack channel at work

Implementing Shadow Stacks in Go

One of the things I like about my org at Datadog, is that we do research weeks regularly. During those one week periods, we are encouraged to learn new things or research difficult problems. I didn’t find much time last week for this yet, but trying out a potentially feasible shadow stack technique was too exciting for me to pass up, so I got to work.

My goal was to see if it was possible to use the shadow stack idea to accelerate frame pointer unwinding in Go even further. Unlike DWARF, frame pointers don’t really need any additional acceleration, but I figured it would be a fun challenge.

So I decided to implement a ShadowFPCallers function that combines frame pointer unwinding and shadow stacks, and benchmark it against a regular frame pointer unwinding function called FPCallers.

You can see the full source code here, but the core of the implementation is rather simple. First I added a new shadowStack field to the goroutine struct g. I also declared a new type for it thinking I’d need multiple fields. But I ended up just needing a single pcs []uintptr field.

type g struct {
 ...
 shadowStack
}

type shadowStack struct {
 pcs []uintptr
}

The ShadowFPCallers function is a little more complex, but generally speaking it’s a frame pointer unwinding loop that injects a special shadowTrampolineASM return address and uses it as a marker to realize that the rest of the return addresses can be copied from the shadow stack. It’s also smart enough to incrementally update the shadow stack when the stack depth increases.

func ShadowFPCallers(pcs []uintptr) (i int) {
 const shadowDepth = 2
 var (
 gp = getg()
 fp = unsafe.Pointer(getfp())
 oldShadowPtr unsafe.Pointer
 newShadowPtr unsafe.Pointer
 )
 for i = 0; i < len(pcs) && fp != nil; i++ {
 pcPtr := unsafe.Pointer(uintptr(fp) + goarch.PtrSize)
 pc := *(*uintptr)(pcPtr)
 if i == shadowDepth {
 newShadowPtr = pcPtr
 }
 if pc == abi.FuncPCABIInternal(shadowTrampolineASM) {
 i += copy(pcs[i:], gp.shadowStack.pcs)
 oldShadowPtr = pcPtr
 break
 }
 pcs[i] = pc
 fp = unsafe.Pointer(*(*uintptr)(fp))
 }
 if oldShadowPtr == newShadowPtr {
 return
 }
 if oldShadowPtr != nil {
 *(*uintptr)(oldShadowPtr) = gp.shadowStack.pcs[0]
 }
 gp.shadowStack.pcs = append(gp.shadowStack.pcs[0:0], pcs[shadowDepth:i]...)
 *(*uintptr)(newShadowPtr) = abi.FuncPCABIInternal(shadowTrampolineASM)
 return
}

The shadowTrampolineASM function itself is a thin wrapper around a shadowTrampolineGo function. Together they take care of popping an address from the shadow stack, injecting the trampoline into the parent caller and restoring the popped address in the link register R30 (arm64).

TEXT ·shadowTrampolineASM<ABIInternal>(SB),NOSPLIT|NOFRAME,$0-0
CALL runtime·shadowTrampolineGo<ABIInternal>(SB)
MOVD R0, R30
RET

func shadowTrampolineGo() (retpc uintptr) {
 gp := getg()
 retpc = gp.shadowStack.pcs[0]
 gp.shadowStack.pcs = gp.shadowStack.pcs[1:]
 *(*uintptr)(unsafe.Pointer(getcallersp())) = abi.FuncPCABIInternal(shadowTrampolineASM)
 return
}

Last but not least, I had to teach Go’s lookup table unwinder how to navigate the injected trampoline functions. Without that change, panic(), copystack() and the cpu/memory profiler would break due to the unwinding getting stuck.

func (u *unwinder) resolveInternal(innermost, isSyscall bool) {
 // ...
 if f.funcID == abi.FuncID_shadowTrampolineASM || f.funcID == abi.FuncID_shadowTrampolineGo {
 frame.pc = gp.shadowStack.pcs[0]
 frame.fn = findfunc(frame.pc)
 f = frame.fn
 }
 // ...

It took quite a bit of hacking to get the details right, but as far as I can tell, the implementation seems to be working.

This allowed me to do some benchmarking. The first benchmark shown below is a best-case scenario of capturing stack traces in a loop at a fixed stack depth. As you can see, the ShadowFPCallers approach is 8x faster than the FPCallers implementation which is already really fast 🚀.

Best-case: Regular FPCaller unwinding vs ShadowFPCallers

But as suspected, performance is mixed when it comes to a worst-case benchmark where the stack depth is increased up to a limit and then decreased back down while taking a stack trace upon entering a certain depth level as well as when leaving it. Here it seems like the shadow stack approach is only competitive for stacks that are a little deeper than 32 frames. At lower stack depths, the approach is up to 4x slower 🐌.

Worst-case: Regular FPCaller unwinding vs ShadowFPCallers

I can think of a few ways to mitigate the worst-case scenario. For example, the shadow stack could be disabled for stack depths below 32 frames. Or perhaps we could keep some statistics about the stack depths and use some more advanced heuristics. The implementation itself could probably also be further optimized. But, since this was just a small side project, I didn’t have time to explore this further.

Conclusion

Software shadow stacks are a potentially viable technique for accelerating stack trace capturing. I have only measured the approach on top of frame pointer unwinding in Go, but I would imagine DWARF unwinding to also become very competitive with this acceleration. And while it wasn’t easy to get right, the approach requires relatively little additional code.

However, the approach also comes with a lot of downsides, in particular:

Worst-case performance can be problematic at lower stack depths. (But it’s not clear how often this will be encountered in the wild).
Upcoming hardware security features will likely break this approach.
External unwinders (such as linux perf) will produce broken stack traces when encountering the manipulated return addresses.
Programs using similar bespoke control-flow techniques might break when running them with shadow stacks.
Great care needs to be taken to avoid breaking unwinding for panics, exceptions and similar situations.
Shadow stacks require 8 bytes of additional memory per stack frame. This should usually be fine, but some programs might not want to make such a tradeoff.

Considering that frame pointer unwinding is less than a few percent of overhead for Go’s execution tracer, which is the most demanding use case we have, my conclusion is that the technique is interesting, but not warranted at this point.

My personal hope is that hardware shadow stacks will become widely available and accessible from user space. This is probably still many years away, but once this future arrives, we should be able to get rid of DWARF unwinding and perhaps also frame pointers and enjoy blazing fast stack trace capturing across the board.

Online Discussions

How to Make a Tech Video for YouTube

Mon, 21 Aug 2023 12:41:38 +0000

My team recently shipped a new profiling feature that we’re very excited about. The feature is highly interactive and capable of slicing and dicing through tons of data. This makes it difficult to show the full experience with text and screenshots. So I decided to create a video that shows the tool in action.

This blog post explains how I made this video.

You can see the video above, and I think it turned out reasonably well. The feedback was also positive, including the quote below:

> […] you need to teach the rest of us what goes into making such awesome videos.

I’m definitely no expert, but in this post I will share the things I learned that will help you getting started.

Create a Storyboard

In my opinion, telling a good story is the most important part of making a video. You can get away with poor production quality, but not with a boring story.

To plan my story, I created a storyboard using Google Slides. I initially put some bullets on the slides to create a rough outline. Then I hacked on my demo application to try out the ideas I had and took screenshots to replace the bullets.

Using Google Slides for creating a storyboard

Finally I used the speaker notes to outline the things I wanted to say, as well as other reminders for the recording process.

Using speaker notes for the script and recording notes

I found the storyboard very helpful with planning the recording. In particular it allowed me to split the video into smaller takes that are easier to shoot than longer ones.

However, I didn’t always follow the storyboard religiously and would sometimes make last minute changes to a scene without updating the board. Plans are worthless, but planning is everything.

Recording Strategy

Initially I tried to record my screen first and then record the voice-over tracks separately. This worked, but I ran into two issues:

Too much editing: I had to insert a lot of freeze frames in order to synchronize the things happening on the screen with my voice.
Bad delivery: I struggled with sounding excited and natural when reading from a script.

So I gave up on this approach and recorded the screen and audio for each scene at the same time. But this led to a lot of mistakes. The best way to handle them is to take a deep breath, clap your hands, and then continue where you left off. Don’t stop the recording, even if you have to reset things on the screen and go back a little. Instead edit out the bad parts once you think you have enough good material. To speed up the editing process, look for the distinct patterns of clapping your hands in the audio track:

Clapping is an easy way to leave visual markers in your audio track

I’m happy this approach, but one disadvantage is that it produces longer videos. Speaking freely simply produces more redundant words and pauses than a tightly scripted narration track.

I also found it also hard to say exactly what I had planned, so in some cases I’m not happy with the words I ended up choosing. For example at one point I’m saying “followed by” while talking about sibling nodes in a FlameGraph. This could give listeners the wrong impression that the X-Axis of a FlameGraph represents chronological ordering. But I decided it’s not worth reshooting the whole scene to fix this.

Get Early Feedback

While recording and editing the first version of the video, I started to doubt myself. So instead of continuing, I shared 4 minutes of incomplete material with my colleagues. Based on their feedback, I ended up making many good changes, for example:

Significantly increase the font size. It was unreadable on smaller screens.
Kill the the recap about profiling. Everybody hates videos that take forever to get to the point.
Use a different camera angle for the intro. My hands looked really weird behind the mic.
Don’t wear AirPods, it looked weird.
Don’t try to read out a script word by word, my delivery and intonation is better when speaking freely. Your milage may vary on this.
Collapse the left-side Datadog navigation to get more screen real estate.
Highlight the elements on the screen you are talking about.

You can see the first version of the video that I ended up discarding below:

Early and incomplete footage I shared with colleagues to get feedback

Resolution vs Font Size

You need to make sure your videos can be watched on smaller laptop screens, and ideally phone screens. If you use a large screen for recording, you need to either lower your resolution or increase the font size.

I was mostly recording my browser window in full screen mode. So I ended up increasing the font size. The benefit of this approach is that you don’t need to switch your resolution back and forth.

Either way, I recommend that you shoot everything in 4k. This especially helps with achieving crisp font display.

Software

I used a commercial video recording and editing software called ScreenFlow. It has a few nice features that are specific to editing screen recordings, but they were not important for producing this video. The main feature I liked for this project is the ability to record from multiple screens, cameras and microphones at the same time.

In the past I’ve also used QuickTime Player for recording and the free version of DaVinci Resolve for editing. I highly recommend this combination if you don’t want to spend any money. The only downside is that learning DaVinci takes a little more time. It’s a tool aimed at professionals and not optimized for editing screen recordings.

To create the thumbnail for the video, I extracted a frame from my intro section and removed the background using remove.bg. Then I used photopea (a free web clone of photoshop) to combine it with a screenshot.

Hardware

My current laptop is a maxed out 16" M1 MacBook Pro. I found it significantly more performant than my previous Intel MBPs for editing video. During recording and editing I had the laptop hooked up to a 27 inch LG UltraFine 5K Monitor.

For audio I use a Samson Q2U. It’s a dynamic microphone which means that it’s less prone to pick up background noise compared to a condenser microphone.

For the camera shots I used my iPhone 13 Pro with a tripod. I also used the Continuity Camera feature of macOS to avoid having to transfer the videos I record. For audio I used my Samson Q2U rather than the iPhone to keep the audio track consistent.

It Takes Time

I won’t sugarcoat this. Producing video is hard. Much harder than writing.

Your milage will vary, but I spent almost a week to produce 10 minutes of content. Over half of the time was spent on creating the story as well as iterating on the example application shown in the video. I wrote a lot of code that I ended up throwing away. In particular, I initially attempted to build a full application with multiple endpoints. But then I realized that focusing on one endpoint will create a better story and keep the video shorter.

The rest of the time went into recording and editing. But this part gets easier with experience. Even within this project, I noticed how I was moving much faster towards the end.

Feedback Welcome

The video is definitely far from perfect, but I hope that this post contained some useful information for other non-professional tech video creators.

If you have any questions or suggestions for producing better videos, please leave a comment. Either way, thank you for reading!

Debug Go Request Latency with Datadog's Profiling Timeline

Wed, 09 Aug 2023 14:56:28 +0000

Have you ever found yourself wrestling with a slow p95 request or timeout in a Go application, desperately seeking that elusive root cause? Were you frustrated by the limited capabilities of existing tools? Would you have loved to have a detailed timeline for the involved goroutines showing you how they spend their time running On-CPU or waiting Off-CPU due to networking, disk access, mutex contention, channel operations, garbage collection or scheduler latency?

If yes, you’ll probably be interested in Datadog’s new timeline feature for Continuous Profiler which is now in public beta for Go (as well as for Java and .NET). I’ve been working on this with my colleagues for a few months now, and we’re very excited about rolling it out.

The feature works best if you have basic request instrumentation with Datadog APM. To get started, upgrade to dd-trace-go v1.52.0+ and set the following environment variables:

os.Setenv("DD_PROFILING_EXECUTION_TRACE_ENABLED", "true")
os.Setenv("DD_PROFILING_EXECUTION_TRACE_PERIOD", "15m")

Once this is enabled, you’ll be able to click on slow requests in APM ❶ and navigate to the Code Hotspots ❷ → Timeline ❸ tab in order to see a timeline of every goroutine that was involved in serving this request. In the default zoom level, we aggregate events of the same type into small time buckets that are rendered as colorful area charts ❹.

Viewing the timelines of the goroutines involved in a request

The displayed data can be sliced and diced by selecting specific goroutines, time ranges and visualizations. For example, in the screenshot below I selected goroutine 104 (http.(*conn).serve) ❶ and opened the Flame Graph ❷ in the bottom drawer in order to understand what code is being executed.

Viewing the Flame Graph of an individual goroutine. Running is On-CPU, the rest if Off-CPU.

One thing to notice is the high amount of Unscheduled time ❸ which indicates that the selected goroutine was experiencing scheduler latency. This can happen when all CPUs (GOMAXPROCS) are busy, but that wasn’t the case here.

To figure out the problem, we can zoom in to a 2.5ms time window ❶ and notice a pattern of spending only ~50 microseconds running ❷ before getting blocked by a send operation ❸ followed by a similar amount of unscheduled time ❹.

Zooming into the timeline of a goroutine to see individual scheduling events.

Arguably this behavior from the scheduler doesn’t seem ideal, but instead of going down this rabbit hole (we’ll do it another time), we can simply increase the batch size of the individual messages that are being sent over the channel.

This greatly reduces the number of times the goroutine has to be scheduled, which leads to a much smoother execution schedule and improves the latency of the endpoint by ~10% (335ms → 299ms ❶). This can be seen in the screenshot below, but you can also check out the source code and commits of this application for more details.

Increasing the batch size of our messages eliminated the scheduling latency we saw earlier.

Of course this is just a small example of how you can use this feature. In practice there are many different things that can cause latency problems (network, disk, mutex contention, channel operations, garbage collection, etc.), and it’s not always practical or even feasible to instrument all of them with APM. But with the profiling timeline there is no need for additional instrumentation – we can finally give you full visibility into everything. This should help you to resolve even the most challenging latency problems.

For Go, we’re using the builtin execution tracer (aka runtime/trace), which lets us capture extremely detailed data that goes far beyond our previous approach of correlating CPU profiling data to our distributed traces using pprof labels.

One aspect that we’re very proud of is the low overhead. Historically Go’s execution tracer came with some noticeable overhead, so we contributed patches to the Go runtime to reduce this overhead to < 1% using the forgotten art of frame pointer unwinding. So please upgrade to the recently released go1.21 if overhead is a big concern for you.

To keep data volumes under control, we currently capture these execution traces for up to 1 minute (or 5 MiB) every 15 minutes. If you click on a span without data, our UI will guide you to similar spans that have coverage. Alternatively you can also find such spans by adding the @go_execution_traced:yes filter in the search bar.

At this point we’ve already gotten a lot of value out of this feature internally, but we’re even more excited to hear about your experiences and success stories. Please let us know what you think!

Update: We’ve also released a 10 minute video showing this feature in action, you can check it out below:

Disclaimer: This is my personal blog, so I’m not barking on behalf of my employer in an official capacity here.

Waiting for go1.21: Execution Tracing with < 1% Overhead

Fri, 30 Jun 2023 19:15:32 +0000

Go’s execution tracer (aka go tool trace) has suffered from high overhead since its inception in 2014. Historically this has forced potential users to worry about up to 20% of CPU overhead when turning it on. Due to this, it’s mostly been used in test environments or tricky situations rather than gaining adoption as a continuous profiling signal in production.

However, in early 2023 my colleague Nick Ripley and I started to become convinced that we could change this by implementing frame pointer unwinding. You can read more about the details in the blog post below:

Reducing Go Execution Tracer Overhead With Frame Pointer Unwinding

Since then we’ve been actively collaborating with the Go runtime team at Google, in particular Michael Knyszek, Michael Pratt and Cherry Mui in order to make this a reality. And at this point it looks like we’ve succeeded: The go1.21 release coming out in ~August will feature a new version of the tracer which should provide less than 1% overhead for most applications 🥳 (see upcoming release notes).

Below is a list of directly or indirectly related patches that contributed to this effort:

2023-03-13 runtime: remove NOFRAME from asmcgocall, systemstack and mcall (Quim Muntal)
2023-03-30 runtime: fix frame pointer loop on amd64 (Felix Geisendörfer)
2023-03-30 runtime/trace: enable frame pointer unwinding on amd64 (Felix Geisendörfer)
2023-03-30 runtime/trace: use regular unwinding for cgo callbacks (Felix Geisendörfer)
2023-03-30 runtime: add a benchmark of FPCallers (Felix Geisendörfer)
2023-04-04 runtime/trace: Fix TestTraceSymbolize on solaris and illumos (Felix Geisendörfer)
2023-04-05 runtime: save frame pointer to the stack in signal handlers for arm64 (Nick Ripley)
2023-04-19 runtime: adjust frame pointer on stack copy on ARM64 (Cherry Mui)
2023-04-20 runtime: turn frame pointer unwinding for tracing off by default (Nick Ripley)
2023-04-28 runtime/trace: avoid frame pointer unwinding for events during cgocallbackg (Nick Ripley)
2023-04-30 runtime: fix systemstack frame pointer adjustment (Felix Geisendörfer)
2023-05-02 runtime/trace: enable frame pointer unwinding by default for amd64 and arm64 (Nick Ripley)
2023-05-03 runtime: add test for systemstack frame pointer adjustment (Felix Geisendörfer)
2023-05-03 runtime: remove systemstack logic from adjustframe (Felix Geisendörfer)
2023-05-12 runtime: remove unused skip arg from fpTracebackPCs (Felix Geisendörfer)
2023-05-22 runtime: rename getcallerfp to getfp (Felix Geisendörfer)

So if you’re into Go performance and observability, this improvement is definitely something to look forward to.

And this is just the beginning:

Michael Knyszek has proposed a lot of exciting improvements that will further improve execution tracing in upcoming releases.
Dominik Honnef’s gotraceui viewer keeps improving.
We’ve also been busy with integrating the execution tracer into our profiling product at Datadog. We’ll share more information about this soon.

9 Unexpected Profiling Use Cases Beyond Performance Optimization

Mon, 03 Apr 2023 22:17:31 +0000

Profiling is often associated with performance optimization, but it has many other valuable use cases. Here are 9 of my favorites:

Detect Workload Shifts: Handler frames get wider when they get more work.
Detect Data Nesting: Frames often get deeper when processing more nested data.
Detect Dead Code: If a profile shows a function, you can reject the hypothesis that the code is unused. This can also help with prioritizing security fixes for new vulnerabilities.
Debug Unresponsiveness: I once used profiling to diagnose a PostgreSQL vacuum process stuck in an infinite loop due to index corruption. Loops with O(N^2) scaling issues also usually stand out in profiles.
Identify Configuration Issues: Profiling can be used to identify bottlenecks caused by misconfiguration. For example spending a lot of CPU time in read/write syscalls is often caused by a lack of buffering. Similarly, a high amount of CPU time in TLS can indicate connection churn problems.
Debug Errors: If your error rates get high, your profile will show you the error path in the code.
Verify Deployments: New frames appear, old frames disappear, regressions become visible.
Improve Maintainability: The call hierarchies in your profiles can reveal deficiencies in the structure of your application.
Faster Onboarding: Studying the hot code paths of an application is a good way to ramp up quickly.

Profiling may not be the ideal tool for all of these use cases, but no observability signal can give you such a broad visibility. Logging, tracing and metrics all depend on having the right instrumentation at the right time.

Have you used profiling for any other unexpected use cases? Please share them in the comments, twitter or mastodon.

Disclaimer: I work on profiling for Datadog, but I’m not barking on behalf of my employer here. I’ve been meaning to write this down for a while now since it often comes up in conversations.

FlameScope for Go

Mon, 06 Mar 2023 13:47:23 +0000

FlameScope is a wonderful performance visualization tool from Netflix. If you have never heard about it, you should start by reading Brendan Gregg’s FlameScope Pattern Recognition blog post as an introduction.

But as Go developer, you might have not have had a chance to use it because pprof profiles don’t contain timestamps. Using Linux perf is an option, but that may not always be available in your environment.

However, thanks to a patch from Rhys Hiltner, go1.19+ execution traces include CPU samples with timestamps when when having the CPU profiler enabled. The only thing missing, until now, was a way to extract these CPU samples, so they can be visualized in FlameScope.

So I spend some time this weekend to implement FlameScope support for my traceutils project. You can follow the steps below to install it, and create an example json.stacks file by running the encoding/json benchmarks:

# install traceutils
go install github.com/felixge/traceutils/cmd/traceutils@latest

# record an example trace with CPU samples
go test -bench . -cpuprofile /dev/null -trace json.trace encoding/json

# convert it to the format used by perf script
traceutils flamescope json.trace json.stacks

The next thing you need to do is installing FlameScope itself. The steps below should work if you’re on an M1 MacBook Pro.

# clone repo
git clone https://github.com/Netflix/flamescope
cd flamescope
# add the json.stacks file
mv ../json.stacks examples
# install libmagic
brew install libmagic
# install python deps
pip install -r requirements.txt
# build ui
npm install
npm run webpack
# run flamescope
python run.py

Once you’ve got it up and running, you should be able to select the json.stacks file by hitting the Open as Linux Perf button next to it.

This will bring up the FlameScope heatmap. You can use it to select one of the hot parts of the profile as shown below.

This will bring up a FlameGraph for the selected time range. In this case the RunParallel frame near the bottom of the stack explains why this part of the profile was hot: The benchmark was executing on all of our CPUs.

And that’s it, you can now enjoy FlameScope visualizations using Go.

But please note that this is still relatively experimental and I’ve identified at least one problem while using it so far. Also note that CPU profiling on macOS is a bit broken these days, so you should prefer to collect this kind of data on Linux.

Anyway, hopefully you’ll find this new tooling useful for exploring the performance of your Go applications. Please let me know in the comments, reddit, twitter, or mastodon.

Go arm64 Function Call Assembly

Sun, 05 Feb 2023 09:17:02 +0000

I am currently working on implementing frame pointer unwinding for the Go execution tracer. This involves debugging various problems and crashes caught by TestTraceSymbolize on arm64.

One of these crashes seems to be caused by a goroutine overwriting the frame pointer on the stack of another goroutine. To debug this, I’m trying to study the assembly output of the compiler. (Un)fortunately I don’t read or write Go assembly every day, so this process usually involves chaotic jumping between the following resources:

A Quick Guide to Go’s Assembler (Go’s Assembly Language is quite quirky)
Go ARM64 Assembly Instructions Reference Manual (arm64 specific quirks)
Go internal ABI specification (Go’s register based calling convention)
Arm Architecture Reference Manual (11.952 pages PDF)
Arm A64 Instruction Set Architecture (HTML subset of the above)
ChatGPT (with a very healthy amount of mistrust,- it’s often wrong)
Various Google Search Results

To make this process a little easier for my future self, I decided it’s time to create some high quality notes for my own needs. In particular, I’ll try to explain every assembly instruction the Go compiler (go1.20 darwin/arm64) emits for the following code in great detail:

//go:noinline
func foo() { bar() }

//go:noinline
func bar() {}

To get the assembly, I use the steps shown below.

$ go build main.go
$ go tool objdump -gnu -s 'main.(foo|bar)' ./main

The -s 'main.(foo|bar)' flag filters the output down the two functions we are interested in, and the -gnu flag adds GNU assembly comments which make it easier to lookup instructions in the official arm documentation. Normally I’d also include the -S flag to see each line of source code above the instructions it generated, but that’s not needed here.

After some manual trimming, the objdump output looks like this:

TEXT main.foo(SB) /Users/felixge/Desktop/main.go
MOVD 16(R28), R16 // ldr x16, [x28,#16]
CMP R16, RSP // cmp sp, x16
BLS 8(PC) // b.ls .+0x20
MOVD.W R30, -16(RSP) // str x30, [sp,#-16]!
MOVD R29, -8(RSP) // stur x29, [sp,#-8]
SUB $8, RSP, R29 // sub x29, sp, #0x8
CALL main.bar(SB) // bl .+0x28
LDP -8(RSP), (R29, R30) // ldp x29, x30, [sp,#-8]
ADD $16, RSP, RSP // add sp, sp, #0x10
RET // ret
MOVD R30, R3 // mov x3, x30
CALL runtime.morestack_noctxt.abi0(SB) // bl .+0xffffffffffffdd54
JMP main.foo(SB) // b .+0xffffffffffffffd0
?
?
?
TEXT main.bar(SB) /Users/felixge/Desktop/main.go
RET // ret
?
?
?

Before we dig into this output I want to mention that I’m not using Compiler Explorer for this, because its output seems to be missing important instructions for some reason. I’m also not using go tool compile -S main.go or go build -gcflags=-S ./main.go because those output additional pseudo instructions that are used by the compiler and linker, but don’t remain in the final assembly.

Ok, let’s dig in!

Prologue 1: Check if the stack needs growing

 MOVD 16(R28), R16 // ldr x16, [x28,#16]
CMP R16, RSP // cmp sp, x16
BLS 8(PC) // b.ls .+0x20

Store the value of g.stackguard0 in the R16 register.
Compare the value of R16 with the value of the stack pointer RSP.
If RSP <= R16, jump 8 instructions forward to grow the stack

In Detail

MOVD, aka LDR (immediate), loads the data from the memory address [R28 + 16 ] into R16. The R28 register points to the current goroutine, and the stackguard0 field is found at offset 16 in the g struct because the stack field above it is 16 bytes wide.
CMP, aka CMP (extended register), compares R16 with RSP and stores the results in the condition flags.
BLS, aka B.cond, checks if the condition flags match the condition code ls (lower or same). If yes, the CPU is instructed to jump 8 instructions (0x20 = 32 bytes) forward. We’ll cover those instructions in the Prologue 3: Growing the stack section. However, usually this is not the case, and execution continues with the instructions below.

Prologue 2: Setup foo’s frame

 MOVD.W R30, -16(RSP) // str x30, [sp,#-16]!
MOVD R29, -8(RSP) // stur x29, [sp,#-8]
SUB $8, RSP, R29 // sub x29, sp, #0x8

Store the return address from the link register R30 at 16 bytes below the stack pointer register RSP and update RSP to that location.
Store the frame pointer register R29 of the caller 8 bytes below the stack pointer.
Set the frame pointer register R29 to point to the the previous frame pointer we just pushed onto the stack.

In Detail

MOVD.W, aka STR (immediate), stores the value of the R30 register at the memory address of [RSP - 16]. R30 is the link register which holds the return address of the caller. The operation is pre-indexed, which means that the memory address is computed before the memory is accessed and that the RSP pointer is updated to the computed address afterwards (i.e. RSP = RSP - 16). Technically this frame only uses 8 bytes above RSP (for storing the return address), but the architecture requires the stack pointer to be 16-byte aligned, so 8 bytes of memory are wasted here.
MOVD, aka STUR, stores the value of the R29 register at the memory address of [RSP - 8]. R29 is the frame pointer register that is holding the caller’s frame pointer.
SUB, aka SUB (immediate), subtracts 8 from the stack pointer register RSP and stores the result in the frame pointer register R29. In other words, it points the frame pointer register to the caller’s frame pointer that was just pushed onto the stack by the previous operation.

Body: Call bar

 CALL main.bar(SB) // bl .+0x28

Call main.bar.

In Detail

CALL, aka BL, implicitly adds 4 bytes (the size of an instruction) to the value of the current program counter and stores it in the link register R30. After that it jumps to the first instruction of the main.bar function which is located 10 instructions (0x28 = 40 bytes) below this instruction.

Epilogue: Return from foo

 LDP -8(RSP), (R29, R30) // ldp x29, x30, [sp,#-8]
ADD $16, RSP, RSP // add sp, sp, #0x10
RET // ret

Restore the frame pointer register R29 and the link register R30 from the values stored 8 bytes below the stack pointer RSP.
Restore the stack pointer RSP to its original value by adding 16 to it.
Return to the caller.

In Detail

LDP, aka LDP, loads 16 bytes from the memory address [RSP - 8] into a pair of registers. The first 8 bytes of the data are loaded into the frame pointer register R29, and the second 8 bytes are loaded into the link register R30. This is needed because these registers are call-preserved, so we need to restore them to their original values. Remember: The link register R30 was overwritten by Body: Call bar and the frame pointer register R29 was overwritten in Prologue 2: Setup foo’s frame.
ADD, aka ADD (immediate), adds 16 to the stack pointer register RSP and stores the result in the stack pointer register RSP. This is needed because RSP is another call-preserved register that we overwrote in Prologue 2: Setup foo’s frame.
RET, aka RET returns to the caller. The implied jump location is the value of the link register R30.

Prologue 3: Growing the stack

 MOVD R30, R3 // mov x3, x30
CALL runtime.morestack_noctxt.abi0(SB) // bl .+0xffffffffffffdd54
JMP main.foo(SB) // b .+0xffffffffffffffd0
?
?
?

This is the continuation of Prologue 1: Check if the stack needs growing.

Store the value of the R30 link register in the R3 register.
Call the runtime.morestack_noctxt function to grow the stack.
Jump back to the beginning of the main.foo function.
? indicates zero padding.

In Detail

MOVD, aka MOV (register), copies the value of the link register R30 to the R3 register. This is done to make it available to runtime.morestack_noctxt. As far as I can tell the value is only used to help with printing debug information, especially when a function is tyring to split the stack when it’s not supposed to.
CALL, aka BL, implicitly adds 4 bytes (the size of an instruction) to the value of the current program counter and stores it in the link register R30. After that it jumps to the first instruction of the runtime.morestack_noctxt function which is located 2219 instructions (0xffffffffffffdd54 = -8876 bytes in two’s complement) above this instruction. The called function usually grows the stack of the goroutine before it implicitly returns. Another use case is the preemption of goroutines.
JMP, aka B, jumps jumps to the first instruction of the current function (main.foo) which is located 12 instructions (0xffffffffffffffd0 = -48 bytes in two’s complement) above this instruction. This causes the function to be executed again, now with a big enough stack to hold all of its values.
The three ? each indicate 4 bytes of zero padding which are emitted by the compiler to 16-byte align all function entry points. I’m unable to find an official reference that mentions this value. But since it was provided by an arm employee, and another arm employee added a similar patch to gcc citing performance benefits, this alignment is probably a good idea.

Function bar

 RET // ret
?
?
?

Return to the caller.
? indicates zero padding.

In Detail

RET, aka RET returns to the caller. The implied jump location is the value of the link register R30.
The three ? indicate zero padding as explained above.

Final Thoughts

I only covered arm64 in this article because that’s the architecture used by my laptop and perhaps also the future of the cloud. Writing a similar article for amd64 would be nice, but I’m not sure if I’ll find the time.

Anyway, I hope that this information will be useful to others as well as to my future self. Please let me know if you spot any mistakes or have any questions!

Reducing Go Execution Tracer Overhead With Frame Pointer Unwinding

Tue, 31 Jan 2023 16:57:47 +0000

The Go Execution Tracer (aka runtime/trace) was designed to achieve low enough overhead to be usable on “a server in production serving live traffic”. This is achieved by writing events into per-P buffers, using RDTSC for timestamps, and encoding into a relatively efficient binary format.

However, one achilles heel has remained since 2014:

… 75% of the overhead is stack unwinding.
– Go Execution Tracer design doc

Stack unwinding (aka stack walking) is part of the process for taking a stack trace. It involves iterating over all stack frames and collecting the return addresses (program counters) in each frame. It may also involve expanding this list if some of the program counters are part of inlined function calls.

The other part of taking a stack trace is converting these program counters to function names, file names and line numbers. This is usually called symbolization and you can read more about it here.

So why is stack unwinding so expensive in Go? The short answer is because Go uses a form of asynchronous unwinding tables called gopclntab that require a relatively expensive lookup in order to traverse the stack frames. The gnarly details of this mechanism can be found in the gentraceback function.

It might be possible to optimize the implementation, but it’s an uphill battle that we will not investigate today. Instead we’ll take a look at another approach: The forgotten art of frame pointer unwinding. It’s a technique that has been around for decades, but has fallen victim to relentless compiler optimizations in recent times.

The best way to explain frame pointer unwinding is using a diagram (Fig 1) that shows how to unwind a arm64 stack consisting of 3 frames.

Fig. 1 Traversing a stack of depth 3 using frame pointer unwinding.

The process starts by reading the value of the r29 (arm64) or rbp (amd64) register (aka base pointer). This takes us to the frame pointer of frame 2. One word (8 bytes) above this value sits the return address of the caller, which is the first program counter that we need to collect. After this we follow the value of the frame pointer that takes us to frame 1, allowing us to discover the second return address. Repeating this process one more time leads us to frame 0, which we realize is the root frame because it contains a frame pointer holding the value 0. So at this point we collect the last return address and stop the process.

If you’ve paid close attention, you may realize that we have just traversed a linked list outside of a tech interview (Fig 2).

Fig 2 CS101 finally gets its revenge.

As you can imagine, it’s not very difficult to implement this, so I decided to turn it into a small experimental patch for the Go runtime. Which brings us back to our original motivation: Can this reduce the overhead of the Go execution tracer?

To answer this question, we’ll evaluate the performance of a benchmark called BenchmarkPingPongHog. As the name implies, it essentially measures the performance of two goroutines ping-ponging messages between each other over a pair of unbuffered channels. This is an absolute worst-case for the execution tracer, because each channel operation ends up recording an event with a stack trace.

Exact numbers will vary between different machines, but here is what I usually get when running this benchmark on a beefy c5n.metal EC2 machine with tracing disabled vs enabled (Fig 3).

goos: linux
goarch: amd64
pkg: runtime
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │ gentraceback.txt │
│ sec/op │ sec/op vs base │
PingPongHog-72 642.8n ± 4% 5616.5n ± 3% +773.82% (p=0.000 n=10)

Fig 3 Execution tracer overhead for BenchmarkPingPongHog with gentraceback.

774% overhead, yikes. But let’s dig in. This data was captured while also running the CPU profiler, so we can easily see that gentraceback was responsible for 94% (!) of this overhead (Fig 4).

Fig 4 CPU profile showing 94% of execution tracer overhead in gentraceback.

Now that we’ve established this baseline, let’s see how frame pointer unwinding performs by comparison (Fig 5).

goos: linux
goarch: amd64
pkg: runtime
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │ fp.txt │
│ sec/op │ sec/op vs base │
PingPongHog-72 642.8n ± 4% 834.8n ± 1% +29.87% (p=0.000 n=10)

Fig 5 Execution tracer overhead for BenchmarkPingPongHog with frame pointer unwinding.

30% overhead, that’s 26 times less overhead than gentraceback! I, for one, welcome our new linked list overlords. But before we get ahead of ourselves, let’s take a look at the new CPU profile (Fig 6).

Fig 6 CPU profile showing 9% of execution tracer overhead in fpcallers.

As we can see, fpcallers is now down to 9% of the tracer’s overhead, which means unwinding is no longer the achilles heel of the execution tracer. But of course new bottlenecks such as cputicks (28%) and (*traceStackTable).put (21%) are starting to emerge. You can take a look for yourself by downloading the pprofs.

That being said, even without any further optimization, frame pointer unwinding seems to have the potential to take execution trace overhead from up to 20% to around 1% for realistic workloads which I’m hoping to explore in a follow-up post.

This raises a final question. If this is so easy, why wasn’t the tracer designed to use frame pointers to begin with? The short answer is that the tracer was added in go1.5, but the Go compiler did not emit frame pointer instructions until go1.7.

Of course the potential for using frame pointers to optimize the tracer became obvious right away, so there is an open issue (and CL 33809, CL 212301) for it from 2016. However, it kind of stalled, partially because getting a frame pointer unwinder to behave exactly like gentraceback is difficult, and partially because no decision was made on whether or not that’s a reasonable requirement. Another issue is that frame pointers are only enabled for amd64 and arm64, so gentraceback will continue to be needed for other platforms regardless. That being said, I’m cautiously optimistic that these hurdles can be overcome by the newly formed runtime diagnostics working group.

Last but not least, there seems to be renewed momentum behind enabling frame pointers for C/C++ applications on Linux where compilers historically disabled them for performance reasons. Those arguments made a lot of sense for 32bit CPUs with limited registers, but they are now being reevaluated. So hopefully the forgotten art of frame pointer unwinding will make a comeback, not just in the tracer, but across the industry as whole. The simplicity and observability benefits would be immense.

Update: The frame pointer unwinding discussed in this article was implemented in go1.21, see this follow-up article for more details.

History of Go's Execution Tracer

Fri, 13 Jan 2023 21:17:52 +0000

Today I spent some time studying the early history of Go’s execution tracer. Here is a very brief summary of what I learned.

The design and initial implementation was done by Dmitry Vyukov as outlined in his Go Execution Tracer (Oct 2014) design document. This was followed by the first upstream patch being merged the following January, an announcement at GopherCon in July (Fig 1), and the release in Go 1.5 in August 2015.

Fig. 1 Dmitry briefly announcing the Go Execution Tracer at GopherCon 2015.

Since then a few new events have been added, along with other improvements, but overall the tracer seems to have retained most of its original design and functionality.

Go Execution Trace Size Breakdown

Thu, 12 Jan 2023 22:34:00 +0000

I’m currently trying to get a better understanding of the data produced by Go’s execution tracer (aka runtime/trace). One thing I’m interested in is the breakdown of trace sizes.

For this purpose I’ve just added a new breakdown command (docs) to my traceutils project. I haven’t had a chance to analyze a lot of different traces yet, but it was already very interesting to see the results for a big 280MiB trace that I had laying around.

I didn’t know what to expect, but seeing almost 22% of the trace being made up of EventGoSysCall events was definitely surprising. Perhaps its indicative of poorly buffered I/O operations?

Another trace I looked at spends only 1% of its size on this event type. It will be interesting to see what patterns emerge when looking at more data.

$ traceutils breakdown bytes big.trace
+------------------------+----------+---------+
| EVENT TYPE | BYTES | % |
+------------------------+----------+---------+
| EventGoSysCall | 63.1 MB | 21.84% |
| EventGoCreate | 60.2 MB | 20.86% |
| EventGoStartLocal | 44.6 MB | 15.44% |
| EventGoUnblock | 38.5 MB | 13.34% |
| EventGoStart | 19.8 MB | 6.85% |
| EventGoBlockRecv | 14.7 MB | 5.09% |
| EventHeapAlloc | 14.1 MB | 4.90% |
| EventGoEnd | 11.1 MB | 3.83% |
| EventGoSched | 8.7 MB | 3.00% |
| EventGoUnblockLocal | 4.8 MB | 1.67% |
| EventGoBlockSync | 4.7 MB | 1.63% |
| EventGoSysExit | 1.1 MB | 0.37% |
| EventStack | 783.3 kB | 0.27% |
| EventCPUSample | 497.4 kB | 0.17% |
| EventProcStart | 445.9 kB | 0.15% |
| EventGoBlockSend | 291.5 kB | 0.10% |
| EventGoSysBlock | 284.1 kB | 0.10% |
| EventGCSweepDone | 275.7 kB | 0.10% |
| EventProcStop | 218.9 kB | 0.08% |
| EventGCSweepStart | 202.3 kB | 0.07% |
| EventGoBlockSelect | 184.8 kB | 0.06% |
| EventString | 72.5 kB | 0.03% |
| EventBatch | 36.2 kB | 0.01% |
| EventGoPreempt | 34.4 kB | 0.01% |
| EventGoBlock | 28.1 kB | 0.01% |
| EventGCMarkAssistStart | 27.3 kB | 0.01% |
| EventGCMarkAssistDone | 21.5 kB | 0.01% |
| EventGoStartLabel | 14.8 kB | 0.01% |
| EventGoWaiting | 9.4 kB | 0.00% |
| EventGoBlockNet | 2.1 kB | 0.00% |
| EventGoSleep | 364 B | 0.00% |
| EventGomaxprocs | 49 B | 0.00% |
| EventGCSTWStart | 39 B | 0.00% |
| EventHeapGoal | 28 B | 0.00% |
| EventGCSTWDone | 24 B | 0.00% |
| EventGCStart | 20 B | 0.00% |
| EventGoInSyscall | 16 B | 0.00% |
| EventGCDone | 8 B | 0.00% |
| EventGoBlockCond | 5 B | 0.00% |
| EventFrequency | 5 B | 0.00% |
+------------------------+----------+---------+
| TOTAL | 288.7 MB | 100.00% |
+------------------------+----------+---------+

Testing binary encoders/decoders via incremental output verification

Wed, 11 Jan 2023 15:50:59 +0000

Occasionally I get the chance to work on encoders and decoders for various binary formats. Most recently I worked on traceutils which contains a package for dealing with the data produced by Go’s execution tracer (aka runtime/trace).

One common trick for testing such code is round-trip testing. The idea is to take an input file that covers many relevant edge cases, read it, decode and then re-encode it. If the output is exactly the same as the input, there is a good chance the implementation is correct.

This works great when the test is passing. But unfortunately it’s prone to produce error messages that are too terse as shown in Fig 1 below.

Fig. 1 This is not enough data to understand the problem.

Alternatively one might end up with error messages that dump huge amounts of data. That’s also difficult to reason about as you can see in Fig 2 here.

Fig. 2 Oh no, this is too much data! And that's just a small part of it.

The solution to this problem is simple. Instead of validating the output at the end of the test, one can incrementally verify the output after every decoding and encoding step. If the implementation is correct, the new data produced after every step should be equal to the input data at the same offset. When it’s not, the amount of data that needs debugging becomes much smaller, as can be seen in Fig 3.

Fig. 3 Yes, that's a comfortable amount of binary data to debug!

In this case the problem was caused by the input data containing a padded 10-byte varint representation of the value 18 (0x92 80 80 80 80 80 80 80 80 00) while the encoder produced a more compact representation of the same value (0x12).

Dealing with such cases where the same data has multiple valid representations is a downside of round-trip testing, and maybe I’ll explore dealing with it in another post.

Regardless it should be clear that incremental output verification makes it much easier to debug round-trip test failures. Of course its a trivial idea and I’m certainly not the first one to come up with it, but let me know if you find it useful.

Connecting Go Profiling With Tracing

Fri, 11 Feb 2022 00:00:00 +0100

Today I’d like to share a few thoughts on a few new profiling features we recently shipped at Datadog. I’ll explain what they do, how they work, and which Go 1.18 contributions were needed in order for things to work well.

Feature Showcase

Let’s start with the end result. Imagine you have a 100ms trace where 10ms are spent on a database query, but 90ms remain unexplained.

When this happens, you usually need to study your code for clues. Did you forget some tracing instrumentation? Time to add it, redeploy and wait. Or perhaps you need to optimize your Go code? If yes, how?

This workflow is manageable, but it turns out there is a better way – we can use profiling data to fill in the gaps. And that’s exactly what our new Code Hotspots feature does. As you can see below, our request used 90ms On-CPU time. This is a strong signal that lets us rule out Off-CPU activity such as uninstrumented service calls, mutex contentions, channel waits, sleeping, etc.

Even better, when clicking the “View Profile” button, we can view this On-CPU time as a per-request flame graph. Here we can see that our time was spent on JSON encoding.

And since our HTTP handler functions don’t show up in the stack traces, we can also indirectly infer that this work was done in a background goroutine spawned by the goroutine handling the request.

In addition to breaking down tracing information using profiling, we can also do the opposite and break down a profile’s CPU Time by Endpoint as shown below. The checkboxes can be used to filter the profile by endpoint.

And to make it easier to understand changes over time, e.g. after a deployment, we can graph this data.

How It Works

The features are built on-top of an existing Go feature called pprof labels which allows us to attach arbitrary key/value pairs to the currently running goroutine. These labels are automatically inherited when a new goroutine is spawned, so they reach into all corners of your code. Crucially these labels are also integrated into the CPU profiler, so they automatically end up in the CPU profiles it produces.

So to implement this feature, we modified the tracing code in our dd-trace-go library to automatically apply labels such as span id and endpoint whenever a new span is created. Additionally we take care of removing the labels when the span is finished. A simplified example of this implementation can be seen below:

func StartSpan(ctx context.Context, endpoint string) *span {
 span := &span{id: rand.Uint64(), restoreCtx: ctx}
 labels := pprof.Labels(
 "span_id", fmt.Sprintf("%d", span.id),
 "endpoint", endpoint,
 )
 pprof.SetGoroutineLabels(pprof.WithLabels(ctx, labels))
 return span
}

type span struct {
 id uint64
 restoreCtx context.Context
}

func (s *span) Finish() {
 pprof.SetGoroutineLabels(s.restoreCtx)
}

Our actual implementation is a bit more complex and also reduces the risk of leaking PII (Personally Identifiable Information). It automatically covers the HTTP and gRPC wrappers in our contrib package, as well as any custom traces your application may implement.

Once our backend receives the tracing and profiling information, we’re able to perform the lookups needed to power the features showcased earlier in this post.

Profiling Improvements in Go 1.18

As part of implementing these new features, we did a lot of testing, including unit tests, micro-benchmarks, macro-benchmarks and more. As expected, this surfaced problems in our code and allowed us to quickly fix them.

Somewhat less expected, we also discovered several issues in the Go runtime that were impacting the accuracy of pprof labels, as well as CPU profiling in general. The good news is that with the help of the community, the Go maintainers and contributions from our end – all of these issues have been fixed in the upcoming Go 1.18 release.

If you’re interested in the full details on this, check out the companion post: Profiling Improvements in Go 1.18.

That being said, unless you use a lot of cgo, the new features should already work great for you in Go 1.17.

Feedback Wanted

I’m currently looking for people who are interested in taking these new features for a spin in order to get feedback.

It doesn’t matter if you’re an existing or potential customer, just send me an email and we can set up a 30 min zoom. To sweeten the deal, I’m also happy to answer general Go profiling questions along the way :).

Appendix: Getting Started

If you want to get started quickly, you can simply copy the code below into your application.

Additionally you can check out this fully working dd-trace-go-demo application that shows an integration including HTTP and PostgreSQL.

package main

import (
 "time"

 "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
 "gopkg.in/DataDog/dd-trace-go.v1/profiler"
)

func main() {
 const (
 env = "dev"
 service = "example"
 version = "0.1"
 )

 tracer.Start(
 tracer.WithEnv(env),
 tracer.WithService(service),
 tracer.WithServiceVersion(version),
 tracer.WithProfilerCodeHotspots(true),
 tracer.WithProfilerEndpoints(true),
 )
 defer tracer.Stop()

 err := profiler.Start(
 profiler.WithService(service),
 profiler.WithEnv(env),
 profiler.WithVersion(version),

 // Enables CPU profiling 100% of the time to capture hotspot information
 // for all spans. Default is 25% right now, but this might change in the
 // next dd-trace-go release.
 profiler.CPUDuration(60*time.Second),
 profiler.WithPeriod(60*time.Second),
 )
 if err != nil {
 panic(err)
 }
 defer profiler.Stop()

 // <your application code>
}

Thanks to my colleague Nick Ripley for reviewing this post.

Profiling Improvements in Go 1.18

Fri, 11 Feb 2022 00:00:00 +0100

Without doubt, Go 1.18 is shaping up to be one of the most exciting releases since Go 1. You’ve probably heard about major features such as generics and fuzzing, but this post is not about that. Instead we’ll talk about profiling and highlight a few noteworthy improvements to look forward to.

A point of personal joy for me is that I was able to contribute to several of the improvements as part of my job at Datadog and that they will significantly improve our new Connecting Go Profiling With Tracing functionality.

If you’re new to Go profiling, you might also enjoy our Guide to Go Profiling before diving into this post.

Better CPU Profiling Accuracy

Let’s start with the most significant profiling change in Go 1.18 – the CPU profiler for Linux has become a lot more accurate. In particular, CPU bursts on multi-core systems used to be underestimated, which is fixed now.

This change goes back to late 2019 (go1.13) when Rhys Hiltner opened GH 35057 to report that he noticed a Go service that was using 20 CPU cores according to top, but only 2.4 cores according to Go’s CPU profiler. Digging deeper, Rhys found that this problem was caused by dropped POSIX signals in the kernel.

To understand the problem, it’s worth knowing that Go 1.17’s CPU profiler is built on top the setitimer(2) syscall. This allows the Go runtime to ask the Linux kernel to get a notification for every 10ms the program spends running on a CPU. These notifications are delivered as SIGPROF signals. Each delivery of a signal causes the kernel to stop the program and invoke the runtime’s signal handler routine on one of the stopped threads. The signal handler takes a stack trace of the current goroutine and adds it to the CPU profile. All of this takes less than ~10usec after which the kernel resumes the execution of the program.

So what was wrong? Well, given Rhys program reporting 20 CPU cores being used by top, the expected signal rate should have been 2,000 per second. However, the resulting profile only contained an average of 240 stack traces per second. Further investigation using Linux Event Tracing revealed that the expected amount of signal:signal_generate events were observed, but not enough signal:signal_deliver events. The difference between signal generation and delivery might seem confusing at first, but is due to the fact that standard POSIX signals do not queue, see signal(7). Only a single signal can be pending delivery at a time. If another signal of the same kind is generated while one is already pending, the new signal gets dropped.

However, this still doesn’t explain why so many signals would be generated during the exact same < ~10usec signal handler window so that they would end up on top of each other. So the next piece to this puzzle is the fact that the resolution of syscalls measuring CPU time are limited to the kernel’s software clock which measures time in jiffies, see time(7). The duration of a jiffy depends on kernel configuration, but the default is 4ms (250Hz). What this means is that the kernel isn’t even able to honor Go’s wish of receiving a signal every 10ms. Instead it has to alternate between 8ms and 12ms as can be seen in the histogram below (details).

Since alternating between 8ms and 12ms still works out to 10ms on average, it is not a problem in and of itself. What is a problem however, is what happens when e.g. 10 threads are running on different CPU cores, causing them to use 40ms of CPU time within a 4ms jiffy window. So when the kernel gets around to do its checking, it needs to generate 4 SIGPROF signals at once, causing all except one to be dropped instead of delivered. Or in other words, setitimer(2) – the kernel’s portable API advertised for profiling – turns out to be unsuitable for profiling busy multi-core systems :(.

However, given that the man pages explain the jiffy limitations of CPU time related syscalls and the semantics of process-directed signals, it’s not entirely clear if this would be recognized as a bug by the kernel maintainers. But even if setitimer(2) would be fixed, it would take a long time for people to upgrade their kernels, so Rhys brought up the idea of using timer_create(2) which offers per-thread accounting of pending signals.

Unfortunately the idea didn’t receive feedback from the Go maintainers, so the issue sat dormant until Rhys and I started discussing it again in May 2021 on twitter. We decided to collaborate, Rhys working on the critical CPU profiler changes, and me on better test coverage for cgo and other edge cases shown in the diagram below.

Another side quest of mine was to create a standalone C program called proftest to observe the signal generation and delivery behavior of setitimer(2) and timer_create(2) which confirmed another bias issue from 2016: setitimer(2)’s process-directed signals are not fairly distributed among CPU-consuming threads. Instead one thread tends to get screwed and receives less signals than all other threads. Now, to be fair, signal(7) states that the kernel chooses an arbitrary thread to which to deliver process-directed signals when multiple threads are eligible. But on the other hand macOS doesn’t show this kind of bias, and I think that’s the point where I need to stop making apologies on behalf of the Linux kernel :).

But the good news is that, except for jiffy resolution, timer_create(2) doesn’t appear to suffer from any of the ailments plaguing setitimer(2). Thanks to per-thread signal accounting, it reliably delivers the right amount of signals and shows no bias towards any particular threads. The only issue is that timer_create(2) requires the CPU profiler to be aware of all threads. This is easy for threads created by the Go runtime, but gets tricky when cgo code is spawning its own threads. Rhys patch deals with this by combining timer_create(2) and setitimer(2). When a signal is received, the signal handler checks the origin of the signal and discards it if it’s not the best signal source available for the current thread. Additionally the patch also has some clever bits to deal with avoiding bias against short-lived threads and is just a great piece of engineering in general.

Anyway, thanks to lots of review and feedback from the Go maintainers, Rhys patch was merged and will ship with Go 1.18, fixing GH 35057 and GH 14434. My test cases didn’t make the cut, partially because it’s hard to make non-flaky assertions on profiling data, but mostly because we didn’t want to risk those patches taking upstream’s attention away from the main changes themselves. Nevertheless this open source community collaboration was one of my favorite computer experiences of 2021!

Profiler Label Bug Fixes

Profiler labels (aka pprof labels or tags) allow users to associate arbitrary key/value pairs with the currently running goroutine which are inherited by child-goroutines. These labels also end up in CPU and Goroutine profiles, allowing to slice and dice these profiles by label values.

At Datadog we use profiler labels for Connecting Go Profiling With Tracing and as part of working on this feature I did a lot of testing on them. This caused me to observe stack traces in my profiles that should have labels attached, but didn’t. I proceeded by reporting the issue as GH 48577 and taking a closer look at the root cause.

Luckily the problem turned out to be really simple. The CPU profiler was sometimes looking at the wrong goroutine when adding the label and required essentially just a one-line fix:

-cpuprof.add(gp, stk[:n])
+cpuprof.add(gp.m.curg, stk[:n])

This might be a bit confusing at first since gp is the current goroutine and usually points to the same place as gp.m.curg (the current goroutine of the thread gp is running on). However, the two are not the same when Go has to switch stacks (e.g. during signal handling, or resizing the stack of the current goroutine), so if the signal arrives at an unfortunate moment, gp points to a purely internal goroutine that is executing on behalf of the user, but lacking the labels belonging to the current stack trace stk. Apparently this is a common issue, so it is even documented in the runtime/HACKING.md guide.

Considering the simplicity of the problem, I submitted a one-line patch for it. However, the patch was greeted by the classic OSS response – “Can you please write a test for it?” – delivered by Michael Pratt. I initially doubted the ROI of such a test case, but I couldn’t have been more wrong. It was definitely very difficult to write a non-flaky test to demonstrate the issue, and the initial version of the patch had to be reverted because of this. However, the process of implementing and improving the test case helped Michael to identify two additional issues related to labels, both of which he ended up fixing:

CL 369741 fixed and off-by-one error in the encoding of the first batch of pprof samples causing a small number of samples to be tagged with the wrong labels.

CL 369983 fixed an issue causing system goroutines (e.g. for GC) to inherit labels when spawned from user goroutines.

So thanks to all three problems being fixed, profiler labels will become a lot more accurate in Go 1.18. The biggest impact will be for programs using a lot of cgo since C calls run on a separate stack with gp != gp.m.curg. However, even regular programs experiencing 3-4% missing labels in Go 1.17 will benefit from this change and achieve full accuracy.

Stack Trace Bug Fix

Another profiling related bug that we discovered at Datadog is GH 49171. The problem manifested itself in delta allocation profiles with negative allocation counts. Digging deeper, we encountered non-deterministic program counter (pc) symbolization as a root cause. What this means is that the exact same stack trace (same program counters) would contain different symbols (e.g. filenames) in profiles taken at a different time, which should be impossible.

The bug was also very hard to reproduce in a standalone fashion, and I spend almost a week before succeeding and reporting it upstream. In the end it turned out to be a compiler regression involving inlined closures in Go 1.17 and was fixed by Than McIntosh.

Epilog

So as you can see, in addition to being packed with exciting new features, Go 1.18 also contains several important profiling improvements. Please let me know if I missed anything or if you have any questions or feedback.

Thank you for reading and thanks to my colleague Nick Ripley for reviewing this post.

Advent of Go Profiling: Day 1-1: Branchless Go

Fri, 03 Dec 2021 00:00:00 +0100

It seems like my previous post has struck a chord and many of you have sent your own solutions via twitter. Before I move on to pick another day as a challenge, I’d like to do a little follow up for day 1-1.

Valentin Deleplace’s Solution

Let’s start with looking at the winning solution from Valentin Deleplace which achieves a spectacular 13.8 ns/op on my machine. This is 6.5x faster than my v3 solution and 23x faster than my v1 🤯.

func Answer(input string) (int, error) {
 var prev int64 = -9999
 var increases int = -1

 val := int64(0)
 for p, N := 0, len(input); p < N; p++ {
 c := input[p]
 if c != '\n' {
 val = (val << 8) + int64(c)
 } else {
 if val > prev {
 increases++
 }
 prev = val
 val = 0
 }
 }
 if val > prev {
 increases++
 }
 return increases, nil
}

I knew that my solution was not ideal, but the magnitude of the improvement still surprised me. So I decided to take a closer look to see how it was done:

The call to strings.IndexByte() was retired in favor of directly looping over the individual characters of the input. This works well here because the average line length is very short (4 characters) and calling assembly functions from Go has a non-trivial overhead associated with it.
Somewhat unexpectedly, the function body fits into Go’s inlining budget. Build with -gcflags='-m' if you want to verify this yourself. This eliminates the overhead of calling the Answer function in the benchmark.
Looping over the input is done without using range on the input string which avoids unicode overhead.
Parsing of the input digits is directly integrated into the for loop and avoids iterating over those chars twice.
val << 8 is used instead of val*10. This changes val, but doesn’t break the val > prev comparison.
Instead of using a separate bool to indicate if a previous value has been seen, prev and increases are initialized to -9999 and -1 respectively which is fair game since we noticed the elves hesitance to throw negative numbers at us.
Error handling such as checking for valid digits has been eliminated.

Most importantly Valentin’s solution is far more compact than my v3 and remains pretty readable given the circumstances 👏.

Branchless Go

In my experience code optimization always involves a lot of trial and error. So I think it’s important to acknowledge that optimization ideas often fail and to discuss what we can learn from these failures.

One such failure was my first attempt to further optimize Valentin’s solution by eliminating branches. My inspiration for this came from Daniel Lemire’s Parsing JSON Really Quickly: Lessons Learned presentation as well as Matt Nakama’s Branchless Coding in Go.

I’ve never done this before, so the code below can certainly be improved, but here is what I came up with:

func Answer(input string) (int, error) {
 var prev int64 = -9999
 var increases int = -1

 val := int64(0)
 for p, N := 0, len(input); p < N; p++ {
 c := input[p]
 notNl := boolToInt64(c != '\n')
 increases += int((^notNl & 1) * boolToInt64(val > prev))
 prev = notNl*prev + val*(^notNl&1)
 val = notNl * ((val << 8) + int64(c))
 }
 increases += int(boolToInt64(val > prev))
 return increases, nil
}

func boolToInt64(b bool) int64 {
 return int64((*(*uint8)(unsafe.Pointer(&b))) & 1)
}

The core idea is to cast bool values into 0 or 1 which allows us to take a branch like this:

if c != '\n' {
 val = (val << 8) + int64(c)
} else {
 val = 0
}

And turn it into the equivalent branch-free expression shown below:

val = ((val << 8) + int64(c)) * boolToInt64(c != '\n')

This works because we always compute the expression from the first branch and then multiply it by 1 if the first branch should be taken, or 0 for the second branch which is the same as setting val = 0 directly.

And while boolToInt64(c != '\n') might look like a compilation nightmare, the Go compiler seems to emit something relatively reasonable (see https://godbolt.org/z/Tzr99vqr3):

CMPB R8B, $10 ; compare c and '\n' (ASCII 10)
SETNE "".b+5(SP) ; store 1 on stack if c != '\n' otherwise 0
MOVBLZX "".b+5(SP), R9 ; move result from stack to register R9
ANDL $1, R9 ; last part of int64((*(*uint8)(unsafe.Pointer(&b))) & 1

From my perspective, something like this would be even better:

CMPB R8B, $10 ; compare c and '\n' (ASCII 10)
SETNE R9 ; store 1 in R9 if c != '\n' otherwise 0

But alas — I wasn’t able to get the compiler to generate this. Anyway, since we eliminated all branches, our solution should run very quickly now, right? Let’s have a look:

$ benchstat v4-valentin.txt v5.txt
name old time/op new time/op delta
Answer-6 13.8ns ± 1% 77.1ns ± 0% +460.24% (p=0.008 n=5+5)

Ouch. Turns out a little knowledge is a dangerous thing 🤕. Initially I was pretty sad about this, as I had put a lot of effort into all this “clever” code. But knowing that managing your emotions is often the hardest part — and because it was 3am — I decided to call it a night.

I ended up guessing the problem while riding my bicycle the next day. However I could have probably also figured it out by looking at the View -> Source output of the CPU profile shown below:

As you can see, we’re spending quite a lot of time on line 36 that performs a conditional increases++ operation.

To understand why this is bad, let’s recall the input data we are using:

As well as Valentin’s code we are trying to optimize:

if c != '\n' {
 val = (val << 8) + int64(c)
} else {
 if val > prev {
 increases++
 }
 prev = val
 val = 0
}

As you can see, all lines have 3 digits plus one newline, so the chance to end up in the original else block is only 1/4. For bigger input numbers the chance would be even lower. Additionally the val > prev predicate has to be true, so our unconditional execution of the increases++ operation doesn’t have a very good chance of paying off in most cases.

However, we know that the val > prev predicate by itself is true 7 out of 10 times, so perhaps we can beat the CPU’s branch predictor by limiting ourselves to only eliminating that branch? The code for this is shown below:

func Answer(input string) (int, error) {
 var prev int64 = -9999
 var increases int = -1

 val := int64(0)
 for p, N := 0, len(input); p < N; p++ {
 c := input[p]
 if c != '\n' {
 val = (val << 8) + int64(c)
 } else {
 increases += int(boolToInt64(val > prev))
 prev = val
 val = 0
 }
 }
 if val > prev {
 increases++
 }
 return increases, nil
}

func boolToInt64(b bool) int64 {
 return int64((*(*uint8)(unsafe.Pointer(&b))) & 1)
}

Let’s have a look how this compares to Valentin’s original solution:

benchstat v4-valentin.txt v6.txt .txt
name old time/op new time/op delta
Answer-6 13.8ns ± 1% 12.7ns ± 2% -7.91% (p=0.008 n=5+5)

Awesome, looks like we saved another nanosecond which the elves are going to turn into a gift for the children!

Alright, that’s it for today. I hope this was fun and that I’ll find time to do a post about another AoC challenge soon.

Thanks for reading this!

Appendix: FAQ

I’d imagine the branch-free code above raises quite a few eye brows. To avoid any misunderstanding, let me try to answer a few questions you may have:

Are you recommending to write Go code like this?

Absolutely not. I like to use advent of code to explore and learn. Please do not write production code like this unless “You know what you are doing”™️. I definitely wouldn’t consider myself to belong to this group :).

Will this work for different data distributions?

Probably not! The code is optimized for the small sample input used by AoC to explain the problem.

Why don’t you just implement the algorithm in assembly?

That could be fun! But for this year’s advent of code, I think I’ll only use assembly if it’s in Go’s stdlib.

Advent of Go Profiling: Day 1-1

Wed, 01 Dec 2021 10:00:00 +0100

Tis the season to be geeky, so it’s time for Advent of Code 2021. My theme for last year was to learn some rust, but for this year I decided to focus on Go profiling.

The idea is to code up naive solutions and use Go profiling tools to optimize them. A few folks on twitter were interested in seeing the results as blog posts, so I’m hoping to write a few of these.

You can find all code and profiles in the history of this repository: github.com/felixge/advent-2021.

Disclaimer: I currently work on Continuous Go Profiling for Datadog, but I’m not barking on behalf of my employer here.

v1: Naive Solution (link to code)

The day 1-1 challenge is pretty easy to solve, so I quickly came up with a naive solution, see below.

func Answer(input string) (int, error) {
 var prev struct {
 val int64
 set bool
 }
 var increases int
 for _, line := range strings.Split(input, "\n") {
 val, err := strconv.ParseInt(line, 10, 64)
 if err != nil {
 return 0, err
 }
 if val > prev.val && prev.set {
 increases++
 }
 prev.set = true
 prev.val = val
 }
 return increases, nil
}

This works, but is it fast enough? Let’s run a benchmark.

$ go test -count 5 -run '^$' -bench . -cpuprofile=v1.cpu.pprof > v1.txt
$ benchstat v1.txt
name time/op
Answer-6 325ns ± 1%

While this may not seem bad to you, it’s definitely not Christmas Scale™️, so let’s figure out what we can optimize by looking at our CPU profile as a FlameGraph:

Looks like 44% of our time is spend in strings.Split(), let’s optimize this.

v2: Optimize strings.Split() (link to code)

Looking at the FrameGraph above, notice that strings.Split() calls strings.IndexByte() which is implemented in assembly. Turns out we can call this directly to do our own newline separation:

func Answer(input string) (int, error) {
 var prev struct {
 val int64
 set bool
 }
 var increases int
 var line string
 for len(input) > 0 {
 i := strings.IndexByte(input, '\n')
 if i == -1 {
 line = input
 input = ""
 } else {
 line = input[0:i]
 input = input[i+1:]
 }
 val, err := strconv.ParseInt(line, 10, 64)
 if err != nil {
 return 0, err
 }
 if val > prev.val && prev.set {
 increases++
 }
 prev.set = true
 prev.val = val
 }
 return increases, nil
}

This is a bit more complicated, but let’s see what this gives us:

$ benchstat v1.txt v2.txt
name old time/op new time/op delta
Answer-6 325ns ± 1% 168ns ± 2% -48.25% (p=0.008 n=5+5)

A 48% decrease in execution time — looks like we’re off to a good start! What’s next?

Seems like strconv.ParseInt() has become our new bottleneck.

v3: Optimize strconv.ParseInt() (link to code)

Even though the elves are not stating it explicitly, they are only giving us positive integers. So we could apply the same trick as before and call strconv.ParseUint() directly instead of going through strconv.ParseInt().

However, looking at the implementation of ParseUint it’s clear that it is doing a bunch of stuff to support different bases and bit sizes. We don’t need any of this, so let’s just implement the subset we actually need:

func parseInt(val string) (int64, error) {
 var intVal int64
 factor := int64(1)
 for i := len(val) - 1; i >= 0; i-- {
 c := val[i]
 if c >= '0' && c <= '9' {
 intVal += int64(c-'0') * factor
 } else {
 return intVal, fmt.Errorf("bad int: %q", val)
 }
 factor *= 10
 }
 return intVal, nil
}

Let’s see if it was worth it:

$ benchstat v2.txt v3.txt
name old time/op new time/op delta
Answer-6 168ns ± 2% 92ns ± 3% -45.29% (p=0.016 n=5+4)

Another 45% — we might be reaching Christmas Scale™️ here after all.

Can we do even better?

The FlameGraph above doesn’t give us any obvious clues for what to try next. But we can dig deeper. For example let’s take a look a look View -> Source:

As you can see we’re spending quite a bit of time on the sub-slice operation on line 41. Is this our next clue?

v4: The Nerd Snipe

The elves are informing me that the current solution over 3x faster than v1 — which is good enough for now:

$ benchstat v1.txt v3.txt
name old time/op new time/op delta
Answer-6 325ns ± 1% 92ns ± 3% -71.68% (p=0.016 n=5+4)

However they are willing to offer a special Go Profiling Star to anybody who manages to further improve upon v3. This is where you can come in. Just send me a link to your v4 solution in this twitter thread including your benchstats v3.txt v4.txt output and I will link to it from here.

Valentin Deleplace’s solution with a -87% delta against v3
Gareth Lewin’s solution with a -83% delta against v3
Pontus Leitzler’s solution with a -74% delta against v3
@Fugiman’s solution with a -50% delta against v3

Note: The perf delta results above are self-reported. In my own testing Valentin’s solution achieves a spectacular 14 ns/op and has -52% delta against Gareth’s solution.

That’s it for today. I’ll definitely only find time for doing a few of these posts, but I’ll try to keep them coming.

CSV Boilerplate for Go

Sun, 27 Sep 2020 00:00:00 +0000

tl;dr: Here is my boilerplate for reading and writing CSV files in Go. I hope you’ll find it useful.

Every once in a while, I have to read and/or write CSV files in Go. In theory that’s easy. One just has to include encoding/csv from the standard library, and write a bit of boilerplate to map between a Go struct and the CSV records.

However, the resulting code has often left me unsatisifed for a variety of reasons:

I spend too much time reinventing the wheel, playing around with different ways to structure the code.
The code is annoying to maintain, e.g. changing the output column order often requires modifications in at least 4 places: the struct definition, header writing, record writing, and record reading.
The result is not very robust when it comes to bad CSV input and doesn’t handle problems such as missing, unexpected, or misordered columns or headers.
Error messages often end up missing important information such as row number, column name, etc..
How do I write good tests for this that will be easy to maintain in the future?

One way to deal with this is to use a high level library, and you should certainly consider that. But the reality is often an ugly mess. You might find that the library of your choice doesn’t handle a particular edge case, e.g. more than one header row being present (my bank loves putting in 6 header lines). Or the error messages you’re getting are not helpful enough for your users. Or the error handling is too strict. Or the reflection based mapping is too magic. Or the performance sucks. I could go on and on, but you get the point - dealing with CSV is nasty and calling it a format is a bit of an insult to the idea of a format. From my point of view, writing a library for this problem is simply not possible without creating a horribly overengineered mess or ignoring many edge cases.

So in the end it seems like you have to chose between two suboptimal options. You can try out a bunch of libraries and use and/or fork the one closest to your needs. Or you can roll your own boilerplate for the millionth time, only to look back at it with horrible disgust shortly after.

Your choice should probably depend on the craziness of your input, and I won’t be able to make it for you. But I’ve decided that I want to make the boilerplate route a bit less annoying for myself in the future. To accomplish this I’ve created a boilerplate template that I’ll copy and paste for my future needs, adjusting it as needed. For the rest of the blog post I want to walk you through my thought process in designing it.

To begin, I decided to use real world CSV as my example, containing bits of the typical ugliness one has to deal with. I quickly found a file called cars.csv which looks like this:

Car;MPG;Cylinders;Displacement;Horsepower;Weight;Acceleration;Model;Origin
STRING;DOUBLE;INT;DOUBLE;DOUBLE;DOUBLE;DOUBLE;INT;CAT
Chevrolet Chevelle Malibu;18.0;8;307.0;130.0;3504.;12.0;70;US
Buick Skylark 320;15.0;8;350.0;165.0;3693.;11.5;70;US
Plymouth Satellite;18.0;8;318.0;150.0;3436.;11.0;70;US
...

As you can see, there are already two ugly bits to be examined here: semicolons instead of commas as a separator, and a second header line that seems to contain type information for whatever reason. Not great, not terrible, but pretty typical for what’s out there.

The goals for my boilerplate is to read and write the file format in a way that solves the five problems outlined in the beginning. It should do so without an excessive amount of abstraction, and certainly no reflection, yet as elegantly as possible. There should also be an intermediate Go struct to hold the data in memory as shown below.

type Car struct {
 Car string
 MPG float64
 Cylinders int64
 Displacement float64
 Horsepower float64
 Weight float64
 Acceleration float64
 Model int64
 Origin string
}

Usually my first naive attempt is to write some code that defines the headers and struct mapping in a way that is separate from the CSV encoding/decoding itself. Here’s an example:

var carHeaders = []string{
 "Car",
 "MPG",
 "Cylinders",
 // remaining columns ...
}

func (c *Car) UnmarshalRecord(record []string) error {
 if got, want := len(record), len(carHeaders); got != want {
 return fmt.Errorf("bad column number: got=%d want=%d", got, want)
 }
 c.Car = record[0]
 if c.MPG, err = strconv.ParseFloat(record[1], 64); err != nil {
 return fmt.Errorf("column=%q: %w", carHeaders[1], err)
 } else if c.Cylinders, err = strconv.ParseInt(record[2], 10, 64); err != nil {
 return fmt.Errorf("column=%q: %w", carHeaders[2], err)
 }
 // remaining columns ...
 return nil
}

func (c *Car) MarshalRecord() ([]string, error) {
 return []string{
 c.Car,
 fmt.Sprintf("%f", c.MPG),
 fmt.Sprintf("%d", c.Cylinders),
 // remaining columns ...
 }, nil
}

This works, but adding, removing, or reordering any of the columns requires us to modify our code in 4 to 5 different places, and as I mentioned in the beginning, we want to avoid that. So let’s try some abstraction that allows us to declaratively specify our columns and how to marshal/unmarshal them:

type carColumn struct {
 Name string
 Type string
 UnmarshalValue func(*Car, string) error
 MarshalValue func(*Car) (string, error)
}

var carColumns = []carColumn{
 {
 "Car",
 "STRING",
 func(c *Car, val string) error {
 c.Car = val
 return nil
 },
 func(c *Car) (string, error) {
 return c.Car, nil
 },
 },
 {
 "MPG",
 "DOUBLE",
 func(c *Car, val string) (err error) {
 c.MPG, err = strconv.ParseFloat(val, 64)
 return
 },
 func(c *Car) (string, error) {
 return fmt.Sprintf("%f", c.MPG), nil
 },
 },
 {
 "Cylinders",
 "INT",
 func(c *Car, val string) (err error) {
 c.Cylinders, err = strconv.ParseInt(val, 10, 64)
 return
 },
 func(c *Car) (string, error) {
 return fmt.Sprintf("%d", c.Cylinders), nil
 },
 },
 // remaining columns ...
}

This is of course a bit verbose, but it solves our problem. Reordering our columns is now a single cut & paste operation, and adding or removing a column has gotten a lot easier as well. This approach even gives us a convenient way to put the Type information found in the second line of the CSV file, so we can easily duplicate this quirk when writing our own CSV files later on.

We still need to update our UnmarshalRecord and MarshalRecord from earlier. The good news is that we’re unlikely to ever have to modify this code again:

func (c *Car) UnmarshalRecord(record []string) error {
 if got, want := len(record), len(carColumns); got != want {
 return fmt.Errorf("bad number of columns: got=%d want=%d", got, want)
 }
 for i, col := range carColumns {
 if err := col.UnmarshalValue(c, record[i]); err != nil {
 return fmt.Errorf("column=%q: %w", col.Name, err)
 }
 }
 return nil
}

func (c *Car) MarshalRecord() ([]string, error) {
 record := make([]string, len(carColumns))
 for i, col := range carColumns {
 val, err := col.MarshalValue(c)
 if err != nil {
 return nil, err
 }
 record[i] = val
 }
 return record, nil
}

Now it’s time to implement the CSV decoding itself. My boilerplate implements this via the ReadCarsCSV function that can be seen below.

const carComma = ';'

func ReadCarsCSV(r io.Reader) ([]*Car, error) {
 cr := csv.NewReader(r)
 cr.Comma = carComma

 records, err := cr.ReadAll()
 if err != nil {
 return nil, err
 }

 cars := make([]*Car, 0, len(records))
 for i, record := range records {
 if got, want := len(record), len(carColumns); got != want {
 return nil, fmt.Errorf("row=%d: bad number of columns: got=%d want=%d", i+1, got, want)
 }

 switch i {
 case 0:
 for i, got := range record {
 if want := carColumns[i].Name; got != want {
 return nil, fmt.Errorf("unexpected header column %d: got=%q want=%q", i, got, want)
 }
 }
 case 1:
 for i, got := range record {
 if want := carColumns[i].Type; got != want {
 return nil, fmt.Errorf("unexpected type column %d: got=%q want=%q", i, got, want)
 }
 }
 default:
 car := &Car{}
 if err := car.UnmarshalRecord(record); err != nil {
 return nil, fmt.Errorf("row=%d: %w", i+1, err)
 }
 cars = append(cars, car)
 }
 }

 return cars, nil
}

The code above accomplishes the main task of returning all cars read from the given io.Reader, but it also validates the number of columns, as well as the header and type names found in the first two rows. Error messages should also be good, including both the row number as well the the offending column name in case something goes wrong for one of the records.

It should also be easy to modify. For example if you want to ignore the second header line during reading, just remove the code. Or instead of requiring all headers to have a fixed position, you could dynamically discover their position from the input by matching their names against the names in the carColumns slice and then reorder the elements in the record accordingly.

Or you might decide you need a streaming interface to lower memory usage and GC pressure like this:

type CarReader struct {
 r io.Reader
 // ...
}

func (r *CarReader) Read(c *Car) error {
 // ...
}

The possibilities are endless, and you won’t find yourself backed into a corner by the choice of your library.

Next up is converting our data back to CSV. My solution to that looks like this:

func WriteCarsCSV(w io.Writer, cars []*Car) error {
 cw := csv.NewWriter(w)
 cw.Comma = carComma

 header := make([]string, len(carColumns))
 types := make([]string, len(carColumns))
 for i, col := range carColumns {
 header[i] = col.Name
 types[i] = col.Type
 }
 cw.Write(header)
 cw.Write(types)

 for _, car := range cars {
 record, err := car.MarshalRecord()
 if err != nil {
 return err
 }
 cw.Write(record)
 }
 cw.Flush()
 return cw.Error()
}

As you can see, we take care of writing out the quirky second header line. We also save some code by not checking the errors for every cw.Write() call and handle them by returning cw.Error() at the end instead.

Now we should think about testing. I’m a big fan of the 80/20 rule when it comes to testing, so below is a test that covers the happy path very efficiently.

func TestCSVReadWriteCycle(t *testing.T) {
 in := strings.TrimSpace(`
Car;MPG;Cylinders;Displacement;Horsepower;Weight;Acceleration;Model;Origin
STRING;DOUBLE;INT;DOUBLE;DOUBLE;DOUBLE;DOUBLE;INT;CAT
Chevrolet Chevelle Malibu;18.0;8;307.0;130.0;3504.;12.0;70;US
Buick Skylark 320;15.0;8;350.0;165.0;3693.;11.5;70;US
Plymouth Satellite;18.0;8;318.0;150.0;3436.;11.0;70;US
`)
 wantOut := strings.TrimSpace(`
Car;MPG;Cylinders;Displacement;Horsepower;Weight;Acceleration;Model;Origin
STRING;DOUBLE;INT;DOUBLE;DOUBLE;DOUBLE;DOUBLE;INT;CAT
Chevrolet Chevelle Malibu;18.000000;8;307.000000;130.000000;3504.000000;12.000000;70;US
Buick Skylark 320;15.000000;8;350.000000;165.000000;3693.000000;11.500000;70;US
Plymouth Satellite;18.000000;8;318.000000;150.000000;3436.000000;11.000000;70;US
`)

 for i := 0; i < 2; i++ {
 cars, err := ReadCarsCSV(strings.NewReader(in))
 if err != nil {
 t.Fatal(err)
 }

 buf := &bytes.Buffer{}
 if err := WriteCarsCSV(buf, cars); err != nil {
 t.Fatal(err)
 }
 gotOut := strings.TrimSpace(buf.String())
 if gotOut != wantOut {
 t.Fatalf("\ngot:\n%s\nwant:\n%s", gotOut, wantOut)
 }
 in = gotOut
 }
}

The test first reads the provided in CSV, and makes sure that converting it back to CSV results in wantOut which is a little different because of the way Go formats our floating point numbers. The test then proceeds to treat the output from the first iteration as the input for the second one. This makes sure that our implementation can read the original format, as well as the slightly different output it produces itself.

Of course we could also aim to reproduce the float formatting quirks from the original file. At first glance it seems like all floats are formatted with one digit after the period. However, a closer look reveals that the Weight column has a period, but skips the following digit, e.g. 3504.. I’ve decided to not go down this rabbit hole, but it should be clear that our boilerplate approach puts us in a great position for dealing with CSV quirks like this.

Depending on how serious you are about parsing CSVs, you probably also want to test a few error cases and maybe even throw some fuzzing at this. But I’ve decided the test above is good enough for my boilerplate, so you’ll have to do this part yourself.

Ok, that’s it. I’d love to hear your thoughts, especially if you have ideas for improving it further. Or let me know if you see good alternatives to the idea of copy & pasting a bunch of boilerplate. It’s not like I love it, but as far as I can tell it hits a sweet spot for working with Go in this case.

Thanks to Thorsten Ball and Tomás Senart for reviewing this post and making many valuable suggestions.

Implementing State Machines in PostgreSQL

Thu, 27 Jul 2017 00:00:00 +0000

Finite-state machine (FSM) is a wonderful model of computation that has many practical applications. One of my favorites is turning business logic into simple FSMs. For example consider the following requirements for an order management system:

Orders must not ship before they are paid.
Orders can be canceled, but only if they haven’t shipped yet.
An order that has already been paid, needs to be refunded after canceling.

Informally, turning such a list of requirements into an FSM is simply the process of coming up with: a set of states an order can be in, a set of events, a transition function that maps the combination of each state and event to a new state, and an initial state.

In practice this is usually done by drawing up a directed graph that shows how the different states are connected to each other via events:

Besides guiding our implementation, these graphs can be very useful for discussions and analysis. For example you can immediately see how there is no way for a customer to return an order after it has shipped. Additionally the process of naming the involved states and events automatically creates a precise language that can be adopted by all stakeholders.

You might have noticed that the graph above does not show all possible combinations of all states and all events. The reason is that the missing combinations implicitly lead to an error state. The complete FSM can be shown as a graph as well, but I don’t think it’s very useful in practice.

Anyway, as promised in the title of this article we’re going to implement this FSM in PostgreSQL. This may not be everybody’s cup of tea, but you might like how this approach us gives advanced analytical powers as a free by-product. Embedding this kind of logic into the database can also help protect against race conditions, but this will perhaps be the topic of a future post ¹.

Let’s begin by creating an order_events table which keeps track of all events for a given order_id.

CREATE TABLE order_events (
 id serial PRIMARY KEY,
 order_id int NOT NULL,
 event text NOT NULL,
 time timestamp DEFAULT now() NOT NULL
);

Next, let’s implement the transition function, which is the heart of every FSM:

CREATE FUNCTION order_events_transition(state text, event text) RETURNS text
LANGUAGE sql AS
$$
 SELECT CASE state
 WHEN 'start' THEN
 CASE event
 WHEN 'create' THEN 'awaiting_payment'
 ELSE 'error'
 END
 WHEN 'awaiting_payment' THEN
 CASE event
 WHEN 'pay' THEN 'awaiting_shipment'
 WHEN 'cancel' THEN 'canceled'
 ELSE 'error'
 END
 WHEN 'awaiting_shipment' THEN
 CASE event
 WHEN 'cancel' THEN 'awaiting_refund'
 WHEN 'ship' THEN 'shipped'
 ELSE 'error'
 END
 WHEN 'awaiting_refund' THEN
 CASE event
 WHEN 'refund' THEN 'canceled'
 ELSE 'error'
 END
 ELSE 'error'
 END
$$;

And before we proceed, let’s test with a few examples to make sure the function is working:

SELECT state, event, order_events_transition(state, event)
FROM (VALUES
 ('start', 'create'),
 ('awaiting_payment', 'pay'),
 ('awaiting_payment', 'cancel'),
 ('awaiting_payment', 'ship')
) AS examples(state, event);

 state | event | order_events_transition
------------------+--------+-------------------------
start | create | awaiting_payment
awaiting_payment | pay | awaiting_shipment
awaiting_payment | cancel | canceled
awaiting_payment | ship | error

The above looks correct, but it’s not immediately clear how we could use this function to enforce our FSM on all rows for the same order_id in the order_events table.

The first thing we need is a good way to take a list of events and call our transition function on them recursively to determine the resulting state. There are multiple ways to accomplish this in PostgreSQL, but perhaps the most elegant is a user-defined aggregate.

In a nutshell, a user defined aggregate is a function that has an internal value (state) that is updated for each input value that is being passed into it using a state transition function. This means it fits our FSM model like a glove:

CREATE AGGREGATE order_events_fsm(text) (
 SFUNC = order_events_transition,
 STYPE = text,
 INITCOND = 'start'
);

The above defines a new aggregate called order_events_fsm which takes a text input (one of our events) and calls the order_events_transition state transition function (FSUNC) for each input along with the current state (STYPE) which is also of type text. The initial state is start (INITCOND).

A quick test shows that it works as expected:

SELECT order_events_fsm(event ORDER BY id)
FROM (VALUES
 (1, 'create'),
 (2, 'pay'),
 (3, 'cancel')
) examples(id, event);

 order_events_fsm
------------------
awaiting_refund

Now let’s use our order_events_fsm to create a BEFORE INSERT trigger for our order_events table that makes sure all events of a given order_id are valid and don’t lead to an error state. We do this using a simple plpgsql function that executes our order_events_fsm against all existing events of the current order_id plus the new event. If the final state is error, we raise an exception which causes the current transaction to roll back. It looks like this:

CREATE FUNCTION order_events_tigger_func() RETURNS trigger
LANGUAGE plpgsql AS $$
DECLARE
 new_state text;
BEGIN
 SELECT order_events_fsm(event ORDER BY id)
 FROM (
 SELECT id, event FROM order_events WHERE order_id = new.order_id
 UNION
 SELECT new.id, new.event
 ) s
 INTO new_state;

 IF new_state = 'error' THEN
 RAISE EXCEPTION 'invalid event';
 END IF;

 RETURN new;
END
$$;

CREATE TRIGGER order_events_trigger BEFORE INSERT ON order_events
FOR EACH ROW EXECUTE PROCEDURE order_events_tigger_func();

We can verify that this works by inserting a valid sequence of order events:

INSERT INTO order_events (order_id, event) VALUES
 (1, 'create'),
 (1, 'pay'),
 (1, 'ship');

INSERT 0 3

As well as an invalid sequence of events:

INSERT INTO order_events (order_id, event) VALUES
 (2, 'create'),
 (2, 'ship');

psql:orders.sql:95: ERROR: invalid event
CONTEXT: PL/pgSQL function order_events_tigger_func() line 14 at RAISE

As expected, only the first series of inserts made it into our table:

SELECT id, order_id, event FROM order_events;

 id | order_id | event
----+----------+--------
1 | 1 | create
2 | 1 | pay
3 | 1 | ship

If you’re still on the fence about embedding this kind of logic into your database, let’s see how our approach gives us advanced analytical powers as a free by-product. Let’s consider a new data set of 3 orders:

TRUNCATE order_events;
INSERT INTO order_events (order_id, event, time) VALUES
 (1, 'create', '2017-07-23 00:00:00'),
 (1, 'pay', '2017-07-23 12:00:00'),
 (1, 'ship', '2017-07-24 00:00:00'),

 (2, 'create', '2017-07-23 00:00:00'),
 (2, 'cancel', '2017-07-24 00:00:00'),

 (3, 'create', '2017-07-23 00:00:00'),
 (3, 'pay', '2017-07-24 00:00:00'),
 (3, 'cancel', '2017-07-25 00:00:00'),
 (3, 'refund', '2017-07-26 00:00:00');

Using our order_events_fsm as a window function, we can easily get the state history of a given order:

SELECT time, order_events_fsm(event) OVER (ORDER BY id)
FROM order_events
WHERE order_id = 3;

 time | order_events_fsm
---------------------+-------------------
2017-07-23 00:00:00 | awaiting_payment
2017-07-24 00:00:00 | awaiting_shipment
2017-07-25 00:00:00 | awaiting_refund
2017-07-26 00:00:00 | canceled

But we can go even further and apply our state machines to multiple orders, e.g. by using the generate_series function and a Lateral sub-query to break down the number of orders per state for each day of a given date range:

SELECT date::date, state, count(1)
FROM
 generate_series('2017-07-23'::date, '2017-07-26', '1 day') date,
 LATERAL (
 SELECT order_id, order_events_fsm(event ORDER BY id) AS state
 FROM order_events
 WHERE time < date + '1 day'::interval
 GROUP BY 1
 ) orders
GROUP BY 1, 2
ORDER BY 1, 2;

 date | state | count
------------+-------------------+-------
2017-07-23 | awaiting_payment | 2
2017-07-23 | awaiting_shipment | 1
2017-07-24 | awaiting_shipment | 1
2017-07-24 | canceled | 1
2017-07-24 | shipped | 1
2017-07-25 | awaiting_refund | 1
2017-07-25 | canceled | 1
2017-07-25 | shipped | 1
2017-07-26 | canceled | 2
2017-07-26 | shipped | 1

There you have it, a FSM implemented as a user defined aggregate in PostgreSQL providing both data integrity and advanced analytics.

That being said, your milage may vary, and embedding your business logic into your database is always a tradeoff. But if you want some reassurance: I’ve had great success in applying this approach in combination with eager materialization to implement a realtime analytics dashboard for an application with over a billion event rows.

Anyway, I’m really looking forward to feedback on this, and am more than happy to answer any questions, so please comment.

Join me at Apple: If the article above has you excited, come and join my team at Apple. We’re hiring Go and PostgreSQL developers in Shanghai, China right now. Relocation is possible, just send me an e-mail to find out more about this role.

Thanks to Thorsten Ball and Johannes Boyne for reviewing.

The code in this article is immune to concurrency anomalies when using the SERIALIZABLE transaction isolation level. Alternative you could modify the trigger to aquire an exclusive lock on the order_events table. But as mentioned, this topic deserves a separate post. ↩︎

PostgreSQL operations that you can't EXPLAIN

Tue, 23 May 2017 16:25:00 +0200

I am currently dabbling in PostgreSQL internals in my spare time, including trying to understand the query planner and executor. It’s a deeply humbling experience, but occasionally I’m delighted to be able to answer simple questions I’m researching.

One of these questions is how PostgreSQL is executing ORDER BY clauses inside of aggregate expressions.

# CREATE TABLE foo AS
SELECT * FROM generate_series(1, 3) i;
SELECT 3
# SELECT * FROM foo;
 i
---
 1
 2
 3
(3 rows)

Consider a simple table that holds 3 rows of an int column ranging from 1 to 3:

Now let’s assume we want to convert those rows into a json array. This can be accomplished using the json_agg aggregate function:

# SELECT json_agg(i) FROM foo;
 json_agg
-----------
 [1, 2, 3]
(1 row)

But what if we want to order the array elements in descending order? No problem, we can simply use the order-by-clause support available to all aggregate expressions:

# SELECT json_agg(i ORDER BY i DESC) FROM foo;
 json_agg
-----------
 [3, 2, 1]
(1 row)

If you’re like me, you’d probably imagine the EXPLAIN output for this query to contain three nodes: a Seq Scan, a Sort, and an Aggregate. However, when we run EXPLAIN, it turns out there is no Sort node.

# EXPLAIN SELECT json_agg(i ORDER BY i DESC) FROM foo;
 QUERY PLAN
-------------------------------------------------------------
 Aggregate (cost=41.88..41.89 rows=1 width=32)
 -> Seq Scan on foo (cost=0.00..35.50 rows=2550 width=4)
(2 rows)

In fact, the plan even remains unchanged even if we change the sort order or remove the clause entirely:

# EXPLAIN SELECT json_agg(i ORDER BY i ASC) FROM foo;
 QUERY PLAN
-------------------------------------------------------------
 Aggregate (cost=41.88..41.89 rows=1 width=32)
 -> Seq Scan on foo (cost=0.00..35.50 rows=2550 width=4)
(2 rows)

# EXPLAIN SELECT json_agg(i) FROM foo;
 QUERY PLAN
-------------------------------------------------------------
 Aggregate (cost=41.88..41.89 rows=1 width=32)
 -> Seq Scan on foo (cost=0.00..35.50 rows=2550 width=4)
(2 rows)

But how is this possible? If there is no Sort node, how does the aggregate perform its sorting?

Well, it turns out that aggregate plan nodes can perform their own sorting. These sorts do not explicitly show up in EXPLAIN, but we can trace them using the trace_sort developer option:

# SET trace_sort=true;
SET
# SET client_min_messages='log';
SET
# SELECT json_agg(i ORDER BY i DESC) FROM foo;
LOG: begin datum sort: workMem = 4096, randomAccess = f
LOG: performsort starting: CPU 0.00s/0.00u sec elapsed 0.00 sec
LOG: performsort done: CPU 0.00s/0.00u sec elapsed 0.00 sec
LOG: internal sort ended, 25 KB used: CPU 0.00s/0.00u sec elapsed 0.00 sec
 json_agg
-----------
 [3, 2, 1]
(1 row)

Another way to see that the sort is part of the plan is to use debug_print_plan option, but the output is not for the faint of heart. Specifically you need to look for the :aggorder field below:

# SET debug_print_plan=true;
SET
# SELECT json_agg(i ORDER BY i DESC) FROM foo;
LOG: plan:
DETAIL: {PLANNEDSTMT
 :commandType 1
 :queryId 0
 :hasReturning false
 :hasModifyingCTE false
 :canSetTag true
 :transientPlan false
 :dependsOnRole false
 :parallelModeNeeded false
 :planTree
 {AGG
 :startup_cost 41.88
 :total_cost 41.89
 :plan_rows 1
 :plan_width 32
 :parallel_aware false
 :plan_node_id 0
 :targetlist (
 {TARGETENTRY
 :expr
 {AGGREF
 :aggfnoid 3175
 :aggtype 114
 :aggcollid 0
 :inputcollid 0
 :aggtranstype 2281
 :aggargtypes (o 23)
 :aggdirectargs <>
 :args (
 {TARGETENTRY
 :expr
 {VAR
 :varno 65001
 :varattno 1
 :vartype 23
 :vartypmod -1
 :varcollid 0
 :varlevelsup 0
 :varnoold 1
 :varoattno 1
 :location 16
 }
 :resno 1
 :resname <>
 :ressortgroupref 1
 :resorigtbl 0
 :resorigcol 0
 :resjunk false
 }
 )
 :aggorder (
 {SORTGROUPCLAUSE
 :tleSortGroupRef 1
 :eqop 96
 :sortop 521
 :nulls_first true
 :hashable true
 }
 )
 :aggdistinct <>
 :aggfilter <>
 :aggstar false
 :aggvariadic false
 :aggkind n
 :agglevelsup 0
 :aggsplit 0
 :location 7
 }
 :resno 1
 :resname json_agg
 :ressortgroupref 0
 :resorigtbl 0
 :resorigcol 0
 :resjunk false
 }
 )
 :qual <>
 :lefttree
 {SEQSCAN
 :startup_cost 0.00
 :total_cost 35.50
 :plan_rows 2550
 :plan_width 4
 :parallel_aware false
 :plan_node_id 1
 :targetlist (
 {TARGETENTRY
 :expr
 {VAR
 :varno 1
 :varattno 1
 :vartype 23
 :vartypmod -1
 :varcollid 0
 :varlevelsup 0
 :varnoold 1
 :varoattno 1
 :location -1
 }
 :resno 1
 :resname <>
 :ressortgroupref 0
 :resorigtbl 0
 :resorigcol 0
 :resjunk false
 }
 )
 :qual <>
 :lefttree <>
 :righttree <>
 :initPlan <>
 :extParam (b)
 :allParam (b)
 :scanrelid 1
 }
 :righttree <>
 :initPlan <>
 :extParam (b)
 :allParam (b)
 :aggstrategy 0
 :aggsplit 0
 :numCols 0
 :grpColIdx
 :grpOperators
 :numGroups 1
 :aggParams (b)
 :groupingSets <>
 :chain <>
 }
 :rtable (
 {RTE
 :alias <>
 :eref
 {ALIAS
 :aliasname foo
 :colnames ("i")
 }
 :rtekind 0
 :relid 404407
 :relkind r
 :tablesample <>
 :lateral false
 :inh false
 :inFromCl true
 :requiredPerms 2
 :checkAsUser 0
 :selectedCols (b 9)
 :insertedCols (b)
 :updatedCols (b)
 :securityQuals <>
 }
 )
 :resultRelations <>
 :utilityStmt <>
 :subplans <>
 :rewindPlanIDs (b)
 :rowMarks <>
 :relationOids (o 404407)
 :invalItems <>
 :nParamExec 0
 }

LOG: begin datum sort: workMem = 4096, randomAccess = f
LOG: performsort starting: CPU 0.00s/0.00u sec elapsed 0.00 sec
LOG: performsort done: CPU 0.00s/0.00u sec elapsed 0.00 sec
LOG: internal sort ended, 25 KB used: CPU 0.00s/0.00u sec elapsed 0.00 sec
 json_agg
-----------
 [3, 2, 1]
(1 row)

In conlusion: While EXPLAIN is a powerful weapon for those seeking to understand query performance, you should be aware of the fact that there are things that you can’t EXPLAIN :).

The research for this article was done using PostgreSQL 9.6.3 and LLDB, but should also apply to recent versions as well as the upcoming PostgreSQL 10.

# SELECT version();
 version
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.6.3 on x86_64-apple-darwin14.5.0, compiled by Apple LLVM version 7.0.0 (clang-700.1.76), 64-bit
(1 row)

Killing a child process and all of its children in Go

Wed, 23 Sep 2015 14:05:08 +0000

This blog post explores some common unix child process control use cases, and how to perform them in Go. I’m mostly sharing it to improve my own understanding of the subject, but hopefully this will be useful to others as well.

Problem: You are using os/exec to run a child process, but need to terminate it under certain conditions, e.g. after a timeout of 3 seconds.

Solution: Just call cmd.Process.Kill() when your condition triggers:

cmd := exec.Command("sleep", "5")
start := time.Now()
time.AfterFunc(3*time.Second, func() { cmd.Process.Kill() })
err := cmd.Run()
fmt.Printf("pid=%d duration=%s err=%s\n", cmd.Process.Pid, time.Since(start), err)

$ go run example1.go
pid=41066 duration=3.00070126s err=signal: killed
$ ps -j
USER PID PPID PGID SESS JOBC STAT TT TIME COMMAND
felix 41008 41002 41008 0 0 Ss s000 0:00.07 -/bin/bash

Problem: The child process is executing a child process of its own (they grow up so fast), but unfortunately the solution above only kills the child, but not the grandchild:

cmd := exec.Command("/bin/sh", "-c", "watch date > date.txt")
start := time.Now()
time.AfterFunc(3*time.Second, func() { cmd.Process.Kill() })
err := cmd.Run()
fmt.Printf("pid=%d duration=%s err=%s\n", cmd.Process.Pid, time.Since(start), err)

$ go run example2.go
pid=41216 duration=3.00145737s err=signal: killed
$ ps -j
USER PID PPID PGID SESS JOBC STAT TT TIME COMMAND
felix 41008 41002 41008 0 0 Ss s000 0:00.11 -/bin/bash
felix 41217 1 41212 0 0 S s000 0:00.00 watch date
$ kill 41217

As you can see, the watch process is still alive and has been adopted by init (PPID = 1). The reason for this is that, under-the-hood, Go is is using kill(2) to send a KILL signal to the PID of the sh process, but not the watch process, turning it into an orphan. This is completely normal behavior for a unix programming language, but not desirable for many use cases.

Solution: In addition to sending a signal to a single PID, kill(2) also supports sending a signal to a Process Group by passing the process group id (PGID) as a negative number. Child processes get the same PGID as their parents by default, so in order to avoid suicide, we’ll also need to ask Go to create a new process group for our child by setting the Setpgid field of syscall.SysProcAttr. Putting it all together, we get this:

cmd := exec.Command("/bin/sh", "-c", "watch date > date.txt")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
start := time.Now()
time.AfterFunc(3*time.Second, func() {
 syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
})
err := cmd.Run()
fmt.Printf("pid=%d duration=%s err=%s\n", cmd.Process.Pid, time.Since(start), err)

$ go run example3.go
pid=41352 duration=3.001809822s err=signal: killed
$ ps -j
USER PID PPID PGID SESS JOBC STAT TT TIME COMMAND
felix 41008 41002 41008 0 0 Ss s000 0:00.14 -/bin/bash

Internally this will cause Go to call setpgid(2) between fork(2) and execve(2), to assign the child process a new PGID identical to its PID. This allows us to kill all processes in the process group by sending a KILL to -PID of the process, which is the same as -PGID. Assuming that the child process did not use setpgid(2) when spawning its own child, this should kill the child along with all of its children on any *Nix systems.

Further reading: Process control in unix a is a complex topic, so I recommend the resources below to learn more.

Advanced Programming in the UNIX Environment, 3rd Edition
My unix bible.
https://www.win.tue.nl/~aeb/linux/lk/lk-10.html
Perhaps a bit dated, but free and easy to understand.

I remember when it was possible to understand Unix.
— Rob Pike (talking about process control)

Originally published on Medium on September 23, 2015.

Go + Raspberry Pi powered Alarm Clock

Thu, 05 Mar 2015 16:30:00 +0100

Invented in 1959, the snooze button is the worst invention in the world. Not only does it defeat the point of setting an alarm, it may actually leave you more tired. Unfortunately however, my zombie morning brain doesn’t care about this, and any alarm within reach has a very poor chance of actually getting me out of bed, trapping me into an open-end snooze fest.

For this reason I usually set an alarm below my loft bed, so I’m forced to climb down the stairs to disable it. This works, but sometimes I forget to set this alarm before getting into bed, and my sleepy evening brain convinces itself that setting the alarm on the phone next to me will do just fine.

And this is exactly how I missed an important early morning conference call this week.

To avoid this from happening again, I finally implement an idea I had a while ago for for turning my raspberry pi into a smartphone controlled alarm clock that I can set from my bed, but only disable by getting up. I’ve published the Go code as Rise 1.0 on GitHub, and you can see the result below:

In a way, this is latest incarnation of a previous AirPlay based project of mine, which unfortunatley suffered from a few issues causing me to abondon it. However, I feel pretty good about this implementation, and already woke up successfully with it this morning.

Given enough interest, I’d be happy to polish this up a bit further and make it easy to install on any Raspberry Pi or other Linux machine you have laying around. Just let me know what you think!

Using TCP keepalive with Go

Tue, 26 Aug 2014 15:40:00 +0100

If you have ever written some TCP socket code, you may have wondered: “What will happen to my connection if the network cable is unplugged or the remote machine crashes?”.

The short answer is: nothing. The remote end of the connection won’t be able to send a FIN packet, and the local OS will not detect that the connection is lost. So it’s up to you as the developer to address this scenario.

In Go you have several methods available to you that can help with this. Perhaps the first one to consider is the SetReadDeadline method of the net.Conn interface. Assuming that your connection is expected to receive data at a regular interval, you can simply treat a timed out read as equivalent to an io.EOF error and Close the connection. Many existing TCP protocols support this way of error handling by defining some sort of heartbeat mechanism that requires each endpoint to send PING/PONG probes at a regular interval in order to detect both networking problems, as well as service health¹. Additionally such heartbeats may also help dealing with proxy servers that look for network activity to determine the health of a connection.

So if your protocol supports heartbeats, or you have the ability to add heartbeats to your own protocol, that should be your first choice for addressing the unplugged network cable scenario.

However, what happens if you have no control over the protocol, and heartbeats are not supported?

Now it’s time to learn about TCP keepalive and how to use it with Go. TCP keepalive is defined in RFC 1122, and is not part of the TCP specification itself. It can be enabled for individual connections, but MUST be turned off by default. Enabling it will cause the network stack to probe the health of an idle connection after a default duration that must be no less than two hours. The probe packet will contain no data², and failure to reply to an individual probe MUST NOT be interpreted as a dead connection, as individual probe packets are not reliably transmitted.

Go allows you to enable TCP keepalive using net.TCPConn’s SetKeepAlive. On OSX and Linux this will cause up to 8 TCP keepalive probes to be sent at an interval of 75 seconds after a connection has been idle for 2 hours. Or in other words, Read will return an io.EOF error after 2 hours and 10 minutes (7200 + 8 * 75).

Depending on your application, that may be too long of a timeout. In this case you can call SetKeepAlivePeriod. However, this method currently behaves different for different operating systems. On OSX, it will modify the idle time before probes are being sent. On Linux however, it will modify both the idle time, as well as the interval that probes are sent at. So calling SetKeepAlivePeriod with an argument of 30 seconds will cause a total timeout of 10 minutes and 30 seconds for OSX (30 + 8 * 75), but 4 minutes and 30 seconds on Linux (30 + 8 * 30).

I found that situation rather unsatisfying, so I ended up creating a small package called tcpkeepalive that gives you more control:

kaConn, _ := tcpkeepalive.EnableKeepAlive(conn)
kaConn.SetKeepAliveIdle(30*time.Second)
kaConn.SetKeepAliveCount(4)
kaConn.SetKeepAliveInterval(5*time.Second)

Currently only Linux and OSX are supported, but I’d be happy to merge pull requests for other platforms. If there is interest from the Go core team, I’ll also try to contribute these new methods to Go itself.

Please let me know if you found this article useful, have any questions, or spotted any errors so I can correct them.

Appendix

Tuning a heartbeat mechanism to detect failures early with a low false positive rate is tricky business. Checkout the ϕ Accrual Failure Detector for a statistically sound model, as well as Damian Gryski’s go-failure implementation. Unfortunately there is no way to use it with TCP keepalive that I can think of.
According to RFC 1122 keepalive probes may contain a single garbage octet for compatibility with broken implementations. However, I’m not sure if this is filtered out by the OS network stack or not, please comment if you know.

GoDrone - A Parrot AR Drone 2.0 Firmware written in Go

Wed, 25 Dec 2013 20:24:00 +0100

Merry Christmas (or Newtonmas if you prefer) everybody.

Today I’m very happy to release the first version of my favorite side project, GoDrone.

GoDrone is a free software alternative firmware for the Parrot AR Drone 2.0. And yes, this hopefully makes it the first robotic visualizer for Go’s garbage collector : ).

At this point the firmware is good enough to fly and provide basic attitude stabilization (using a simple complementary filter + pid controllers), so I’d really love to get feedback from any adventurous AR Drone owners. I’m providing binary installers for OSX/Linux/Windows:

http://www.godrone.io/en/latest/index.html

But you may also choose to install from source.

Depending on initial feedback, I’d love to turn GoDrone into a viable alternative to the official firmware, and breathe some fresh air into the development of robotics software. In particular I’d like to show that web technologies can rival native mobile/desktop apps in costs and UX for providing user interfaces to robots, and I’d also like to promote the idea of using high level languages for firmware development in linux powered robots.

If you’re interested, please make sure to join the mailing list / come and say hello in IRC:

http://www.godrone.io/en/latest/user/community_support.html

Vim Trick: Open current line on GitHub

Thu, 08 Aug 2013 20:50:00 +0100

Update: The fugitive vim plugin supports the same feature via :Gbrowse - thanks to @CrypticSwarm for the tip!

In my never-ending quest to automate my work flow, I recently came up with a neat little vim trick I’d like to share.

One of the things I do quite frequently, is pasting links to GitHub files/lines in email and chat conversations, e.g. Protocol.js#L144. So far my workflow for this has been to navigate to the GitHub repo, hit “t” to select the file, click on the line I want, and then copy the address bar location into my clipboard.

After doing this several times in a row the other day, I decided to automate it. The first piece of the puzzle was to create a git alias for determining the GitHub URL of the current repository to put into my ~/.gitconfig file:

[alias]
 url =! bash -c 'git config --get remote.origin.url | sed -E "s/.+:\\(.+\\)\\.git$/https:\\\\/\\\\/github\\\\.com\\\\/\\\\1/g"'

This allows me to easily get the URL of the current repository I am in, e.g.:

$ git url
https://github.com/felixge/node-mysql

Now I can do cool things like quickly opening this URL in my browser:

$ git url | xargs open
# or
$ open `git url`

But that still requires me to manually navigate to the file / line I am currently interested in, and I’m lazy. So I came up with this key binding for my ~/.vimrc:

nnoremap <leader>o :!echo `git url`/blob/`git rev-parse --abbrev-ref HEAD`/%\#L<C-R>=line('.')<CR> \| xargs open<CR><CR>

Now I can simply press “,o” ("," is my leader key), and my browser will automatically navigate to the file/line underneath my cursor.

Feel free to adopt this for your editor / environment and let me know if you make any improvements. For example, one thing I didn’t get around to yet is opening visual selections as line ranges.

Let's Fix File Uploading

Wed, 27 Mar 2013 15:52:00 +0100

tl;dr: We are creating an open source project to provide a turnkey file uploading solution called tus. Today we are happy to show you our first resumable file uploading demo.

It’s 2013, and adding reliable file uploading to your app is still too damn hard. And if content is king, this poses a serious problem to those interested in acquiring it. In this post I’ll describe the current uploading landscape and how we are planning to fix it.

Resumable Uploads

Have you ever tried uploading a big file over an unreliable internet connection? Depending on the site you were using, the experience can be extremely frustrating. Except for a few bigger sites, network errors will force you to restart your upload from the beginning. Even worse, some sites will be unable to detect this problem, leaving it up to you to figure out why the progress bar is stuck.

The solution of course is to offer resumable file uploads, however, this is easier said than done. While there is no shortage of bits and pieces that can help you with building your own solution, there is no simple solution that you can get up and running in a few minutes.

The first thing you’ll need to figure out is a server side API. Google has published several http protocols for resumable uploading (e.g. YouTube, Google Drive), but they all rely on a non-standard http code 308 Resume Incomplete which is on collision course with upcoming standards. Amazon S3 offers multipart uploads (not to be confused with multipart/form-data), but they are hard to use and the minimum chunk size is 5MB which is far too large to be useful under bad networking conditions.

And even if you figure out the server side, you’ll still need to take care of the client side. For HTML5, your best bet is to start out with something like jQuery File Upload which has some basic support for resumable uploads, but you’ll still have a lot of tweaking ahead of you in order to make it work with your server side API. If you are also running native iOS / Android apps, you will be entirely on your own to come up with a solution.

Fuck this. It’s 2013, and you shouldn’t have to waste several days for what will only be a rudimentary solution, hard to scale, and a pain to maintain. This is where we come in with tus. Tus is an open source project that aims to provide you with a turnkey solution to all your file uploading needs. We are designing a protocol, building a server, and are working on clients for HTML5, iOS and Android. Once these are solid, we will also provide a hosted service, making it even easier for people to get going, as well as allowing us to recoup our investment in this project.

Today is a great day to have a first look, as we have just completed our first resumable file uploading demo for HTML5. The demo is pointed at a tusd instance running on EC2, and uses jQuery File Upload under the hood. What’s exciting about it, is that it not only handles network errors, but also browser crashes and user errors like closing the page. No matter what happens, a previous upload will always continue where it left off.

So check it out, and give us your feedback. We are also very interested to hear what other issues you’d like to see solved. If you are not sure about the possibilities, just read on.

A world of possibilities

Resumable file uploads are just the tip of the iceberg, and there are many more problems ripe for a solution. However, we need your help in prioritizing them, so please let us know which of the things below you would find valuable.

File upload acceleration: There are many situations where file uploads fail to achieve optimal performance due to latency and TCP limitations. We hope to tackle this from multiple angles: Hosting a reverse CDN to improve latency regardless of user location, utilizing multiple TCP connections to increase the effective tcp window size, and maybe in the far future implement a specialized UDP protocol.
Huge files: With HD cameras on the rise, uploads become bigger and bigger. A good solution should be able to handle files > 2GB with ease.
Streaming: A lot of files can be processed as a stream, so tus should make it easy to let you download and process an active upload.
Service Integrations: It should be easy to let your users select files from their Dropbox, Facebook, Google Drive or other cloud services.
Processing pipelines: Most files are media files and require additional processing before being consumed by other users. Tus should easily integrate with existing processing solutions such as Transloadit, Zencoder and others.
Scalability: It should be easy to run a cluster of tus instances, allowing you to add or remove capacity as needed.
Security: HTTPS is a no-brainer, but we are hoping to raise the bar by supporting file encryption on the fly for applications that can benefit from it.
Protocols: HTTP is not the only protocol out there. We are interested in exploring uploads via FTP, E-Mail and other commonly used protocols.

If any of the above sounds exciting to you, please let us know. For now our priority is to release a solid server along with great client libraries for HTML5, iOS and Android, but at the end of the day it will be your feedback that will drive our roadmap.

The Pull Request Hack

Mon, 11 Mar 2013 16:43:00 +0100

After writing about Open Source And Responsibility, I’d like to share a small collaboration hack I came up with last year. Here it is:

Whenever somebody sends you a pull request, give them commit access to your project.

While it may sound incredible stupid at first, using this strategy will allow you to unleash the true power of Github. Let me explain. Over the past year, I realized that I could not allocate enough time to my open source projects anymore. And I’m not even talking about fixing bugs or adding features, I’m talking about pull requests piling up.

So why wouldn’t I simply merge them? Well, a lot of them were actually not good enough. They were lacking tests or documentation, violated coding standards, or were introducing new issues the contributor had not considered. I would often explain the problems in detail, only to find that the contributor was now lacking the time to make the changes. Of course I could make those adjustments myself, but that would often take as much time as if I had done the patch myself to begin with. So after seeing this pattern play out many times over, I started to neglect most incoming pull requests that couldn’t be merged right away.

And then I came across the hack I mentioned above. I wish I could take credit for designing it, but it really happened by coincidence. Somebody sent a pull request for a project I was no longer using myself, and I could see an issue with it right away. However, since I no longer cared about the project, and the person sending the pull request did, I simply added him as a collaborator and said something like this: “I don’t have time to maintain this project anymore, so I gave you commit access to make any changes you’d like. It would be nice to follow the style used by the rest of the code and add some tests to this patch.”.

The result was pure magic. Within a few hours the same person who had just submitted a rather mediocre patch, had now fixed things up and committed them. This was highly unusual, so I started using the same strategy for a few other small projects I was no longer interested in maintaining. And it worked, over and over again. Of course, sometimes it wouldn’t make a difference, but it was clearly working a lot better than my previous approach.

Given the success for my smaller projects, I eventually decided to also try it for my two most popular projects, node-mysql and node-formidable. Initially I was very worried about giving up control over these projects, but the results speak for themselves. Both projects are now maintained by a bunch of amazing developers, writing much better code than I ever received in the form of pull requests before.

So why does it work? Well, I have a few ideas. Once people have commit access, they are no longer worried that their patch might go unmerged. They now have the power to commit it themselves, causing them to put much more work into it. Doing the actual commit/push also changes their sense of ownership. Instead of handing over a diff to somebody else, they are now part of the project, owning a small part of it.

But the magic does not stop here. In addition to their contribution quality going up, I’ve observed many people continuing to help out with issues and patches sent by other users. This is of course fueled by Github notifying every contributor on a repository of all activity on it.

So should you really do this for all pull requests? Probably not. While I’ve given a large amount of users access to various projects of mine, I’m still looking for:

Github profile: Does this user stand to lose a reputation by doing something stupid?
Skill: Based on the patch, do I think the user could be a good developer?
Usefulness: Is this patch solving a valid problem?

With these checks in place, I think this approach is a fantastic way to keep projects from going stale as well as turning one man projects into small communities. However, you’re obviously free to run your projects any way you’d like.

Last but not least, I’d like to thank a few of the great people who have made significant contributions to some of my projects recently:

Diogo Resende for maintaining the hell out of node-mysql.
Oz Wolfe for contributing a connection pool to node-mysql.
Nate Lillich for improving the connection pool for node-mysql.
Sven Lito for fixing bugs and merging patches for node-formidable.
@egirshov for contributing many improvements to the node-formidable multipart parser.
Andrew Kelley for also helping with fixing bugs and making improvements to node-formidable.
Mike Frey for contributing JSON support to node-formidable.
Alex Indigo for putting serious amounts of work into improving and maintaining node-form-data.
Domenic Denicola for doing a lot of improvements on node-sandboxed-module.
Everybody else who I forgot to mention or who made smaller contributions.

You guys are absolutely amazing and I can’t thank you enough for all the help.

Update: Looking back at it, talking to Peter Hintjens at MixIT was definitley an inspiration for this.

Update 2: There is an interesting discussion about this on Hacker News right now, seems like others have used this strategy with success as well!

Update 3: There is even more discussions going on at Reddit.

Open Source And Responsibility

Thu, 07 Mar 2013 15:00:00 +0100

Today I read a comment on Github about an important feature that is missing in one of my open source libraries. I won’t link to it, but it basically said: “If you care about your library and the community, you must implement this feature”. These were not his exact words, but author of the comment was clearly demanding more gratis attention for his problem based on my responsibility to the community.

And he is not alone. Several people in the discussion agreed with him, and there are many public examples of people going as far as suggesting that open source authors have parental responsibilities for their projects.

Tom Dale has suggested that you should not “release more than you can realistically maintain”. And that “it takes maturity to realize that open source is a responsibility”.

I disagree.

Open source is not perfect. And we would be better off accepting this.

Don’t get me wrong. I have been on both sides of this table, experiencing critical problems with open source software that I was unable to fix myself, receiving no support. It sucks.

But whose fault is it? We can actually objectively answer this question. When using somebody else’s software, I need to follow the law, more specifically copyright. So I have to check for a license file and comply with it. Which usually means I’ll come across something like this:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

This is the ugly side of open source. Nobody is responsible for the vast majority of projects.

And unfortunately simply demanding open source authors to become more responsible does not work very well. I mean you can try, but hopefully I can give you some advise that will yield better results.

In my opinion, the key to successfully leveraging open source projects, is to become a responsible consumer. By that I mean acquiring the ability to estimate the “total costs of ownership” that arise from relying on somebody’s open source project. The idea being that you “cannot consume more than you can afford”.

There are many approaches to this, but here are a few questions that help me in this process:

How many other people are using this software? Does it work for them?
What is the worst case scenario if this software fails on me?
Do I have the ability to debug and fix such problems?
What does the license say?
What is the quality of the code?
Is good documentation available?
Are there automated tests?
Is the project maintained by a community or by a single author?
How does the author of the software deal with bug reports?
What does my own Github profile look like? Will I be taken seriously?
Does the author accept contributions?
Is the author available for consulting / support?
Who is his employer, how does he make money?
What is his motivation for writing this software?
Could I pay somebody else to fix issues with this software?
Do I have enough money to pay somebody?
Can I easily use a modified version of this software in production?
Is it viable to fork this software if the author is not cooperating?
What features do I actually need?
How much would it cost to implement the features I need from scratch?

And while it’s not pleasant, this approach has led me to realize, that in some cases, I simply couldn’t afford certain software, even so it was offered to me at no charge.

Of course this process is problematic, and there has to be a better way. We need to find better incentives for open source authors to take on more responsibility, while continuing to enjoy the benefits of low marginal costs.

Luckily I have a few ideas for this, and going forward, I will explain how we are using them for our next product, and how to make money with open source as a small bootstrapped company.

However, becoming a responsible consumer will continue to pay off, so please consider it.

Hello World

Sun, 03 Mar 2013 16:00:00 +0100

After writing over 300 articles between 2006 - 2009, I went on a bit of a blogging hiatus for the past three years. This was mostly because node.js took off, and being one of the first contributors sent a steady stream of speaking opportunities my way. This was a lot of fun, but at this point I am a bit sick of traveling, and I have also realized that conferences are not a good platform to share some of the things I’m currently excited about. So for 2013 I’ve decided to cut my traveling to a minimum, using this time to do a lot more writing instead.

So what will I be writing about?

Well, if you’re following me on twitter, you may have noticed, that I’ve been talking a lot about Go lately. That’s because after a weekend of fooling around with it, I’ve started using it quite a bit over the past few months, and have become very excited about it. So there are many Go related things I’d like to share here in the future, including how it compares to node.js.

Additionally Go has inspired me to come up with a new product and business model we’re currently pursuing at transloadit. This is exciting because I’ve never written much about the process we used to bootstrap our business, and our new product should be an exciting case study for anybody interested in bootstrapping software businesses without relying on external funding. You may also be interested in the fact that our new product will be 100% open source, and that I am planning to write a lot about the underlaying strategy and business model.

And for those who are curious, the product itself is going to be a dedicated file upload server called tus, so expect to read a lot about the challenges of file uploading, and how we’re tackling them with Go.

Last but not least, I’m still heavily involved with the NodeCopter movement that Robin, Tim, Thorsten, Kida, Matti and I started last year, so as time allows, I’ll write about quad copter hacking and the events we’re organizing.

So, that’s it for now, but please make sure to subscribe to new posts if you’re interested in Making Money with Open Source, Go, File Uploads or Quad Copter hacking - it should be interesting.