Serializing go types to JSON

Written for a proposal I gave to my coworkers at Monzo.

When building an API endpoint, there is a translation that occurs between Go code and client code. For that to happen, we marshal the go structs into a JSON representation, which is then unmarshalled into the client types.

Due to our setup of using omitempty on non-nullable types, unexpected behaviour happens.

This is best shown with a bool type, as it's the simplest primitive type. (see below for the disclaimer about strings)

TLDR:

An omitempty should only be used with a pointer type to allow three distinct values

true
false
undefined

The Problem

When an omitempty attribute is added to a type that has a valid (useful) zero value, then it makes it impossible for clients to know the intent.

Take for example the boolean flag ShowCurrency on the ListComponentItemMoneyContent type

type ListComponentItemMoneyContent struct {
	Currency         string `json:"currency"`
	AmountMinorUnits int64  `json:"amount_minor_units"`
	// MdsContentColor defaults to "primary" in the client if not provided
	MdsContentColor            string `json:"mds_content_color,omitempty"`
	ShowFractionalPartWhenZero bool   `json:"show_fractional_part_when_zero,omitempty"`
	ShowCurrency               bool   `json:"show_currency,omitempty"`
	ShowSign                   bool   `json:"show_sign,omitempty"`
}

This means that when serialising, there are only two possible values:

true or undefined

// Run this yourself at https://go.dev/play/p/EBrtmdD3Ok_v
type ListComponentItemMoneyContent struct {
	Currency         string `json:"currency"`
	AmountMinorUnits int64  `json:"amount_minor_units"`
	// MdsContentColor defaults to "primary" in the client if not provided
	MdsContentColor            string `json:"mds_content_color,omitempty"`
	ShowFractionalPartWhenZero bool   `json:"show_fractional_part_when_zero,omitempty"`
	ShowCurrency               bool   `json:"show_currency,omitempty"`
	ShowSign                   bool   `json:"show_sign,omitempty"`
}

var content = ListComponentItemMoneyContent{
	Currency:         "GBP",
	AmountMinorUnits: 1200,
	ShowCurrency:     true,
}

marshalledContent, _ := json.Marshal(content)
fmt.Println(string(marshalledContent))

content.ShowCurrency = false
marshalledContent, _ = json.Marshal(content)
fmt.Println(string(marshalledContent))

Outputs:

{"currency":"GBP","amount_minor_units":1200,"show_currency":true}
{"currency":"GBP","amount_minor_units":1200}

This is a problem because client engineers believe they can receive 3 possible values: true, false, or undefined, whereas in reality there can only be 2 values.

And the go expectation is that undefined is really false because I’ve given the value of false either explicitly or implicitly through the zero-value instantiation. But clients may falsely assume that undefined maps to true.

So this is a problem.

Client’s assume defaults

When client engineers can receive 3 possible values, but backend engineers can only create 2 possible values, then clients must make assumptions on what to do with that 3rd option.

For the above case, a backend engineer might assume, “well if you receive undefined, then you should default to false”, but that is not always a good assumption.

This also may hide problems in the data structure, for example if you make a typo in the backend, like show_curency, the interface will be broken, but we might not notice because the frontend client will instead assume that the key is missing and therefore it’s false. This is a good argument for using null instead of undefined to represent the empty state.

ShowCurrency defaults to true, which means that as it is built into the app today, it’s impossible to hide currency. Either the backend supplies true, or the client receives undefined and falls back to true. (false is impossible to achieve)

ShowSign defaults to false, which is maybe more expected, but still implicit.

Check out this Swift code that we had in our iOS app code.

self.shouldShowFractionalPartWhenZero = dictionary["show_fractional_part_when_zero"].to(Bool.self) ?? false
self.shouldShowCurrency = dictionary["show_currency"].to(Bool.self) ?? true
self.shouldShowSign = dictionary["show_sign"].to(Bool.self) ?? false

Also, what if my type really had three options? What if true , false, or undefined are all valid?

type Possible struct {
	// If this is undefined, the client will ask the user before performing the action
	UserPreferenceSomeFeature *bool `json:"user_preference,omitempty"`
}

The problem arises because clients don’t see any difference between this option and the previous. In either case they see a bool that may be undefined. So they have 3 options whether or not it was intended to be so.

And if clients see three possible options, they will handle true, false, and then have to decide what to do when it is undefined. And as we showed here, it’s not always the same.

This problem arises because of the Go annotation.

How can we fix this?

To fix the problem, we can simply be explicit about whether or not a type can actually be undefined.

This means that we never use omitempty with a non-pointer type (see disclaimer)

And if we want to represent possible undefined values, use a pointer!

So for our original example, we can change the type to be the following:

type ListComponentItemMoneyContent struct {
	Currency         string `json:"currency"`
	AmountMinorUnits int64  `json:"amount_minor_units"`
	// MdsContentColor defaults to "primary" in the client if not provided
	MdsContentColor            string `json:"mds_content_color,omitempty"`
	ShowFractionalPartWhenZero *bool  `json:"show_fractional_part_when_zero,omitempty"`
	ShowCurrency               *bool  `json:"show_currency,omitempty"`
	ShowSign                   *bool  `json:"show_sign,omitempty"`
}

For this struct, the zero-value is now no longer a false, it is nil. And thus, backend engineers can actually choose between three possible values.

// Run this code yourself at https://go.dev/play/p/n8se3cpTXhj
type ListComponentItemMoneyContent struct {
	Currency         string `json:"currency"`
	AmountMinorUnits int64  `json:"amount_minor_units"`
	// MdsContentColor defaults to "primary" in the client if not provided
	MdsContentColor            string `json:"mds_content_color,omitempty"`
	ShowFractionalPartWhenZero *bool  `json:"show_fractional_part_when_zero,omitempty"`
	ShowCurrency               *bool  `json:"show_currency,omitempty"`
	ShowSign                   *bool  `json:"show_sign,omitempty"`
}

func main() {

	var content = ListComponentItemMoneyContent{
		Currency:         "GBP",
		AmountMinorUnits: 1200,
		ShowCurrency:     ToPtr(false),
	}

	marshalledContent, _ := json.Marshal(content)
	fmt.Println(string(marshalledContent))

	content.ShowCurrency = nil
	marshalledContent, _ = json.Marshal(content)
	fmt.Println(string(marshalledContent))

	content.ShowCurrency = ToPtr(true)
	marshalledContent, _ = json.Marshal(content)
	fmt.Println(string(marshalledContent))

}

// ToPtr returns a pointer copy of value.
func ToPtr[T any](x T) *T {
	return &x
}

Outputs:

{"currency":"GBP","amount_minor_units":1200,"show_currency":false}
{"currency":"GBP","amount_minor_units":1200}
{"currency":"GBP","amount_minor_units":1200,"show_currency":true}

Now backend engineers can actually specify between 3 possible values, and client engineers can properly see all three options.

Monzo Specific problems

Most dynamic views are not created directly in an API service, they come through one extra translation layer (proto).

As such, this is extra complicated.

While our API could easily be extended to add *bool pointers everywhere, our proto does not currently support optional properties.

This means that while a dynamic view built directly into the API could use an optional bool, all bools that come through RPC will be coerced to be a real value.

Now you might think that this might actually be okay. Our app should not even need to worry about optional bools. Every time a bool isn’t explicitly initialized it’s false.

So, then we might think to fix this we could just remove all the omitempty, and then our app will always get the expected value.

The problem with this is that the app (iOS) will map “undefined” to true in some cases. So while the backend returns a false, it is mapped to undefined, and then the frontend will use a value of true.

If we just change the show_currency to not have omitempty, then across the app, most dynamic views won’t show the currency, which is unexpected. Since for that particular toggle, returning true or false has zero impact on what happens, most developers won’t have included that boolean value as true. As such, removing omitempty will change developer’s previous expectations, even though it is more correct.

To fix this problem, our proto needs to change to allow bool pointers. We can do this by introducing a wrapped boolean

message OptionalBool {
    // The internal value of the boolean. Use this to set the value of the boolean.
    // However, for getting the value, you should normally use the `Value()` or `GetValue()` methods
    // on the OptionalBool type. This will let you check for the presence of the value without needing
    // to first check for nil.
    bool val = 1;
}

With a couple of go helpers in the same package

// helpers.go
package dynamicviewproto

// Value returns the underlying value of the OptionalBool. Or nil if it isn't set.
// This is safe to call on a nil pointer.
func (ob *OptionalBool) Value() *bool {
	if ob == nil {
		return nil
	}
	return &ob.Val
}

// GetValue returns the value of the OptionalBool or false if it is nil.
// You should probably be using Value() instead.
// But if you do want coercion, use this instead of the GetVal() method
func (ob *OptionalBool) GetValue() bool {
	if ob == nil {
		return false
	}
	return ob.Val
}

Then our proto can be updated to have the following for ShowCurrency

message ListComponentItemMoneyContent {
    string amount = 1 [(validation) = "required,libdecimal"];
    // MdsContentColor defaults to "primary" in the client if not provided
    string mds_content_color = 2;
    bool show_fractional_part_when_zero = 3;
    bool show_sign = 5;
    // show_currency_symbol determines whether or not the £ symbol is shown.
    // If you would like the default behavior, leave this as nil.
    // It defaults to true in the client if not provided
    OptionalBool show_currency_symbol = 6;

    reserved 4; // `show_currency` has been replaced by `show_currency_symbol`
}

And now the callers, can decide if they want to pass in true, false, or nil

MoneyContent: &dynamicviewproto.ListComponentItemMoneyContent{
	Amount:                     limit.String(),
	ShowFractionalPartWhenZero: false,
	ShowCurrencySymbol:         &dynamicviewproto.OptionalBool{Val: false},
	MdsContentColor:            dynamicviewproto.MDSContentColorPrimary,
},

// or

MoneyContent: &dynamicviewproto.ListComponentItemMoneyContent{
	Amount:                     limit.String(),
	ShowFractionalPartWhenZero: false,
	ShowCurrencySymbol:         &dynamicviewproto.OptionalBool{Val: true},
	MdsContentColor:            dynamicviewproto.MDSContentColorPrimary,
},

// or

MoneyContent: &dynamicviewproto.ListComponentItemMoneyContent{
	Amount:                     limit.String(),
	ShowFractionalPartWhenZero: false,
	ShowCurrencySymbol:         nil, // default is nil
	MdsContentColor:            dynamicviewproto.MDSContentColorPrimary,
},

Then when we want to serialise it, we can simply marshal using our helper function

	return &ListComponentItemMoneyContent{
		Currency:                   amount.Currency(),
		AmountMinorUnits:           amountMinorUnits,
		MdsContentColor:            p.MdsContentColor,
		ShowFractionalPartWhenZero: p.ShowFractionalPartWhenZero,
		ShowCurrency:               p.ShowCurrencySymbol.Value(),
		ShowSign:                   p.ShowSign,
	}, nil

Now we can properly pass true , false, and undefined from a service all the way to the client.

Discussion about other types

This discussion has focused primarily on bools, however it’s valuable to look at the other types.

The key insight about whether or not you should do what is described in this document is whether or not the zero value of a type is useful on its own. Obviously that means that it depends on the what you are trying to achieve. Context matters! This is not a catch-all.

For a bool, the zero value (false) is meaningful on its own, and doesn’t mean "empty" on its own. This is true of the int types, as well as the float types. 0 doesn’t necessarily mean the absence of a number.

Where it gets particularly tricky are string types. The zero value of a string is "", which does imply emptiness.

For strings, the general assumption would be that an undefined or null string is equal to "" . This is normally true. There’s not a meaningful difference between a blank string and a null string. Think, how would a user even enter a blank string as opposed to null; if user’s can’t distinguish, then it’s probably semantically identical.

For most cases, using omitempty for a string is okay.

My API style opinion While using omitempty is okay for strings. I generally would prefer to not use omitempty. I would rather have a single empty state, rather than create multiple options that clients have to deal with. Many subtle bugs have been introduced this way. I would take the backend code and map nils to "" It is for this same reason that I prefer to return empty arrays rather than undefined or null. While we’re at it, I also generally don’t like to use a mixture of both null and undefined to represent “empty” values. I generally prefer null

This also applies to objects. If you have a struct in your response with omitempty it will never be omitted. It needs to be a pointer too if you want it to actually become undefined.

// Run me at https://go.dev/play/p/IVFRYelFCbk
package main

import (
	"encoding/json"
	"fmt"
)

type Response struct {
	NestedStruct Nested `json:"nested,omitempty"`
}

type ResponseWithPointer struct {
	NestedStruct *Nested `json:"nested,omitempty"`
}

type Nested struct {
	Feature bool `json:"feature"`
}

func main() {
	rsp := Response{}

	responseBody, _ := json.Marshal(rsp)
	fmt.Println(string(responseBody))

	rspWithPtr := ResponseWithPointer{}

	responseBody, _ = json.Marshal(rspWithPtr)
	fmt.Println(string(responseBody))
}

Outputs:

{"nested":{"feature":false}}
{}