Safer Binary Decoding in Go

Posted on in Engineering by Blain Smith

Go is a popular language choice for building web services. Typically, when building those web services, we end up encoding/decoding JSON as the data format. The encoding/json package provides a safe way to turn JSON payloads into Go structs, and vice versa.

However, if we need to handle raw []byte that follow a binary encoding format that is not self-describing, we need to do a bit more work and implement the encoding.BinaryMarshaler and encoding.BinaryUnmarshaler directly. Since we're dealing with []byte, we need to respect slice bounds to avoid triggering a panic() and crashing our service.

Let's look at the two ways we can decode data into Go structs and compare how one way will be safer than the other while yielding the same result. As an added bonus, we'll end up with easier to understand code.

ICMP Packet Format

To make this a bit more fun and challenging, lets decode a packet format we all tend to take for granted, Internet Control Message Protocol or ICMP. If you have every used ping, then you've sent ICMP packets in order to test and measure a remote host.

Reading the ICMP specification, we see that the packet format is as follows for Echo and Echo Reply messages only, since they are more interesting than some of the other ICMP payloads with fewer fields.

  • Type (1 byte)
  • Code (1 byte)
  • Checksum (2 bytes)
  • ID (2 bytes)
  • Sequence Number (2 bytes)

Another way to read these payloads is to look at the protocol layout in ASCII, like you see in the ICMP RFC:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Type      |     Code      |          Checksum             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Identifier          |        Sequence Number        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Data ...
+-+-+-+-+-

In this view, we can visualize the bounds of the fields by number of bits. For example, we see that Code occupied bits 8 to 15 which is 8 total bits or 1 byte.

Defining the Struct and Decoding Function

Now that we know the layout of the binary data, we can define our Go struct. Notice we are defining explicit sizes to our fields and not using regular int types. We explicitly want uint8 or uint16 types for our fields so we can properly decode it in binary.

type Echo struct {
  Type        uint8
  Code        uint8
  Checksum    uint16
  Identifier  uint16
  SequenceNum uint16
}

Next, we can set up the stub decoding function. We define the function with this name and signature to implement the encoding.UnmarshalBinary interface to make this flexible throughout the Go standard library.

func (e *Echo) UnmarshalBinary(buf []byte) error {
  // Read through buf and assign to Echo fields
  return nil
}

Decoding by Subslicing the []byte

First, we are going to perform the decoding by subslicing the []byte into our required uint8 and uint16 fields.

func (e *Echo) UnmarshalBinary(data []byte) error {
  if len(data) < 8 {
    return errors.New("invalid packet size")
  }

  e.Type = data[0]
  e.Code = data[1]

  e.Checksum = binary.BigEndian.Uint16(data[2:4])
  e.Identifier = binary.BigEndian.Uint16(data[4:6])
  e.SequenceNum = binary.BigEndian.Uint16(data[6:8])

  return nil
}

We can assign Type and Code fields directly since a byte is just a uint8. You can see this in the type definition for byte. However, since the rest of our fields are uint16, we need to take 2 bytes and convert both of them at the same time to a single uint16 value using big endian.

Looking back at our protocol format, we see that Checksum occupies bits 16 to 32 which is a total of 16 bits or 2 bytes. That is why we are taking bytes 2 up to, but not including 4 (2 and 3 only). We follow the same pattern for the rest of the uint16 fields.

You might be thinking this is a perfectly good way to decode a byte slice into a struct since we're also checking the length of the incoming slice to ensure we won't slice out of bounds and cause a panic. While you would be correct, it is a bit awkward to reason about the slicing semantics as you're scanning the code. The buf[2:4] isn't very clear if your goal is to scan the code and understand it quickly. What if this payload had 5-10 times as many fields? Slicing could lead to incorrect field bounds. Fortunately, there is a clearer and safer way to perform the same decoding.

Deocoding with bytes.Buffer and binary.Read

For the clearer and safer method, we use bytes.Buffer and binary.Read. Converting our BinaryUnmarshal function now looks like this.

func (e *Echo) UnmarshalBinary(data []byte) error {
  buf := bytes.NewBuffer(data)

  if err := binary.Read(buf, binary.BigEndian, &e.Type); err != nil {
    return err
  }

  if err := binary.Read(buf, binary.BigEndian, &e.Code); err != nil {
    return err
  }

  if err := binary.Read(buf, binary.BigEndian, &e.Checksum); err != nil {
    return err
  }

  if err := binary.Read(buf, binary.BigEndian, &e.Identifier); err != nil {
    return err
  }

  if err := binary.Read(buf, binary.BigEndian, &e.SequenceNum); err != nil {
    return err
  }

  return nil
}

Now we have a much clearer and safer decoding function that reads from the bytes.Buffer that wraps the data []byte, and since our struct fields contain the correct sizes (uint8 and uint16), we no longer have to keep track of the subslicing indicies. Even though we have much more of the dreaded if err != nil Go error checking, we end up with panic free code since binary.Read will safely return an io.EOF should something bad happen. We'll end up catching that error and returning it to the caller.

Bonus: Decoding the Entire struct at Once

Since Echo's fields are all fixed sizes, we can pass the entire thing into binary.Read and get the same result as above, but with a single call.

func (e *Echo) UnmarshalBinary(data []byte) error {
  buf := bytes.NewBuffer(data)

  return binary.Read(buf, binary.BigEndian, &e)
}

This method works great for structs that have all fixed sized fields and the binary data on the wire matches our struct, but that will not always be the case. Sometimes we'll have to account for n-length strings and n-repeated fields.

Variable Length Strings

For these examples, let's assume we have a new message that is encoded as binary.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Identifier  |  HostnameLen  |          Hostname ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

Since we have a variable length string Hostname, the length is first encoded before the string data itself to tell us how many bytes we should read that represent the hostname.

Knowing this, we can now set up our type to store the relevant fields like we did with Echo.

type Message struct {
  Identifier uint16
  Hostname   string
}

func (m *Message) UnmarshalBinary(data []byte) error {
  buf := bytes.NewBuffer(data)

  // Decode Identifer the same we did for others above in Echo
  if err := binary.Read(buf, binary.BigEndian, &m.Identifier); err != nil {
    return err
  }

  // Read the HostnameLen into a temporary variable
  var hostnameLen uint16
  if err := binary.Read(buf, binary.BigEndian, &hostnameLen); err != nil {
    return err
  }

  // Make a slice with sizing it with the value of the temporary variable
  // from above and read the next n bytes into it. Finally, we convert
  // that []byte into string and set the field on the Message.
  hostname := make([]byte, hostnameLen)
  n, err := buf.Read(hostname)
  if err != nil {
  	return err
  }
  if n != int(hostnameLen) {
  	return io.ErrUnexpectedEOF
  }
  m.Hostname = string(hostname)

  return nil
}

This length-prefixed string format is very common for binary encodings. The only thing we need to know for sure is what type the length is encoded as so we know what kind of temporary variable to use.

Repeated Fields

Repeated fields can work in a similar fashion by size-prefixing the number of times the field occurs in the data.

For these examples, lets assume we have a new message that is encoded as binary.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Identifier  |    NumPorts   |      Port0      |    Port1    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Port2    |      Port3    |      PortN ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

Now we can decode this data with the following strategy.

type Message struct {
  Identifier uint16
  Ports      []uint16
}

func (m *Message) UnmarshalBinary(data []byte) error {
  buf := bytes.NewBuffer(data)

  if err := binary.Read(buf, binary.BigEndian, &m.Identifier); err != nil {
    return err
  }

  // Read the HostnameLen into a temporary variable
  var numPorts uint16
  if err := binary.Read(buf, binary.BigEndian, &numPorts); err != nil {
    return err
  }

  // Make a slice with sizing it with the value of the temporary variable
  // from above and read the next n fields into it.
  m.Ports = make([]uint16, numPorts)
  for n := range numPorts {
    if err := binary.Read(buf, binary.BigEndian, &m.Ports[n]); err != nil {
      return err
    }
  }

  return nil
}

Since we're still using binary.Read and managing any returned error, we can safely handle situations where the data we're decoding indicates there should be 5 ports, but only provides 2. We'll catch the io.EOF error and return accordingly instead of needing to handle subslicing and potential panics being thrown.

Conclusion

We've seen that using bytes.Buffer and binary.Read leads to safer reading of []byte and clearer code for other engineers on your team who might read it later.

So, the next time you need to decode binary data and have a specification to follow, try your hand at these techniques to make your code safe, correct, readable, and composable.

The complete source code is avaialable along with supporting tests.