Data model evolution

One of desert’s main goals is allowing stored or transmitted data to survive controlled changes to Rust data types.

Evolution support is generated by #[derive(BinaryCodec)] for structs and enums. The derive macro writes a compact version header and enough metadata for older and newer versions to skip, default, or reinterpret fields where that is safe.

Compatibility rules in short

Compatible changes:

• Convert a multi-field tuple into a struct with the same field order.
• Wrap a value in a #[desert(transparent)] single-field struct.
• Replace one collection type with another collection type using the same item representation.
• Add a struct field with FieldAdded.
• Make a field optional with FieldMadeOptional.
• Remove a field with FieldRemoved, with the limits described below.
• Make a field transient with FieldMadeTransient plus #[transient(default)].
• Add an enum variant at the end of the enum.
• Insert or remove #[transient] enum variants.

Breaking or risky changes:

• Rename a field without preserving the serialized field name. The current Rust derive macro does not have a field rename attribute.
• Reorder enum variants without #[desert(sorted_constructors)].
• Remove an enum variant that has already been serialized.
• Change a field’s type unless the old and new types intentionally share a binary representation.
• Use DeduplicatedString in evolvable fields unless all readers and writers agree on the exact stream shape.

Tuples and structs

Version-0 structs are compatible with tuples of the same arity and field order:

use desert_rust::{deserialize, serialize_to_byte_vec, BinaryCodec, Result};

#[derive(Debug, PartialEq, BinaryCodec)]
struct Point {
    x: i32,
    y: i32,
}

fn main() -> Result<()> {
    let bytes = serialize_to_byte_vec(&(10, 20))?;
    let point: Point = deserialize(&bytes)?;

    assert_eq!(point, Point { x: 10, y: 20 });
    Ok(())
}

This is useful for moving from positional values to named records.

Transparent wrappers

Transparent single-field structs are encoded exactly like the inner field:

use desert_rust::{deserialize, serialize_to_byte_vec, BinaryCodec, Result};

#[derive(Debug, PartialEq, BinaryCodec)]
#[desert(transparent)]
struct UserId(u64);

fn main() -> Result<()> {
    let bytes = serialize_to_byte_vec(&42u64)?;
    let id: UserId = deserialize(&bytes)?;

    assert_eq!(id, UserId(42));
    Ok(())
}

This lets a model evolve from primitive values to domain newtypes without changing stored bytes.

Collections

Generic collection codecs use a shared iterable format, so many collection changes are compatible:

use desert_rust::{deserialize, serialize_to_byte_vec, Result};
use std::collections::LinkedList;

fn main() -> Result<()> {
    let bytes = serialize_to_byte_vec(&vec![1i32, 2, 3])?;
    let values: LinkedList<i32> = deserialize(&bytes)?;

    assert_eq!(values.into_iter().collect::<Vec<_>>(), vec![1, 2, 3]);
    Ok(())
}

Set and map compatibility depends on the target type’s Eq, Hash, or Ord requirements and on whether duplicate values make sense.

Adding a field

When a struct receives a new field, record it with FieldAdded and provide the default expression used when reading old data:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
struct ProductV1 {
    name: String,
    price: i32,
}

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
#[desert(evolution(FieldAdded("in_stock", true)))]
struct ProductV2 {
    name: String,
    in_stock: bool,
    price: i32,
}

With this change:

• ProductV2 can read ProductV1 data and uses true for in_stock.
• ProductV1 can read ProductV2 data by skipping the added field.

The added field does not have to be the last field in the Rust struct. The evolution metadata records which generation introduced it.

Making a field optional

A field can be changed from T to Option<T>:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
#[desert(evolution(
    FieldAdded("in_stock", true),
    FieldMadeOptional("price")
))]
struct ProductV3 {
    name: String,
    in_stock: bool,
    price: Option<i32>,
}

With this change:

• New code reads old data as Some(old_value).
• Old code can read new data if the option is Some(value).
• Old code cannot read new data if the option is None, because it expected a non-optional field value.

This can be used as an intermediate migration before removing a field.

Removing a field

A removed field must be recorded:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
#[desert(evolution(
    FieldAdded("in_stock", true),
    FieldMadeOptional("price"),
    FieldRemoved("price")
))]
struct ProductV4 {
    name: String,
    in_stock: bool,
}

With this change:

• New code can read old data by skipping price.
• Old code can read new data only if the removed field had already become optional, in which case it is read as None.
• Old code that expects a non-optional removed field cannot read new data.

Transient fields

Adding a new transient field does not change the binary representation:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
struct WithCache {
    value: String,
    #[transient(None::<usize>)]
    cached_len: Option<usize>,
}

If an existing serialized field becomes transient, record it with FieldMadeTransient and keep a transient default expression:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
#[desert(evolution(
    FieldAdded("in_stock", true),
    FieldMadeOptional("price"),
    FieldRemoved("price"),
    FieldMadeTransient("name")
))]
struct ProductV5 {
    #[transient("unknown".to_string())]
    name: String,
    in_stock: bool,
}

FieldMadeTransient behaves like FieldRemoved for the wire format. New code can read old data and uses the transient default. Older code cannot read the new data if it requires the missing field.

Evolving enums

Enums are encoded by constructor id. Constructor ids follow source order unless #[desert(sorted_constructors)] is used.

Adding a new variant at the end is compatible for old values:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
enum EventV1 {
    Started,
    Message(String),
}

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
enum EventV2 {
    Started,
    Message(String),
    Stopped,
}

With this change:

• EventV2 can read old EventV1 values.
• EventV1 can read EventV2 data only when the stored constructor id also existed in EventV1.
• EventV1 cannot read EventV2::Stopped.

Enum variants with fields can have their own evolution steps:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
enum Event {
    #[desert(evolution(FieldAdded("source", "api".to_string())))]
    Message { text: String, source: String },
}

Transient variants are not assigned constructor ids:

use desert_rust::BinaryCodec;

#[derive(Debug, Clone, PartialEq, BinaryCodec)]
enum RuntimeState {
    Stored,
    #[transient]
    InMemoryOnly,
}

Serializing RuntimeState::InMemoryOnly fails. The benefit is that such variants can be inserted or removed without shifting persistent constructor ids.

Evolution encoding

For derived structs and non-transparent enum variant payloads, desert writes:

1. a version byte
2. for non-zero versions, compact evolution metadata describing chunks and removed or optional fields
3. the field data split into generation chunks

Version 0 data has no extra evolution metadata beyond the version byte. This keeps initial records compact and is why version-0 structs and tuples can share the same format.

PointV1

#[derive(Debug, Clone, PartialEq, desert_rust::BinaryCodec)]
#[desert(evolution())]
struct PointV1 {
    x: i32,
    y: i32,
}

let bytes = desert_rust::serialize_to_byte_vec(&PointV1 { x: 10, y: 20 })?;

[0x00, 0x00, 0x00, 0x00, 0x0A, 0x00, 0x00, 0x00, 0x14]

When a field is added, the new generation is written as a later chunk. Older readers can skip chunks they do not know about. Newer readers can detect missing chunks and use defaults from FieldAdded.

PointV2 adds label

#[derive(Debug, Clone, PartialEq, desert_rust::BinaryCodec)]
#[desert(evolution(FieldAdded("label", "origin".to_string())))]
struct PointV2 {
    x: i32,
    label: String,
    y: i32,
}

let value = PointV2 { x: 10, label: "origin".to_string(), y: 20 };
let bytes = desert_rust::serialize_to_byte_vec(&value)?;

[0x01, 0x10, 0x0E, 0x00, 0x00, 0x00, 0x0A, 0x00, 0x00, 0x00, 0x14, 0x0C, 0x6F, 0x72, 0x69, 0x67, 0x69, 0x6E]

If a later version makes the field optional, the metadata records the field position. Older readers can still read Some(value) as the original field type; None is only understood by readers that know the optional step.

PointV3 makes label optional

#[derive(Debug, Clone, PartialEq, desert_rust::BinaryCodec)]
#[desert(evolution(
    FieldAdded("label", "origin".to_string()),
    FieldMadeOptional("label")
))]
struct PointV3 {
    x: i32,
    label: Option<String>,
    y: i32,
}

let value = PointV3 { x: 10, label: None, y: 20 };
let bytes = desert_rust::serialize_to_byte_vec(&value)?;

[0x02, 0x10, 0x02, 0x01, 0x01, 0x00, 0x00, 0x00, 0x0A, 0x00, 0x00, 0x00, 0x14, 0x00]

When a field is removed, the metadata records the removed field name so readers that still know the field can either treat it as None if it is optional, or fail clearly if it is required.

PointV4 removes label

#[derive(Debug, Clone, PartialEq, desert_rust::BinaryCodec)]
#[desert(evolution(
    FieldAdded("label", "origin".to_string()),
    FieldMadeOptional("label"),
    FieldRemoved("label")
))]
struct PointV4 {
    x: i32,
    y: i32,
}

let bytes = desert_rust::serialize_to_byte_vec(&PointV4 { x: 10, y: 20 })?;

[0x03, 0x10, 0x00, 0x03, 0x0A, 0x6C, 0x61, 0x62, 0x65, 0x6C, 0x03, 0x01, 0x00, 0x00, 0x00, 0x0A, 0x00, 0x00, 0x00, 0x14]

Keeping evolution safe

Use these practices for long-lived formats:

• Append evolution steps. Do not rewrite old evolution history.
• Keep field names stable.
• Prefer Option<T> as a compatibility bridge before removing a field.
• Keep enum constructor order stable, or opt into sorted constructors from the first released version.
• Add roundtrip and cross-version compatibility tests for every model that is persisted or sent across process boundaries.