Christopher Prohm

In the last post, I looked into accessing characters at random byte-offsets in Rust strings. That code used the structure of UTF-8 to find the start of a character and then assemble a Rust char from the underlying bytes. Shortly after writing it, I found myself parsing strings in const code to validate identifiers at compile time. In this post, I would like to show how the code of the last post can be adapted for const functions.

My application was the validation of identifiers at compile time. For this use case, the natural shape would have been something like:

pub const fn validate(s: &str) -> bool {
    s.chars().all(is_valid_char)
}

With rustc 1.97.1, this code does not compile. The main problem is that iteration through the Iterator trait is not available in const code. The standard library now has several UTF-8-related const functions, such as str::is_char_boundary, str::floor_char_boundary, str::ceil_char_boundary, and str::split_at. However, character-level iteration through str::chars is still not supported.

Luckily, the code from the last post can be adapted for this use case. The get_number_of_bytes helper can be copied as is:

pub const fn get_number_of_bytes(c: u8) -> Option<usize> {
    match c.leading_ones() {
        0 => Some(1),
        2 => Some(2),
        3 => Some(3),
        4 => Some(4),
        _ => None,
    }
}

The function that assembles a char needs one small adjustment. In normal code, I would usually reach for u32::from. Since that conversion is not available as a const trait call here, I converted it to explicit casts with as.

pub const fn assemble_character(bytes: &[u8]) -> Option<char> {
    let codepoint = match bytes {
        [b0] => *b0 as u32,
        [b0, b1] => (((*b0 & 0b_0001_1111) as u32) << 6) + ((*b1 & 0b_0011_1111) as u32),
        [b0, b1, b2] => {
            (((*b0 & 0b0000_1111) as u32) << 12)
                + (((*b1 & 0b_0011_1111) as u32) << 6)
                + ((*b2 & 0b_0011_1111) as u32)
        }
        [b0, b1, b2, b3] => {
            (((*b0 & 0b0000_0111) as u32) << 18)
                + (((*b1 & 0b_0011_1111) as u32) << 12)
                + (((*b2 & 0b_0011_1111) as u32) << 6)
                + ((*b3 & 0b_0011_1111) as u32)
        }
        _ => return None,
    };

    char::from_u32(codepoint)
}

The next step is to combine these helpers. The next_char function returns the first character together with the rest of the string. One thing to note is the use of str::split_at to replace &s[..number_of_bytes]. Range indexing goes through the Index trait, which is not available as a const trait call here.

pub const fn next_char(s: &str) -> Option<(char, &str)> {
    if s.is_empty() {
        return None;
    }

    let Some(number_of_bytes) = get_number_of_bytes(s.as_bytes()[0]) else {
        panic!("invalid utf8");
    };
    let (char, rest) = s.split_at(number_of_bytes);

    let Some(char) = assemble_character(char.as_bytes()) else {
        panic!("invalid utf8");
    };

    Some((char, rest))
}

A simple application is a function that collects all characters of a string into a fixed-size array. Since the example is still const code, it uses a while let loop instead of a for loop. The array size is passed as a const generic argument.

/// Collect the chars encoded in the first N bytes.
const fn collect_chars<const N: usize>(s: &str) -> [char; N] {
    let mut chars = ['\0'; N];

    let mut current = s;
    while let Some((char, next)) = next_char(current) {
        let offset = s.len() - current.len();
        if offset < N {
            chars[offset] = char;
        }
        current = next;
    }

    chars
}

This function stores each char at the byte offset where it started. For multi-byte characters, the following slots remain '\0'.

#[test]
fn test_collect_chars() {
    const EXAMPLE: &'static str = "héllo €gain";

    assert_eq!(
        const { collect_chars::<{ EXAMPLE.len() }>(EXAMPLE) },
        ['h', 'é', '\0', 'l', 'l', 'o', ' ', '€', '\0', '\0', 'g', 'a', 'i', 'n'],
    );
}

#[test]
fn test_assemble_character() {
    assert_eq!(assemble_character("a".as_bytes()).unwrap(), 'a');
    assert_eq!(assemble_character("é".as_bytes()).unwrap(), 'é');
    assert_eq!(assemble_character("€".as_bytes()).unwrap(), '€');
    assert_eq!(assemble_character("𝄞".as_bytes()).unwrap(), '𝄞');
}

For the actual identifier parser, I used this pattern to walk through the string at compile time. I validate each character against the expected schema.

I expect this kind of workaround to have a limited shelf life, though. The steadily expanding const support in Rust is one of the highlights of new releases for me. Some of these restrictions may soon disappear. For now, manually stepping through UTF-8 bytes is a practical way to keep simple string checks in const code.