In the worst situation creating four iterators but using only the first & last yields

It's about expectations.
To cast a u32 to a char, you have to use char::from_u32(). This returns an option, making it explicit that the result may not be a valid unicode value, but when it does (result is Some(char), then you know it is what you expected it to be based on the value held by the u32. The types match. This simply isn't the case with an i32, where you have a 50% chance to get it catastrophically wrong. So, as a programmer, you aren't allowed to simply cast signed to char that way. But it's your choice to "make a detour via u32 or smaller", because then it should be clear to you that casting a signed value to an unsigned value will yield "weird" results.
Note that when casting to u8 first, there is no possibility that you'll get an invalid unicode value because the entirety of 0-255 is valid, hence you're allowed to directly cast u8 as char

char may not be what you think it is; it does not make sense to allow casting from a signed integer to a unicode scalar value. However, a signed integer can be cast to an unsigned of greater or equal size by simply interpreting the bits accordingly. Any unsigned up to 32bits can then be cast technically into a char, however it might not map into the valid unicode range and panic, but at least as far as the type system is concerned it's possible to do.