Moreover, I'd still have to consider what happens at the limits of the
  integer type even if I did use `usize` for more general cases to make
  sure that I properly handle overflow (generally by using saturating or
  checked arithmetic in my use cases).

That's one of those things I don't particularly like about rust. The most terse means of converting between numeric types doesn't do any bounds checking.

  use std::convert::TryFrom;
  
  fn main() {
    let a = usize::MAX;
    let b = a as u32;
    println!("{a} == {b}");
    let c = u32::try_from(a).expect("this will always fail");
    println!("{a} == {c}");
  }