In Rust you can hold references directly into the buffer backing the array

Yes! But I am arguing that this prevents having multiple mutable references

My Java knowledge is quite rusty > Also because Java is a GC'ed VM ...

Your Java knowledge is fine :-) But I'm arguing that you don't strictly need a GC'ed, or RC'ed language: if done "correctly", multiple mutable references are possible. Just not with fat pointers! The programming language I'm building allows this even today. You can try it in the playground [1]:

    fun main()
        list := List+(int[4]) # <<= owned list
        borrow : &list        # <<= mutable borrow
        for i := until(4)
            borrow.add(i)
            list.add(10 * i)
        for i := until(8)
            println(borrow.array[i])
    
    type List
        array int[]
        size int
    
    fun List+ add(x int)
        if size >= array.len
            n : int[array.len * 2]
            for i := until(array.len)
                n[i] = array[i]
            array = n
        array[size] = x
        size += 1

So "List+" is owned type (just "List" without "+" would be reference counted). You may want to look at the generated C code at the end of the page.

[1]: https://thomasmueller.github.io/bau-lang/