brof on What if SELECT, FROM, WHERE were functions?

Comment on What if SELECT, FROM, WHERE were functions?parent

Got it! One more question if I may: in https://arxiv.org/pdf/2607.26356, all the JOB queries are around a second or less, if I read correctly.

My understanding is the query:

movie

    .with(keyword.eq("my-kw"))

    .select(title)

is inlined to something by the compiler approximately like

for movie in 0..movie_count {

    let lo = keyword_offsets[movie];

    let hi = keyword_offsets[movie + 1];

    for pos in lo..hi {

        let kw = keyword_ids[pos];

        if keyword_text[kw] == "my-kw" {

            emit(movie, movie_title[movie]);

            break;

        }

    }

}

So there's no index lookup on keyword, right? E.g., to use a hash index on keyword to find the resulting movie rows. If I wanted to do so, would I re-normalize the data in some way? This is what surprised me: that even without the secondary indexes, it is still the same (or more) performant than DuckDB.

Perhaps the index-lookup version would use something like

let movies_by_keyword: HashIdx<_, _> =

      keyword.text().inv().collect();

but it does not seem like it does (even though I assume DuckDB may).

remywangOP18d

Yes basically. But DuckDB actually does not build an index on the keyword text. It only builds primary key indices automatically on data load. The standard benchmark schema specifies fk indices but not one for the keyword text.

Comments