Thanks for posting your thoughts. I'm having a bit of difficulty breaking down "semantic layer" into concrete technological concepts. An attempt:
* Schema definition. PK/FK columns and their relationships. Standard ER fare.
* Dimensions. Partially overlap with PK/FK structure, but may include fields that don't map to an explicit key column, e.g. [trunc] date or zip code or even binned measures. Can see the value of having dimensions documented across a team.
* Measures. Mainly named aggregations, e.g. value = sum(price * quant), which can be aggregated over many dimensional combinations. Definitely useful, though I'd expect PRQL "functions" to be usable in the same role.
Nope, those are the primary components of a semantic layer. Most other semantic layer products have three main components: input data sources -- typically SQL queries or table names, configuration -- the semantic layer describing all the things you mention above, relative to the input data sources, and the access layer -- usually a non-SQL API that consumers must use to consume data that has been modeled by the semantic layer. Check out the docs for Cube [1] for an example of this. Cube also has a SQL API, but it's not fully fleshed out yet.
What makes Malloy shine is that all 3 of these things are integrated in the same language, so users don't have to jump between different tools to model and explore their data. You can query/explore data in Malloy, iterate on functions to express your business logic, and immediately view the results. Doing this in something like Cube would require you to: (1) write SQL queries to prototype the function, (2) update the Cube configuration files with your changes, and (3) hit the REST API with a request to view results. In Malloy, it's all just writing and running Malloy queries.
I would also suggest prql, which is also great. Plus:
- the license of prql is better (APACHE v2 vs GPL v2)
- although having modeling layer is good, but something like dbt-metrics maybe better. In this point, prql is more focus, on querying part.
Comments
There is also PRQL which is more intuitive and simpler IMO - https://github.com/prql/prql
PRQL is very cool, but I think Malloy is meaningfully different (and more useful) because of its inclusion of a semantic layer into the core language.
Thanks for posting your thoughts. I'm having a bit of difficulty breaking down "semantic layer" into concrete technological concepts. An attempt:
* Schema definition. PK/FK columns and their relationships. Standard ER fare.
* Dimensions. Partially overlap with PK/FK structure, but may include fields that don't map to an explicit key column, e.g. [trunc] date or zip code or even binned measures. Can see the value of having dimensions documented across a team.
* Measures. Mainly named aggregations, e.g. value = sum(price * quant), which can be aggregated over many dimensional combinations. Definitely useful, though I'd expect PRQL "functions" to be usable in the same role.
* Formatting rules.
Am I missing something crucial?
Nope, those are the primary components of a semantic layer. Most other semantic layer products have three main components: input data sources -- typically SQL queries or table names, configuration -- the semantic layer describing all the things you mention above, relative to the input data sources, and the access layer -- usually a non-SQL API that consumers must use to consume data that has been modeled by the semantic layer. Check out the docs for Cube [1] for an example of this. Cube also has a SQL API, but it's not fully fleshed out yet.
What makes Malloy shine is that all 3 of these things are integrated in the same language, so users don't have to jump between different tools to model and explore their data. You can query/explore data in Malloy, iterate on functions to express your business logic, and immediately view the results. Doing this in something like Cube would require you to: (1) write SQL queries to prototype the function, (2) update the Cube configuration files with your changes, and (3) hit the REST API with a request to view results. In Malloy, it's all just writing and running Malloy queries.
[1]: https://cube.dev/docs/query-format
I would also suggest prql, which is also great. Plus: - the license of prql is better (APACHE v2 vs GPL v2) - although having modeling layer is good, but something like dbt-metrics maybe better. In this point, prql is more focus, on querying part.