If you can express the DSL SPARQL query as a transformation to JSON Schema, you can have GPT give you the JSON predictably in strict mode, and then reverse the transformation.
Frankly the problem was not in producing a correctly formed SPARQL query; the issue was in the myriad of entity types and relations that the Wikidata project uses and that you need to know about in their detail to use the system. Which often just produces nothing or times out (from the web UI). Not having a previous experience with Wikidata (or SPARQL for that matter) it felt trickier than expected.
Interesting and good to know - the way I've handled similar kinds of things before (for SQL), is if you have a list of predicates or entity types you need to pick from you can populate an enum in a JSON schema. If the list is way too long (it probably is), you can get K nearest neighbor of the entity types to the natural language query and populate the enum with K values (where K might be something reasonable like 10 to 20). You'd need to have a dataset of all entity types stored locally to do this of course.
I know this sounds hand-wavy, but I have had good results doing similar things when trying to pick the correct foreign key out of a table with 1000s of rows.
Think of it like this: don't ask it to construct the query, ask it for the parts of the query, in a predefined format. Then populate one of many pre-defined templates with the values.
For example, you might have ten types of queries. Each query will have parameters and a classification/description. Get the classification, and get the parameters, and stick them into your respective template.
This gives you way more safety and predictability.
I've gone through the whole gamut to do this since first using the GPT's API in early 2023.
As first I was asking for HTML with few-shot since getting reliable JSON was impossible. Then I switched to function calling with JSON mode when that was released later in 2023, but still with fallbacks because sometimes it would spit out markdown with JSON inside, and other fun mistakes. Finally summer 2024 strict mode became available which "guarantees" a well-formed JSON response corresponding to a given schema. I've run hundreds of thousands of queries through that and have not had a formatting issue (yet).
Comments
If you can express the DSL SPARQL query as a transformation to JSON Schema, you can have GPT give you the JSON predictably in strict mode, and then reverse the transformation.
Frankly the problem was not in producing a correctly formed SPARQL query; the issue was in the myriad of entity types and relations that the Wikidata project uses and that you need to know about in their detail to use the system. Which often just produces nothing or times out (from the web UI). Not having a previous experience with Wikidata (or SPARQL for that matter) it felt trickier than expected.
Interesting and good to know - the way I've handled similar kinds of things before (for SQL), is if you have a list of predicates or entity types you need to pick from you can populate an enum in a JSON schema. If the list is way too long (it probably is), you can get K nearest neighbor of the entity types to the natural language query and populate the enum with K values (where K might be something reasonable like 10 to 20). You'd need to have a dataset of all entity types stored locally to do this of course.
I know this sounds hand-wavy, but I have had good results doing similar things when trying to pick the correct foreign key out of a table with 1000s of rows.
Have you done this and had it working reliably for you in practice?
Not for SPARQL, no. But I have done it for other query DSLs like SQL and Opensearch.
Fair enough. Any particular tricks that helped you get valid queries out of it the first time?
Think of it like this: don't ask it to construct the query, ask it for the parts of the query, in a predefined format. Then populate one of many pre-defined templates with the values.
For example, you might have ten types of queries. Each query will have parameters and a classification/description. Get the classification, and get the parameters, and stick them into your respective template.
This gives you way more safety and predictability.
I've gone through the whole gamut to do this since first using the GPT's API in early 2023.
As first I was asking for HTML with few-shot since getting reliable JSON was impossible. Then I switched to function calling with JSON mode when that was released later in 2023, but still with fallbacks because sometimes it would spit out markdown with JSON inside, and other fun mistakes. Finally summer 2024 strict mode became available which "guarantees" a well-formed JSON response corresponding to a given schema. I've run hundreds of thousands of queries through that and have not had a formatting issue (yet).
Very smart approach, thanks