the first step into creating an open source knowledge engine
I'm working on this too, but I'm tackling it from the non-math side (ie, NLP+Knowledge Graph).
one of the most challenging technical problems we faced was free-form input
Yes, it's a horrible problem(!) I'm using Quepy[1] (which in turn uses NLTK), and it does a decent job. It's still not automatically general purpose (you need to write code to map classes of queries), but it can handle questions like "Who directed The Social Network"[2].
The other thing is that W|A has trillions of data points and what I am only allowed to describe as the beginnings of a semantic network for inferring relations between them. It was a vastly overcomplicated system that was difficult to work with, so I am quite confident that some day there will be an open-source alternative that anyone can contribute to.
This is interesting to me (for obvious reasons).
Can you expand on what made it so complicated (I assume beyond the standard RDF-style inference engines)?
One of the main reasons things why the system was so complex was because it had literally been built from scratch using a weakly typed language that has no support for data structures or object-oriented programming, and had absolutely horrendous error handling (any Mathematica user can attest to this). I do not believe Mathematica was the appropriate tool for a project as large as Wolfram Alpha, and obviously the performance hit from an interpreted language like Mathematica is very significant when writing computation at the scale of a search engine (or "knowledge engine" as the engineers around me were quick to correct).
One great decision the Wolfram Alpha people made was to put together an excellent set of internal documentation on how to add new parsing capabilities to the language. So suppose you were tasked with adding queries about something like pregnancy data, you would just write a fairly straightforward module that would capture queries like "I am 6 months pregnant" and return a list of pods (a pod is the computed interpretation of your query, most Wolfram Alpha queries will return at least 5 of them). For pregnancy data, there is a pod that shows you how big the fetus should be, another for how much amniotic fluid there is, and so on. You would then write some Mathematica code to either scrape a website with pregnancy data or integrate with some data set that was curated by a data curator. This is not difficult to do, and I know of several WA-like projects that have accomplished this already. The problem with this is that data gets siloed, and data curators have a weak standard for how data and its relations should be expressed.
This leads to difficulty arising when you are tasked with handling a complex query like "Which country has the greatest ratio of population to GDP?" Now you're talking about interoperability between two data sets, and although it can be done quite easily using Mathematica's CountryData function:
... it is nearly impossible to handle these kinds of situations for general queries that could ask about ratios of anything. A possible solution was for some time to make "ratio of population to GDP" a column in the database table, but ostensibly this leads to an exponential explosion of columns if you are trying to answer general queries.
By the time I had joined the Alpha team (after working for 2 years on Mathematica) they were already moving some of their most poorly designed data sets into a much better system that used a more rigid set of standards for describing things, places, concepts, relations, etc. I wish I could elaborate more on this because it was really very cool technology running in the background, but Wolfram Research has a real track record of suing people who violate their NDA (Matthew Cook). What I can say is that it fixed some absolutely ridiculous database design decisions - for example, in one table storing athlete performance, there were multiple rows for athletes who played multiple years where the name would be BabeRuth1942, BabeRuth1943, BabeRuth1944, and so on. It was then up to the developer to know that the name and year need to be separated, and that their code needs to handle athletes who play one year and multiple years separately.
tl;dr: Don't over-glorify Wolfram Alpha - it gets things done, but the poor performance and unpredictable results are caused by bad planning and poor organization within. If I was going to make my own knowledge engine, I would spend a long time drawing up an incredibly detailed schema about how every single thing would be represented and how a developer would write a new module for it before writing a single line of code. These are the lessons gleaned from spending two and a half years wallowing in a Big Ball of Mud (http://laputan.org/mud/).
So it used a traditional database for storage? I've gone down the triplestore route, with some trepidation. Working out okish so far, although I wish there were better resources around on SPARQL.
I've also been working with Quepy a bit lately. Very cool stuff. Are you able to comment at all on what you're working on, or is it "super secret stealth mode" stuff?
For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not.
If you're not in super-secret stealth mode and ever want to compare notes to talk about this stuff offline, feel free to shoot me an email.
Not especially super-stealth, but it's not really ready for public consumption.
Basically I've been glueing lots of pieces pre-existing software together, sticking webservice front ends on them and making them work together. It's all Dockerfied so things can be run separately.
For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not.
Very similar here.
I'm (currently) using DBPedia dumps, loaded into Jena. I'm experimenting with content extraction (for eg CIA Factbook).
Nice, sounds like we're using a very similar stack. We are using Jena as our triplestore, but we don't touch the dbpedia triples directly, but rely on Stanbol to do the entity extraction processing for us. And we're also starting down the path of using Quepy.
Jena here too. 14G of data total. Total Triples: 124,294,115 (SELECT (COUNT(*) AS ?no) { ?s ?p ?o })
I believe that's actually pretty big for a triplestore. Seems to work ok, but loading is pretty slow.
I'm contemplating switching to YAGO2[1] or Freebase, but I think I'd be better served doing entity extraction myself (DBPedia & YAGO tend to be out of date).
Comments
the first step into creating an open source knowledge engine
I'm working on this too, but I'm tackling it from the non-math side (ie, NLP+Knowledge Graph).
one of the most challenging technical problems we faced was free-form input
Yes, it's a horrible problem(!) I'm using Quepy[1] (which in turn uses NLTK), and it does a decent job. It's still not automatically general purpose (you need to write code to map classes of queries), but it can handle questions like "Who directed The Social Network"[2].
The other thing is that W|A has trillions of data points and what I am only allowed to describe as the beginnings of a semantic network for inferring relations between them. It was a vastly overcomplicated system that was difficult to work with, so I am quite confident that some day there will be an open-source alternative that anyone can contribute to.
This is interesting to me (for obvious reasons).
Can you expand on what made it so complicated (I assume beyond the standard RDF-style inference engines)?
[1] http://quepy.machinalis.com/
[2] http://quepy.machinalis.com/#Who%20directed%20The%20Social%2...?
One of the main reasons things why the system was so complex was because it had literally been built from scratch using a weakly typed language that has no support for data structures or object-oriented programming, and had absolutely horrendous error handling (any Mathematica user can attest to this). I do not believe Mathematica was the appropriate tool for a project as large as Wolfram Alpha, and obviously the performance hit from an interpreted language like Mathematica is very significant when writing computation at the scale of a search engine (or "knowledge engine" as the engineers around me were quick to correct).
One great decision the Wolfram Alpha people made was to put together an excellent set of internal documentation on how to add new parsing capabilities to the language. So suppose you were tasked with adding queries about something like pregnancy data, you would just write a fairly straightforward module that would capture queries like "I am 6 months pregnant" and return a list of pods (a pod is the computed interpretation of your query, most Wolfram Alpha queries will return at least 5 of them). For pregnancy data, there is a pod that shows you how big the fetus should be, another for how much amniotic fluid there is, and so on. You would then write some Mathematica code to either scrape a website with pregnancy data or integrate with some data set that was curated by a data curator. This is not difficult to do, and I know of several WA-like projects that have accomplished this already. The problem with this is that data gets siloed, and data curators have a weak standard for how data and its relations should be expressed.
This leads to difficulty arising when you are tasked with handling a complex query like "Which country has the greatest ratio of population to GDP?" Now you're talking about interoperability between two data sets, and although it can be done quite easily using Mathematica's CountryData function:
... it is nearly impossible to handle these kinds of situations for general queries that could ask about ratios of anything. A possible solution was for some time to make "ratio of population to GDP" a column in the database table, but ostensibly this leads to an exponential explosion of columns if you are trying to answer general queries.By the time I had joined the Alpha team (after working for 2 years on Mathematica) they were already moving some of their most poorly designed data sets into a much better system that used a more rigid set of standards for describing things, places, concepts, relations, etc. I wish I could elaborate more on this because it was really very cool technology running in the background, but Wolfram Research has a real track record of suing people who violate their NDA (Matthew Cook). What I can say is that it fixed some absolutely ridiculous database design decisions - for example, in one table storing athlete performance, there were multiple rows for athletes who played multiple years where the name would be BabeRuth1942, BabeRuth1943, BabeRuth1944, and so on. It was then up to the developer to know that the name and year need to be separated, and that their code needs to handle athletes who play one year and multiple years separately.
tl;dr: Don't over-glorify Wolfram Alpha - it gets things done, but the poor performance and unpredictable results are caused by bad planning and poor organization within. If I was going to make my own knowledge engine, I would spend a long time drawing up an incredibly detailed schema about how every single thing would be represented and how a developer would write a new module for it before writing a single line of code. These are the lessons gleaned from spending two and a half years wallowing in a Big Ball of Mud (http://laputan.org/mud/).
Thanks for the reply.
So it used a traditional database for storage? I've gone down the triplestore route, with some trepidation. Working out okish so far, although I wish there were better resources around on SPARQL.
Thank you for going into detail describing these things. It is all very fascinating :)
I'm working on this too,
Have you looked at any of the research to come out of http://start.csail.mit.edu/index.php?
I've also been working with Quepy a bit lately. Very cool stuff. Are you able to comment at all on what you're working on, or is it "super secret stealth mode" stuff?
For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not.
If you're not in super-secret stealth mode and ever want to compare notes to talk about this stuff offline, feel free to shoot me an email.
Not especially super-stealth, but it's not really ready for public consumption.
Basically I've been glueing lots of pieces pre-existing software together, sticking webservice front ends on them and making them work together. It's all Dockerfied so things can be run separately.
For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not.
Very similar here.
I'm (currently) using DBPedia dumps, loaded into Jena. I'm experimenting with content extraction (for eg CIA Factbook).
I have the Quepy->SPARQL mapping working, though.
I'll send you an email.
Nice, sounds like we're using a very similar stack. We are using Jena as our triplestore, but we don't touch the dbpedia triples directly, but rely on Stanbol to do the entity extraction processing for us. And we're also starting down the path of using Quepy.
Great minds think alike? :-)
Jena here too. 14G of data total. Total Triples: 124,294,115 (SELECT (COUNT(*) AS ?no) { ?s ?p ?o })
I believe that's actually pretty big for a triplestore. Seems to work ok, but loading is pretty slow.
I'm contemplating switching to YAGO2[1] or Freebase, but I think I'd be better served doing entity extraction myself (DBPedia & YAGO tend to be out of date).
[1] http://www.mpi-inf.mpg.de/yago-naga/yago/