Without looking under the hood I'd say there could be at least four reasons why they fail (based on what most of the NLP literature is lacking):
- did not remove contradicting information from the training sets (two very similar vectors having contradicting labels)
- did not try enough feature selection algorithms
- did not estimate ALL learner parameters using the training sets with internal CV
- did not include domain knowledge
The last one refers to Paul Houle comment. Just, beside using tools like OpenCyc, WordNet, UMLS, there many other ways to embed domain expertise in an automated classification process. Injecting semantically related features into a vector representation of a document is extremely difficult. Forward feature selection doesn't work well for sparse and noisy data.
the curse of dimensionality is the worst problem that affects machine learning
customers don't want to create training sets large enough to train text classifiers; often the number of documents they need to sort into a category is too small to fit in a category.
As for semantic indexing, it was hard to do in 2005. In 2011 it's easy. DBpedia and Freebase are a chromosome map for the human memome. With large amounts of instance information, it's possible to do things that a big rulebox can't.
These tools are aiming for the market segment that Cyc aimed for, but will use very different methodologies.
Comments
Without looking under the hood I'd say there could be at least four reasons why they fail (based on what most of the NLP literature is lacking):
- did not remove contradicting information from the training sets (two very similar vectors having contradicting labels)
- did not try enough feature selection algorithms
- did not estimate ALL learner parameters using the training sets with internal CV
- did not include domain knowledge
The last one refers to Paul Houle comment. Just, beside using tools like OpenCyc, WordNet, UMLS, there many other ways to embed domain expertise in an automated classification process. Injecting semantically related features into a vector representation of a document is extremely difficult. Forward feature selection doesn't work well for sparse and noisy data.
the curse of dimensionality is the worst problem that affects machine learning
customers don't want to create training sets large enough to train text classifiers; often the number of documents they need to sort into a category is too small to fit in a category.
As for semantic indexing, it was hard to do in 2005. In 2011 it's easy. DBpedia and Freebase are a chromosome map for the human memome. With large amounts of instance information, it's possible to do things that a big rulebox can't.
These tools are aiming for the market segment that Cyc aimed for, but will use very different methodologies.