It's great to see people tackling this problem, but I wouldn't declare victory for sweble just yet ("The Parser That Cracked..."). There are other promising MediaWiki parser efforts out there.
For one, sweble is a Java parser, and I'm not sure this makes it a good drop-in replacement for the current MediaWiki PHP code. The DBPedia Project also has what looks like a decent AST-based Java parser [1]. I would be interested in a comparison between sweble and DBPedia's WikiParser.
I stumbled across a very nice MediaWiki scanner and parser in C a while ago [2]. It uses ragel [3] for the scanner; the parser is not a completely generic AST builder, but is rather specific to the problem of converting MediaWiki markup to some other wiki markup. It does do quite a bit of the parser work already though.
Presumably a PHP extension around a C or C++ scanner/parser could someday replace the current MediaWiki parsing code.
Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP, and would be well worth doing if it had significant benefits.
Of course, a PHP parser would still have to be maintained in parallel as not everyone would be able to do the Java option.
> Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP...
No doubt the incremental complexity for Wikipedia would be small in relative terms. I assume that argument would support a variety of proposals.
A solid scanner and parser in C/C++ would benefit a broader audience though. All the major scripting languages can be extended in C/C++. In fact, the ragel-based parser I mentioned earlier [1] was built to be used from within Ruby code.
Comments
It's great to see people tackling this problem, but I wouldn't declare victory for sweble just yet ("The Parser That Cracked..."). There are other promising MediaWiki parser efforts out there.
For one, sweble is a Java parser, and I'm not sure this makes it a good drop-in replacement for the current MediaWiki PHP code. The DBPedia Project also has what looks like a decent AST-based Java parser [1]. I would be interested in a comparison between sweble and DBPedia's WikiParser.
I stumbled across a very nice MediaWiki scanner and parser in C a while ago [2]. It uses ragel [3] for the scanner; the parser is not a completely generic AST builder, but is rather specific to the problem of converting MediaWiki markup to some other wiki markup. It does do quite a bit of the parser work already though.
Presumably a PHP extension around a C or C++ scanner/parser could someday replace the current MediaWiki parsing code.
[1] http://wiki.dbpedia.org/DeveloperDocumentation/WikiParser?v=...
[2] http://git.wincent.com/wikitext.git
[3] http://www.complang.org/ragel/
Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP, and would be well worth doing if it had significant benefits.
Of course, a PHP parser would still have to be maintained in parallel as not everyone would be able to do the Java option.
> Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP...
No doubt the incremental complexity for Wikipedia would be small in relative terms. I assume that argument would support a variety of proposals.
A solid scanner and parser in C/C++ would benefit a broader audience though. All the major scripting languages can be extended in C/C++. In fact, the ragel-based parser I mentioned earlier [1] was built to be used from within Ruby code.
[1] http://git.wincent.com/wikitext.git