Very apt, but what n-gram length is being used? n=1 is my guess, since "as often as" is a common English construct. Obvious feature request: tweakable lengths.
edit: I'd make the fix myself and send a pull request but I don't know haskell and am too lazy to figure it out.
n=1 is indeed being used. The problem with a larger n is that you get original tweets really often, because the dataset is limited (as much tweets as you can get in one request).
Along those lines, I've been playing recently with using the google ngram data for markov chaining. The size of the corpus allows using 5grams without the problem of seeing text that has actually been written before (mostly.. it could just decide to spit out the complete works of shakespeare any second!), and the results were more interesting to me than most markov chains I've seen before. http://kitenet.net/~joey/blog/entry/dadagoogoo/
Random example 1: nothing had pleased God to bestow upon you as to participation in physical activity and exercise . Don ' t rain .
Random example 2: sad and terrifying each time . After a quick nap .
I did not read the paper, but what does more accurate mean in this case ? Likelihood of some unseen data ? Seems pretty hard to define or measure to me, if the goal is shear amusement.
Comments
@jennyholzer:
OFTEN AS OFTEN AS OFTEN AS OFTEN AS POSSIBLE
Very apt, but what n-gram length is being used? n=1 is my guess, since "as often as" is a common English construct. Obvious feature request: tweakable lengths.
edit: I'd make the fix myself and send a pull request but I don't know haskell and am too lazy to figure it out.
n=1 is indeed being used. The problem with a larger n is that you get original tweets really often, because the dataset is limited (as much tweets as you can get in one request).
Along those lines, I've been playing recently with using the google ngram data for markov chaining. The size of the corpus allows using 5grams without the problem of seeing text that has actually been written before (mostly.. it could just decide to spit out the complete works of shakespeare any second!), and the results were more interesting to me than most markov chains I've seen before. http://kitenet.net/~joey/blog/entry/dadagoogoo/
Random example 1: nothing had pleased God to bestow upon you as to participation in physical activity and exercise . Don ' t rain .
Random example 2: sad and terrifying each time . After a quick nap .
are you using smoothing for large n? kneser-ney smoothing seems to give the best results.
http://nlp.stanford.edu/~wcmac/papers/20050421-smoothing-tut...
I did not read the paper, but what does more accurate mean in this case ? Likelihood of some unseen data ? Seems pretty hard to define or measure to me, if the goal is shear amusement.
Interesting link, thanks!