POS is an unfortunate acronym. I genuinely thought you were being derogatory and therefore sarcastic at first! Thus, the challenge of natural-language processing; I wonder if the tagger is actually more intelligent than I am ...
I'm scared. Does this mean there's some value to parsing this kind of gibberish? If so, does the fact that I consider it gibberish mean I'm going to miss the bus on future developments because I've decided I'm too old to take people who write like this seriously?
CMU does some of the most interesting work around with data from Twitter. Some of the same people responsible for this worked on predicting NFL games with Twitter data[0]
We provide a tokenizer, a part-of-speech tagger,
hierarchical word clusters, and a dependency parser for
tweets, along with annotated corpora and web-based
annotation tools.
Part of it is genuine differences between online conversational language versus standard written English, like emoticons, Twitter-specific discourse markers, and hard-to-segment compounds or clitic constructions (see the Gimpel and Owoputi papers (2011, 2013) linked on the page, and/or the annotation guidelines document too). Part of it is just that it's easier for humans to annotate the coarse-grained POS tagset, and we didn't have many resources for annotation when we did it.
These things also intersect ... for example, you'd have to figure out how dialectical English verbal auxiliaries like "finna", or the second or so word in "imma", map to PTB tags. It's possible but just takes more work and thinking through the descriptive linguistics and what you want to use it for. Someday I'd like to update the whole thing for a more PTB-like POS tagset, if it can be done well. I feel like Chris Manning's whitepaper on issues in PTB POS data convinced us (well, it convinced me, at least) that it might be a good idea to focus on making high quality tag annotations. (http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf )
The default tagset that this employs is much more coarse-grained than the Penn tagset. For example, it has only two verb tags, whereas Penn has separate tags for infinitives, gerunds, past forms, etc. On the other hand, it supports the peculiar kinds of lexemes that abound in tweets, having separate tags for hashtags, @-mentions, URLs, emoticons, etc.
Twitterese English is very different from standard English, and by taking the dedicated-tagset approach that embraces this fact, the authors were able to accurately attribute structure even to text that appears gibberish at first sight. To me, this is very impressive.
Yeah, that's the main PTB-style tagset resource for tweets (that I know of). Our tool can also be run to produce tags in their form (we just retrained our software on their annotated data). See further down the webpage for information on how to get this.
Very cool. I did some authorship attribution using a twitter corpus from MSR during grad school. The size constraint for a tweet causes users to have a different writing style then on other document types, which makes for interesting NLP problems.
You may want to look at Factorie (https://github.com/factorie/factorie), that has a decent POS tagger and it's not crippled by the license. It also has dependency parsing which works reasonably well.
I've been looking at Factorie for a while but haven't actually done anything heavy with it.
I planned to replace the optimizer in CMU's POS tagger with the one implemented in OpenNLP to make the tagger fully Apache. Unfortunately, so busy right now. Currently, I'm running the tagger on AWS, so the GPL doesn't hurt me much.
BTW, besides the POS tagger, CMU's TweeboParser depends on Turbo Parser which again is licensed under GPL.
GPL is a killer for any project that's not a toy or for school.
My paycheck - financed by some of the largest companies in my country - disagrees. There are other ways to make money besides locking others from your code.
It depends how the project ownership and copyright is structures. We have used GPL projects successfully when they are dual licensed. It never hurts to ask anyway.
