{"id":7279,"date":"2010-08-25T18:27:04","date_gmt":"2010-08-25T18:27:04","guid":{"rendered":"http:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/"},"modified":"2010-08-25T18:27:04","modified_gmt":"2010-08-25T18:27:04","slug":"n-grams","status":"publish","type":"post","link":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/","title":{"rendered":"N-grams"},"content":{"rendered":"\u23f3 Approximate reading time: <\/span> < 1<\/span> min<\/span><\/span>

In the few spare minutes I’ve had this week, I’ve been trying out n-grams<\/a> as a comparable with other text-mining processes. This still fits squarely in the category of “a lot to learn,” but I’m happy to be running Perl and various Lingua::EN modules on this (nothing super-complicated in the switch, but all of my previous Perl tinkering was on a PC). Today’s corpus was perhaps too small to yield many insights: just 2300 words across 10-12 email messages sent via WPA-L on Tuesday, August 24. All the same, insights or no, here the top five bigrams, with T-Units enumerated to the fourteenth decimal place, i.e., something like hundred-trillionth or quadrillionth position. Such precision is useful, I suppose, for avoiding ties.<\/p>\n

Bi-grams (T-Score, count, bigram)\n2.22372310460229\t5\taudience awareness\t\n1.72376348313075\t3\twriting centers\t\n1.72376348313075\t3\twriting spaces\t\n1.41265204574184\t2\tdevelop new\t\n1.41109052911059\t2\tnew ways\t<\/pre>\n
It looks like “develop new ways” is part of a trigram that shows up twice in the corpus.  This script–a fine one, by the way–renders those three words into a 2×2 bigram.  But that’s exactly what it was assigned to do.<\/p>\n","protected":false},"excerpt":{"rendered":"
\u23f3 Approximate reading time: <\/span>  < 1<\/span> min<\/span><\/span>In the few spare minutes I’ve had this week, I’ve been trying out n-grams as a comparable with other text-mining processes. This still fits squarely in the category of “a lot to learn,” but I’m happy to be running Perl and various Lingua::EN modules on this (nothing super-complicated in the switch, but all of my…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[170],"tags":[2437,352,2438,2439,2440,576],"class_list":["post-7279","post","type-post","status-publish","format-standard","hentry","category-method","tag-mining","tag-perl","tag-phrase","tag-processing","tag-unit","tag-wpa"],"yoast_head":"\nN-grams - Earth Wide Moth<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"N-grams - Earth Wide Moth\" \/>\n<meta property=\"og:description\" content=\"\u23f3 Approximate reading time:  < 1 minIn the few spare minutes I’ve had this week, I’ve been trying out n-grams as a comparable with other text-mining processes. This still fits squarely in the category of “a lot to learn,” but I’m happy to be running Perl and various Lingua::EN modules on this (nothing super-complicated in the switch, but all of my...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/\" \/>\n<meta property=\"og:site_name\" content=\"Earth Wide Moth\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/derek.mueller\" \/>\n<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/derek.mueller\" \/>\n<meta property=\"article:published_time\" content=\"2010-08-25T18:27:04+00:00\" \/>\n<meta name=\"author\" content=\"Derek Mueller\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Derek Mueller\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/\"},\"author\":{\"name\":\"Derek Mueller\",\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#\\\/schema\\\/person\\\/66fc5e80a62f53057760790b1cfeb24b\"},\"headline\":\"N-grams\",\"datePublished\":\"2010-08-25T18:27:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/\"},\"wordCount\":168,\"publisher\":{\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#\\\/schema\\\/person\\\/66fc5e80a62f53057760790b1cfeb24b\"},\"keywords\":[\"mining\",\"perl\",\"phrase\",\"processing\",\"unit\",\"WPA\"],\"articleSection\":[\"Methods\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/\",\"url\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/\",\"name\":\"N-grams - Earth Wide Moth\",\"isPartOf\":{\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#website\"},\"datePublished\":\"2010-08-25T18:27:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/2010\\\/08\\\/25\\\/n-grams\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"N-grams\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#website\",\"url\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/\",\"name\":\"Earth Wide Moth\",\"description\":\"\",\"publisher\":{\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#\\\/schema\\\/person\\\/66fc5e80a62f53057760790b1cfeb24b\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"http:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/#\\\/schema\\\/person\\\/66fc5e80a62f53057760790b1cfeb24b\",\"name\":\"Derek Mueller\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i0.wp.com\\\/www.earthwidemoth.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-bmartwork.png?fit=1220%2C480\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/www.earthwidemoth.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-bmartwork.png?fit=1220%2C480\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/www.earthwidemoth.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-bmartwork.png?fit=1220%2C480\",\"width\":1220,\"height\":480,\"caption\":\"Derek Mueller\"},\"logo\":{\"@id\":\"https:\\\/\\\/i0.wp.com\\\/www.earthwidemoth.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-bmartwork.png?fit=1220%2C480\"},\"sameAs\":[\"http:\\\/\\\/www.derekmueller.net\\\/\",\"https:\\\/\\\/www.facebook.com\\\/derek.mueller\",\"https:\\\/\\\/www.instagram.com\\\/dereknmueller\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/dnmueller\"],\"url\":\"https:\\\/\\\/www.earthwidemoth.com\\\/blog\\\/author\\\/dmueller\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"N-grams - Earth Wide Moth","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/","og_locale":"en_US","og_type":"article","og_title":"N-grams - Earth Wide Moth","og_description":"\u23f3 Approximate reading time:  < 1 minIn the few spare minutes I’ve had this week, I’ve been trying out n-grams as a comparable with other text-mining processes. This still fits squarely in the category of “a lot to learn,” but I’m happy to be running Perl and various Lingua::EN modules on this (nothing super-complicated in the switch, but all of my...","og_url":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/","og_site_name":"Earth Wide Moth","article_publisher":"https:\/\/www.facebook.com\/derek.mueller","article_author":"https:\/\/www.facebook.com\/derek.mueller","article_published_time":"2010-08-25T18:27:04+00:00","author":"Derek Mueller","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Derek Mueller","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/#article","isPartOf":{"@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/"},"author":{"name":"Derek Mueller","@id":"http:\/\/www.earthwidemoth.com\/blog\/#\/schema\/person\/66fc5e80a62f53057760790b1cfeb24b"},"headline":"N-grams","datePublished":"2010-08-25T18:27:04+00:00","mainEntityOfPage":{"@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/"},"wordCount":168,"publisher":{"@id":"http:\/\/www.earthwidemoth.com\/blog\/#\/schema\/person\/66fc5e80a62f53057760790b1cfeb24b"},"keywords":["mining","perl","phrase","processing","unit","WPA"],"articleSection":["Methods"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/","url":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/","name":"N-grams - Earth Wide Moth","isPartOf":{"@id":"http:\/\/www.earthwidemoth.com\/blog\/#website"},"datePublished":"2010-08-25T18:27:04+00:00","breadcrumb":{"@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.earthwidemoth.com\/blog\/2010\/08\/25\/n-grams\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/www.earthwidemoth.com\/blog\/"},{"@type":"ListItem","position":2,"name":"N-grams"}]},{"@type":"WebSite","@id":"http:\/\/www.earthwidemoth.com\/blog\/#website","url":"http:\/\/www.earthwidemoth.com\/blog\/","name":"Earth Wide Moth","description":"","publisher":{"@id":"http:\/\/www.earthwidemoth.com\/blog\/#\/schema\/person\/66fc5e80a62f53057760790b1cfeb24b"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/www.earthwidemoth.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"http:\/\/www.earthwidemoth.com\/blog\/#\/schema\/person\/66fc5e80a62f53057760790b1cfeb24b","name":"Derek Mueller","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/wp-content\/uploads\/2023\/04\/cropped-bmartwork.png?fit=1220%2C480","url":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/wp-content\/uploads\/2023\/04\/cropped-bmartwork.png?fit=1220%2C480","contentUrl":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/wp-content\/uploads\/2023\/04\/cropped-bmartwork.png?fit=1220%2C480","width":1220,"height":480,"caption":"Derek Mueller"},"logo":{"@id":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/wp-content\/uploads\/2023\/04\/cropped-bmartwork.png?fit=1220%2C480"},"sameAs":["http:\/\/www.derekmueller.net\/","https:\/\/www.facebook.com\/derek.mueller","https:\/\/www.instagram.com\/dereknmueller\/","https:\/\/www.youtube.com\/user\/dnmueller"],"url":"https:\/\/www.earthwidemoth.com\/blog\/author\/dmueller\/"}]}},"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7tdGk-1Tp","jetpack-related-posts":[{"id":6248,"url":"https:\/\/www.earthwidemoth.com\/blog\/2005\/02\/26\/perl\/","url_meta":{"origin":7279,"position":0},"title":"Perl","author":"Derek Mueller","date":"February 26, 2005","format":false,"excerpt":"For the last twenty-four hours, I've been obsessing and scrambling to figure out how to modify a perl script for a project I'm working on. And just when I was beginning to feel like things were near collapse and failure was\u00a0imminent, along came an email reminding me that I've already\u2026","rel":"","context":"In "Technologies"","block_context":{"text":"Technologies","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/technologies\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/img\/perl.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":6239,"url":"https:\/\/www.earthwidemoth.com\/blog\/2005\/02\/14\/rgb-averages-pi\/","url_meta":{"origin":7279,"position":1},"title":"RGB Averages; Pixel as Metonymy","author":"Derek Mueller","date":"February 14, 2005","format":false,"excerpt":"{125, 127, 73} (Average RGB from the butterfly image below when rinsed into a single pixel.\u00a0 Expanded again for easy viewing.) Bumgardner explains Color Pickr in the comments over here. I use a Perl script to retrieve all the thumbnails of all the photos in the group, which takes a\u2026","rel":"","context":"In "Media"","block_context":{"text":"Media","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/media_massage\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.earthwidemoth.com\/blog\/img\/moth.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":6284,"url":"https:\/\/www.earthwidemoth.com\/blog\/2005\/04\/12\/blogospheric-eq\/","url_meta":{"origin":7279,"position":2},"title":"Blogospheric Equilibration","author":"Derek Mueller","date":"April 12, 2005","format":false,"excerpt":"Okay, so they're probably not connected, but they are two notable blips in my morning glance at the blogosphere. Penn State professor Stuart Selber announces he will disconnect his blog. The Council of WPA launches a fresh (er...idle for one month, but hey) site.\u00a0 With RSS. What's a WPA?\u00a0 This,\u2026","rel":"","context":"In "On Weblogs"","block_context":{"text":"On Weblogs","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/on_weblogs\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6694,"url":"https:\/\/www.earthwidemoth.com\/blog\/2007\/02\/06\/trouble-shot\/","url_meta":{"origin":7279,"position":3},"title":"Trouble Shot","author":"Derek Mueller","date":"February 6, 2007","format":false,"excerpt":"Even if the following fixes are only useful to one or two people, posting them to the blog makes them differently available for searching and bookmarking. Since I installed MT3.34, I ran across a couple of small snags. Nothing too off-putting, really. Just bumps along the up-gradual way. First, the\u2026","rel":"","context":"In "On Weblogs"","block_context":{"text":"On Weblogs","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/on_weblogs\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":7112,"url":"https:\/\/www.earthwidemoth.com\/blog\/2007\/08\/24\/peeples-seeing\/","url_meta":{"origin":7279,"position":4},"title":"Peeples, “‘Seeing’ the WPA With\/Through Postmodern Mapping”","author":"Derek Mueller","date":"August 24, 2007","format":false,"excerpt":"Peeples, Tim. \"'Seeing' the WPA With\/Through Postmodern Mapping.\" The Writing Program Administrator as Researcher. Shirley K. Rose and Irwin Weiser, eds. Portsmouth, NH: Boynton\/Cook, 1999. 153-167. Peeples devises a set of maps in an effort to \"capture a sense of spatial positioning and the fragmentation of [Wedy] Bishop's position\" as\u2026","rel":"","context":"In "Reading Notes"","block_context":{"text":"Reading Notes","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/reading_notes\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6841,"url":"https:\/\/www.earthwidemoth.com\/blog\/2007\/10\/27\/multiple-sequen\/","url_meta":{"origin":7279,"position":5},"title":"Multiple, Sequential, Reciprocal","author":"Derek Mueller","date":"October 27, 2007","format":false,"excerpt":"This one is from the same Nagi Noda who made \"Sentimental Journey,\" the other when I'm observed, I watch this. I think these three--multiple, sequential, reciprocal--ought to apply to teaching observations. Were I a WPA, I would prefer an approach to classrooms observations that involved multiple visits in a sequence\u2026","rel":"","context":"In "Dry Ogre Chalking"","block_context":{"text":"Dry Ogre Chalking","link":"https:\/\/www.earthwidemoth.com\/blog\/category\/dry_ogre_chalking\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/posts\/7279","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/comments?post=7279"}],"version-history":[{"count":0,"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/posts\/7279\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/media?parent=7279"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/categories?post=7279"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.earthwidemoth.com\/blog\/wp-json\/wp\/v2\/tags?post=7279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}