YuviSense: Codin Kid

After reading Sriram’s old post on building a search engine, I wanted to teach myself screen scrapping. So, I just knocked up Visual Basic 2005 Express Edition, and for no apparent reason, named the project PaleRash. Then after learning the hard way that most sites aren’t XHTML compliant (heck, even the old digg!), I looked around for a .NETty forgiving HTML parser, and thankfully found the awesome HTMLAgilityPack by Simon Mourier.

Now, with the technical side settled, I needed a target. A simple blog with a semantically meaningful structure. And the first name that popped up in my head was Scoble.

I’ve been reading Scoble since I arrived at the blogsphere, mostly as a news source: A human filter who filters the news so that it might not alter my faith in Microsoft Religion, add some opinions of his own, and boldly debunk some of the facts that could make the heretics lure out the faithful.

Anyway, so after a weekend of coding, a week of debugging, and another weekend of rewriting, I finally had a base which could be modified to run link and word statistics on any blog. The results were interesting, to say the least.

Now, since writing a single post about all that would be a crime on humanity, I’m going to split it up. So, here’s the part zero, just the lame General Statistics. Stay tuned, for I have more in the works.

General Posting Statistics

I used all the posts from scobleizer.com, from the time he switched to Wordpress.com (1/10/05) including the “Hello World” post, till 22/12/06, or the time during which he didn’t post for two days. That gives us 447 days of blogging, which equals to 2405 posts, at an average of 5.3 posts a day.

What’s a statistics post without graphs? I’ll forewarn you that today’s charts aren’t pretty interesting. Here’s our first:

That worm’s pretty straight. So straight, that after September, I could barely see the straight trend line I drew! So, though a bit rough in the early days, it has pretty much stabilized now.

General Links Statistics

In those 2405 posts, he’s linked to 4992 pages, 4321
of them unique, spread out across 1858 domains(including subdomains). That gives you an average of 11.16 links a day and 2 links a post, with 86% of links being unique.     

I love graphs!

Dang, another straight line! A slight drop in linking frequency at May-06 to Sep-06, but then it’s resumed to normal propotions.

General Word Usage Statistics

He used 464,078 words in all, at an average of 192 words a post and 1038 words a day. He however used only 14,898 unique words, including derivatives, meaning that each word repeated, on average, 31
times. In contrast, Shakespeare used 884,647 words in all, of which 31,534 were unique, and repeated each word 28 times on average^[source]. Translation: Shakespeare knew twice as many words, but Scoble wrote as much as half of all Shakespeare in a year and a half! I guess that maybe if I had considered his full blogging career, he might’ve surpassed Shakespeare in quantity, but…

One more Graph!

Another Straight line! So, the number of words he’s using is pretty constant. There’s a small cliff in September, where he started writing bigger posts, before getting back to his normal routine of smaller posts…

General Character Usage Statistics

2,103,816 Characters (or more precisely, Unicode Code Points) were used by him, including spaces, full stops, and all sorts of punctuation. That makes an average of 874 characters a post, 4706 characters a day. Poor keyboard. Poor Fingers.:) He used an average of 4 characters a word, which seems okay to me. This excludes HTML markup.
Unicode Code Points 

Yippee!! Another Straight Line!

That ends it for today. As I said, today’s post is pretty tame. As my Math teacher once said, Data is not interesting: Information is. Today’s post is data. Coming posts are information:D

Lot more charts to come!

You know those teasers they air at the end of every episode of most Animes? Here’s the equivalent:

Teasers

• Scoble linked more to Ze Frank than to Scoble Show (The domain, that is. It redirects to podtech)
• Before Joining PodTech, he’s linked only 7 times to Pod Tech. Today, PodTech is at #3 in the total number of links, and is actually #2 if you count in links to scobleshow.com, which redirect to PodTech.
• In a move that confounds me, he had more closing parenthesis than opening ones!
• The top ten sites he links to make up about one fifth of his total links.
• He has linked more to typepad.com hosted blogs than any other. Livejournal blogs are so low on the list you’d think they barely exist.
• Wordpress’s WYSIWYG editor saved him and his keyboard quite a lot of keystrokes. A lot.
• He used the word Microsoft more than the word Google:D
• He used the word I about twice as much as the word You. That egoistical bastard:D

Part I coming soon!

Any guesses on the most linked to site?

P.S. Incase you are wondering, I did respect the robots.txt file at scobleizer.com.

P.P.S: I have a whole mountain of data, and am pretty short on Ideas on what to do with it. Any suggestions?

P.P.P.S: Scoble’s now busy with a political campaign. Let’s see how long it takes till he finds this post by an almost unknown Z-lister (if he ever does!)