Web Crawler Status…
January 25, 2006 | 8:10 pmWell, I just started writing a Stone-Age style WebCrawler some 2 days ago. But, I’ve actually spent just 3 hours on it since that time, due to my Exams….
Well, so far till today, I’ve got quite something to show for : It’s written in VB.NET, is completely Multi-Threaded, extracts plaintext as well as regular hyperlinks, and MessageBoxes the contents of the HTML Files it downloads:P
Well, I haven’t run into any Multi-Threading issues yet. And, I’ve yet to start on Lucene….
Well, it’s still got lots of things to get done : Respecting Robots.txt, Index the stuff using Lucene, and quite a lot of stuff….
Well, trying to get the stuff I’ve done till now online.. Will Post a Link as soon as I get it online…
Update : Here is the Link for the Source ZIP File. About 423 KB. Requires .NET 2.0, and Visual Basic .NET 2005 Express or the Visual Studio 2005 any Editions….
Well, just download it and see how it works. The Search and Index Facility is not yet in though, but expect to be flooded with MessageBoxes without any sign of Ending:P
Well, Just comment how it is. Comments on the Code Quality and Design are especially welcome, and are even Requested 
Thanks, and Keep the comments coming….
