Visualizing the Internet

Tuesday, July 01, 2008

[Images on this page may take a while to load.]

A few months ago I wrote about Internet as a high-stakes game played by millions of people.

The Internet, you see, is an enormous, three-dimensional gaming surface. This gameboard consists of billions of nodes (web pages) joined by trillions of connectors (hyperlinks), fluctuating and evolving in real-time. It helps if you visualize it as a physical thing, and to do that, we'll need to put on our magic decoder glasses.

In the comments to that post, I said I was working on an Internet visualization tool or agent. Today I'd like to introduce that tool, code-named Inviz, short for "Internet Visualizer". It's the piece of software responsible for the images you're about to see. They were generated entirely inside this tool, in a matter of a few minutes.

Get those magic decoder glasses ready. Our first Internet snapshot is in black and white:

That's a living, breathing picture of a small galaxy of web sites as they exist in the real world. Each block in the above diagram is a single web page. Each line is a single hyperlink (or other association between two pages). Everything you see can be manipulated, dragged and dropped, resized, queried for its properties and content. If you double click a node, the tool traces the nodes hyperlinks and opens up another galaxy of nodes, and so forth. As you can see, after just a few minutes of working with the tool, the relationships can get complex.

Because the Internet itself is complex, and all Invis really does is take a (somewhat) intelligent picture of that complexity.

Let's zoom out for the bird's-eye view, and tell Inviz to color-code sites according to the domain they belong to:

It's a battleground, our Internet. Can you guess which galaxies belong to Digg.com? Microsoft? Coding the Wheel? Or Jeff Atwood over at Coding Horror?

Give up?

Let's zoom in (Inviz supports zooming by any arbitrary scale factor) and take a look. Here's the diagram at 500%.

So, at least in this particular view, Inviz has chosen a red-white gradient fade for Digg.com.

The galaxy of red blips you saw in the 2nd picture? The Digg.com home page, and (some of) the pages it links to. Light blue nodes? Various Microsoft sub-domains. Dark blue nodes? Good old Jeff Atwood over at CH. Black nodes? Sites we don't care about.

Of course, colors, shapes, and connectors can all be customized by the user. In fact, we can do away with the graphical view entirely, and boil the entire web down to a simple tree control.

This link hierarchy is the well-known factor behind Google's PageRank. And although not shown here, Inviz has an option allowing you to tweak the size, shape, color, or other aspects of a given element based on its incoming links, resulting in a sort of visual PageRank. Now if I could only get access to the Google master index...

Anyway, this diversion has been brought to you by...

Inviz was implemented using C# and .NET 2.0+ exclusively. It is a work in progress. Inviz is not currently available for download or purchase. That may change if there's any interest. If you'd like to see a picture of your own website as seen through Inviz's eyes, leave a comment. It's capable of some very strange visualizations, and even has some useful HTTP functionality under the hood.

Until next time.

Tags: PageRank, .NET, C#, generative graphics

29 comment(s)

Here's a couple of [url=http://flickr.com/photos/mywot/sets/72157605676861378/]similar pictures[/url] of a some major porn sites.

I'm developing a similar tool as part of an exam. Yours looks really better of course... i'd really like to take a look at your code... Which graph drawing algorithms does Inviz use?

Amazing pictures. You say you can do full drag and drop with this tool? How does that hold up using C#?

Pretty.. but what does this have to do with building a poker bot? ;) Or can the poker bot surf the Internet too? :)

James, I'm curious. Obviously in order to generate these pictures of the Internet, the "bot" (that's what Inviz is, right?) has to send out HTTP requests and parse the returned responses. In other words, this is essentially a spider bot such as that used by the search engines. My question is what user agent string you're using for it? "Inviz v1.0"? Or does it use an IE user-agent string and so forth? Or are you perhaps using a third-party spider?

If C#/WinForms is capable of generating that UI with drag and drop I'll eat my shirt. What're you using for the GUI? Be honest now... DirectX? Open GL? Maybe an SVG library of some sort?

If you can package the display code as a component I would gladly pay money for it. Assuming it's interactive as you say it is, which I doubt because C# is so slow. Post a working demo so we can have a chance to use this tool. Or just admit you built everything in Photoshop. ;)

like slacker, i am a student and like slacker, i also would like to hear more about the drawing algorithms. are you using a graph layout algorithm or just winging it? does it spider nodes recursively and if so, how does it store them? and can you send me a demo i'd like to take a few prints of some of my websites.

The above images remind me of another blogger, I forget his name but the link is here:

http://www.flight404.com

Look at his work on visualizations.. performed in the open-source Processing language.

I'd like to see some more of these. You should do a big collage and post to Flickr.

Hi James,

I wrote you a private message about a link exchange. I'm going to scrap the idea of a link exchange and if you mention our site somewhere in this blog we'll post your link up on the front page. The anchor text can be "How to make a poker bot" and seeing as our site is "www.pokerisrigged.com"... I think you'll be seeing a lot of interest. We are currently getting ~70 UVs a day.

Look forward to hearing back, Nick.

I find this kind of work really intriguing.. I think you need to make some enhancements when the nodes are shown at high zoom- surely you can do something better with that screen real estate than to just show the site URL? but I'm guessing you're aware of that. Visually the colors and layouts are quite striking, but how much functionality is under the hood?

And: where's the demo? This tool loses a lot of its value if people aren't allowed to play with it. ;) Just my .02.

These diagrams are fascinating. If Google would hook something like this into their crawlers maybe we could have a map over the entire Internet (as reachable by links, that is).

One interesting question you could answer with such a map is "What is the least number of clicks you need to take yourself from page X to Y."

Is it possible to hook a windows application using Java?

this is awesome. distribute this!

You didn't answer to me :(

However, if someone is interested i've found an interesting tool at [url]http://whatsonweb.diei.unipg.it/[/url]

You should package this as a Mozilla add-in.

Or a standalone tool like Fiddler.

Hey James, I know you've got a full plate, what with PokerStars harassing you & all. But when is the 'Inviz' product shown above going to go live and will it be open-source? Or at least free? That would be cool. Cheers~

[quote]That may change if there's any interest.[/quote]

I'm interested!!! Very much so!

Been a while since you've blogged. Thanks for the follow up. I do a lot of SEO and it's really interesting to see how the internet can be demonstrated in a diagram.

Hey James can you visualise my website please. Would be really interesting to see what it looks like. http://www.pokerisrigged.com.

Hey James, I know you've got a full plate, what with PokerStars harassing you & all. But when is the 'Inviz' product shown above going to go live and will it be open-source? Or at least free? That would be cool. Cheers~

I'm interested. I don't really want to visualize the internet (it's too big to be interesting). I want to visualize our non-hierarchically structured intranet that contains a web of hyperlinks but not that many pages. Finding a tool like this from teh intarwebs is proving difficult if not impossible.

ghd rettetang heat up really quick, and they leave your hair shiny and soft! Also your hair stays straight for ages! I loved them! They are the next best thing. Great ghd glattetang performers and even the next day your hair looks freshly straightened - and I'm talking about someone with really frizzy hair! Best billig ghd ever bought and I've had all the big named ones before I bought these ghd rettetang i norge!! I got ghd slettetang today. My girlfriend very like it. it make us very happy. thanks for the surprise gift. I love my ghd rettetang! It heats up very quickly and works well. My hair is annoyingly thick and poofy, but this works wonders!

4日の参院予算委員会では、小沢グループの奥村展三文部科学副大臣と中塚一宏内閣府副大臣が消費税法案に賛成する意向を表明した。奥村、中塚両氏は小沢元代表の意向に反し辞表を提出していない。

 首相は辞任した4人の後任について、消費税法案への賛否を確認した上で起用する方針だ。

Use the form below to leave a comment.






Coding the Wheel has appeared on the New York Time's Freakonomics blog, Jeff Atwood's Coding Horror, and the front page of Reddit, Slashdot, Digg.

On Twitter

Thanks for reading!

If you enjoyed this post, consider subscribing to Coding the Wheel by RSS or email. You can also follow us on Twitter and Facebook. And even if you didn't enjoy this post, better subscribe anyway. Keep an eye on us.

Question? Ask us.

About

Poker

Here there be Code. And poker. And technology, arrrr. And games. And...


Hire

You've read our technical articles, you've tolerated our rants and raves. Now you can hire us anytime, day or night, for any project large or small.

Learn more

We Like

Speculation, by Edmund Jorgensen.