Last Updated: 2003/01/07

• What happened on Oct 22, 2002 to cause that huge spike of more than 3500 listings?
• How far back does the data go?
• On what do you base the SkillMarket numbers?
• Why do you chart C++ and not just plain C?
• Ask another question

What happened on Oct 22, 2002 to cause that huge spike of more than 3500 listings?

A few weeks before that date, dice.com modified their site. If you remember, they launched a redesign in early October. (Anyone remember the exact date?) I thought I had adjusted for it the day it happened, but I was wrong. What they had done, I since learned, was forked their database. The "old" database was still up and running, but no new listings were being posted to it, and their site didn't point to it anymore. As time passed, the listings that were in the database at the time started expiring out and the number of listings started to fall.

After investigating the issue, that's when I realized the database I was querying was no longer thier "live" database -- the one that was getting the new listings. I adjusted my polling routine to poll the new, live database.

The huge spike on Oct 22, was the data being re-aligned with the dice.com database. Unfortunately, this means that for a few weeks prior to that date, each day represents an incremental diversion from reality.

I now do an eyeball check every couple of days to compare the number of overall listings, just to be sure it doesn't happen again.

You'll also note that's when I was able to start tracking C# and C++ separately. Dice.com improved their search engine to support that. So, there is that one plus.

How far back does the data go?

It varies. I first started gathering data on April 3, 2002. There is no data before that.

On April 18, 2002, I added 'cf' and refined the query for 'vb'. Numbers for 'vb' before that are artificially inflated. On May 25, I added 'apache' and 'iis', thanks to a user who pointed out the omission. Others may be added or deleted at later dates.

Sorry, I have no information to be culled relating to the dot-bomb or 9/11.

On what do you base the SkillMarket numbers?

Every morning, a process polls the leading tech job listing site (dice.com) for each of the search terms to get the number of hits returned.

If you go to dice.com and search for 'perl,' you should get the same number as reported by SkillMarket -- give or take a few because their number is dynamic and may change minute by minute.

A note on the way the query is built, so you can verify the accuracy:

A term like 'cf' is built like so:
[cf "cold fusion"] (without the brackets)
with the 'match any term' parameter set.

Similarly, 'mssql' is built like so:
[mssql "ms sql" "sql server"]
with the 'match any term' parameter set.

The polling process sleeps a short while in between requests, of course, and runs at a low-traffic time.

Why do you chart C++ and not just plain C?

'C', 'C++' and 'C#' all produce the same search results on dice.com, so they can't be tracked separately. I settled on c++ as the official term I use, and will modify if and when dice.com distinguishes them from each other. This is far from perfect.
UPDATE: This problem has been alleviated a bit. I am now able to search for 'c++' and 'c#' separately, but just plain 'c' doesn't produce any meaningful search results;