What do contractors in machine learning charge by the hour?

I saw this question on Quora this morning: "What do contractors in machine learning charge by the hour?"

Obviously the answer depends on the skill level, but based on the few machine learning contractors I've worked with on oDesk, I would guess about $30-$40/hour is average. However, I wanted to check with some real data. I have access to our full database, but a more useful answer would allow people to see how to do this for other skills by simply scraping our search results (we have an API, but I don't know how to use it yet and I wanted to try the BeautifulSoup package).

First step was to reverse-engineer our search syntax and the HTML for profile rates. A contractor search of "machine learning" gives this:

"https://www.odesk.com/contractors?nbs=1#q=machine+learning"

and if I click on the next 10 results, the url is:

"https://www.odesk.com/contractors?nbs=1#q=machine+learning&skip=10"

I then checked to see what happens if I set "skip=0"---it returns that same thing as the first search URL, so now know that I can just write one function that returns the query URL, with parameters of "q" and "skip."

Next, I needed to find out how the rates are stored. Using Chrome's awesome "Inspect element" feature, I found that the rates for the 10 returned results as listed as "rate_1", "rate_2", and so on:

The rest was pretty easy---I just wrote two loops to collect up wages. I saw that we had some clear false positives, which I filtered out. This actually brings up a big problem on oDesk that we've been working on---namely that until recently, we had no standardization of skills, which made it hard to match people or do really good, highly specific queries. We've now moved to a closed (but expandable) vocabulary of skills, ala StackOverflow which in the long run will make it much easier to do matching and recommendations (and little data projects like this). That's a topic for another blog post. So, returning to the original question, my answer based on a pretty tiny sample:


Min: 16.67
Max: 100.0
Mean: 39.6983333333


And here's my pretty crappy but for-the-moment-functional code on GitHub:

Should Online Labor Markets Set a Minimum Wage?

Some critics of online labor markets mistakenly believe that the platform creators have an incentive to keep wages low. Employers have that incentive, but all else equal, the creators of online labor markets want to see see wages rise, since almost all of them take a percentage of earnings. At least from a revenue standpoint, they should be indifferent between more work at a lower price or less work at a higher price, so long as the wage bill stays the same. It would be a different story if the platforms could somehow tax employers on the value-added by the platform, but so far, that's not possible.

One tool for raising wages might be for the platform to impose a minimum wage. It's certainly possible that imposing a binding minimum wage would increase platform revenue---it depends on the relative labor supply and demand elasticities, as well as the marginal cost of intermediating work. A platform's costs of good sold (i.e., intermediation service) is not precisely zero---there are server costs, customer service costs, fraud risks etc, so there is some wage at which the platform would be better off not allowing parties to contract.

Moving from generalities, let's look at workers from the Philippines, who (a) make a big chunk of the workforce on oDesk and (b) generally do relatively low-paid work (e.g., data entry, customer service, writing etc.) and thus would be most affected by a minimum wage imposition. If we look from about 2009 on (when the Philippines first started to become important), we can see that wages are basically flat, perhaps with a slight rise in some categories.

We can see that mean hourly wages range from $3 (for low-skilled data entry work) to about $8/hour for software development. By US standards, $3/hour is quite low---it's less than half the US federal minimum wage. However, let's look at where $3/hour wage puts someone in the the Philippine household income distribution, assuming they work 40 hours a week, 50 weeks a year: 

Unfortunately the I couldn't get a more refined measure of income, but my eye-ball estimate is that $3.00/hour is at about the 50th percentile of the distribution. The equivalent hourly wage for median household income in the US is about $31/hour (using 2006 measure from Wikipedia) using the same 50 weeks a year, 40 hours a week formulation. It's important to note that this is household income, meaning that in many cases it is the combined income of a husband, wife and working-age children. And although online work does require a computer and a good internet connection, it does not require spending money on transportation, work clothes, food prepared outside the home etc. It also probably lets workers economize on child care e.g., I might be willing to let a 10 year old watch a 3 year old if I'm in the next room, but not if I'm across town.

So, what's the conclusion?

From a platform perspective, I can concede that imposing a minimum wage could be revenue-increasing, but it depends on some pretty hard to estimate factors: how well do we know the elasticities? Are the long and short-term elasticities the same? What happens if we can get our intermediation costs down? Implementation-wise, enforcement might be very hard---I could easily imagine workers giving under-the-table rebates.

From a worker/welfare perspective, a minimum wage would clearly help some but hurt others. Any binding minimum wage is going to price some workers out of the market. How do we weigh their lost opportunities against the increased wages paid to those that see a bump? This starkly highlights one of the real drawbacks of a minimum wage as social policy, which is that it might be globally progressive and yet highly locally regressive for workers on the bad side of the cut-off.     

I'd love to hear both employer and worker perspectives on this---feel free to comment here & I'll respond.

Like this? Follow me on twitter. 

Workers-as-Bundled-Goods

 Image by Simon Miller 

A standard pricing strategy in many industries is bundling goods, e.g., productivity "suites" like Microsoft Office, value meals at fast food restaurants, hotel and flight combos, etc. In the labor market, we also see a kind of bundling, though not by design: each worker is a collection of skills and attributes that can't be broken apart and purchased separately by the firm. For example, by hiring me, my company gets my writing, meeting attendance, programming, etc.; they can't choose to not buy my low-quality expense-report-filing service.

Good mangers deal with this bundling by keeping workers engaged at their highest value activity. However, every activity has decreasing marginal returns, so even activities that start out as high-value eventually reach the "flat of the curve" where the marginal benefit of more of X gets pretty small. This phenomena gives large firms an advantage, in that their (generally) larger problems give workers more runway to ply their best skills (by the same token, small firms have to worry much more about "fit" within their existing team).

While pervasive, this flat-of-the curve dynamic and the resulting small-firm handicap is not a fundamental feature of organizations or labor markets--it springs from the binary nature of employment. It goes away or it least is diminished if a worker can instead being partly employed (i.e., freelance) at a number of firms, each paying the worker to do what they do best. To date, the stated value proposition of most freelancing sites has been that they allow for global wage arbitrage. Obviously that's important, but I suspect this "unbundling" efficiency gain will, in the long term, have a more profound effect on how firms organize and how labor markets function.

Some light data munging with R, with an application to ranking NFL Teams

I recently submitted this blog to R-bloggers, which aggregates R-related blog posts. It's a fantastic site and has been invaluable to me as I've learned R. One of my favorite kinds of articles is the hands-on, "hello world"-style weekend project that dips into a topic/technology, so here's my first attempt at one in this style.

First, some background: I've been working with Greg on a project that analyzes the results of two-person contests. An important part of the problem is comparing different ranking systems that can adjust for the strength of the opponent (e.g., Elo rating system, TrueSkill, Glicko, etc.). As I understand it, all of these systems are working around the intractability of treating this as a purely Bayesian solution and try to deal with things like trends in ability, the distribution of the unobserved component, etc.

We're still collecting data from a pilot, but in the interim, I wanted to start getting my feet wet with some real competition data. Sports statistics provide a readily available source of competition data, so my plan was:

1. Pull some data on NFL games on the 2011 season to date.
2. Fit a simple model that produces a rank ordering of teams.
3. Pull data on ESPN's PowerRanking of NFL teams (based on votes by their columnists), using the XML package. 
4. Make a comparison plot, showing how the two ranks compare, using ggplot2.

For the model, I wanted something really simple (hoping no one from FootballOutsiders is reading this). In my model, the difference in scores between the two teams is simply the difference in their "abilities," plus an error term:

where the alpha's are team-and-venue (e.g., home or away) specific random effects. For our actual rating, we can order teams based on the sum of their estimate home and away effects, i.e.:



Estimating the 32 x 2 parameters---given how little data we actually have---would probably lead to poor results. Instead, I used the excellent lme4 package which approximates a Bayesian estimation where we start with a prior that the alpha parameters are normally distributed.

Putting the last thing first, here's the result of 4), comparing my "homebrew" ranking to the ESPN ranking, as of Week 5 (before the October 9th games):

No real comment on my model other than it thinks (a) that ESPN vastly overrates the Chargers and (b) more highly of the Ravens.

The code for all the steps is posted below, with explanatory comments:

All public government data should be easily machine readable

The Bureau of Labor Statistics (BLS) has an annual budget of over $640 million (FY 2011), a budget  they use to create and then distribute detailed labor market data and analysis to policy makers, researchers, journalists and the general public. I can't speak to the "creation" part of their mission, but on the "distribution" part, the are failing---organizations with tiny fractions of their resources do a far better job.

It's not the case that government IT is invariably bad---the Federal Reserve Bank of St. Louis has an amazing interface (FRED) and API for working with their data. Unfortunately, not all government statistics are available here, especially some of the more interesting BLS series.

The essential problem with BLS is that all of their work products---reports, tables etc.---are designed to be printed out, not accessed electronically. Many BLS tables are embedded in PDFs, which makes the data they contain essentially impossible to extract; non-PDF, text-based tables, which are better, are difficult to parse electronically: structure is conveyed by tabs and white space, column headings are split over multiple lines with no separators; heading lengths vary etc.


Why does it matter? For one, when users can access data electronically, via an API,  they can combine it with other sources, look for patterns, test hypotheses, find bugs / measurement errors, create visualization and do all sorts of other things that make the data more useful.


BLS does offer a GUI tool for downloading data, but it's kludgy, requires a Java Applet, requires series to be hand-selected and then returns an Excel(!) spreadsheet w/ extraneous headers and formatting. Furthermore, it's not clear what series and what transformations are needed from GUI-data to make the more refined, aggregated tables.

To illustrate how hard it is to get the data out, I wrote a python script to extract the results this table (which shows the expected and estimated changes in employment for a number of industries). What I wanted to do was make this, which I think is far easier to understand than the table alone:

To actually create this figure, I needed to get data into in R by way of a CSV file.  The code required to get table data into a useful CSV file, while not rocket science, isn't trivial---there's lots of one-off/hacky things to work around the limitations of the table. Getting the nested structure of the industries e.g., ("Durable Goods" is a subset of "Manufacturing" and "Durable Goods" has 4 sub-classifications) required recursion (see the "bread_crumb" function). FWIW, here's the code:






Most of the code is dealing with the problems shows in this sketch:

My suggestion: BLS should borrow someone from FRED and help them create a proper API.

We can always get jobs working at the local robot factory

The key quote from Slate's recent "robots-will-take-our-jobs" article:

"Most economists aren't taking these worries very seriously. The idea that computers might significantly disrupt human labor markets--and, thus, further weaken the global economy--so far remains on the fringes."

And rightly so. Obviously predicting the future is a fool's errand, and perhaps advances in AI and robotics will ultimately radically reduce the demand for human labor, but a recent article highlights how technological advances can just as easily increase labor demand: NPR reports on an oil-related boom in North Dakota that's driven unemployment close to zero and brought thousands into the state. The cause of the boom is not high oil prices per se---it's technological developments like fraking & horizontal drilling that make formerly non-viable deposits economical to extract.

Obviously technological can displace human laborers and have large effects on the composition of jobs and the returns to different skills, but history and economics both suggest that technology-driven fears about labor markets are overblown.    

Using oDesk to Satisfy Curiosity

Note: This is a re-posting of a Quora post a wrote a few days ago.


A few days ago, László Sándor, one of the economists in my "economics" Google+ circle posted a photograph showing the Hungarian PM's notes for a speech to parliament. As you can see from the picture, the final product was not the result of, uh, careful policy analysis:

Linke on Google+: https://plus.google.com/104111354079417095932/posts/UdLvwbMXtL1

Like most non-Hungarians, I can't read what was crossed out nor what was added. Normally when I find something in a language that I can't read but want to understand, I try Google Translate, but that would not work here since the text is an image and the handwritten notes are probably the more interesting part anyway.

In response to the post, Markus Mobius asked "Is there a site that translates this stuff?" and the answer is yes, in the sense that there is a site where you can (try) to get anything done, so long as the work can be sent down a wire. That site is run by my employer, A few days ago, László Sándor, one of the economists in my "economics" Google+ circle posted a photograph showing the Hungarian PM's notes for a speech to parliament. As you can see from the picture, the final product was not the result of, uh, careful policy analysis:

Linke on Google+: https://plus.google.com/104111354079417095932/posts/UdLvwbMXtL1

Like most non-Hungarians, I can't read what was crossed out nor what was added. Normally when I find something in a language that I can't read but want to understand, I try Google Translate, but that would not work here since the text is an image and the handwritten notes are probably the more interesting part anyway.

In response to the post, Markus Mobius asked "Is there a site that translates this stuff?" and the answer is yes, in the sense that there is a site where you can (try) to get anything done, so long as the work can be sent down a wire. That site is run by my employer, oDesk. I often post small jobs on the site , partly for fun and partly because I want to see what I can get done at what level of quality.

I decided to try getting the speech translated. I like posting translation jobs as demonstrations, because they nicely illustrate one important aspect of online labor markets, which is that you can quickly find workers that can do very specialized tasks. I tend to think that the real efficiency gain of these markets comes not from arbitraging wages (though that's a real component and a very powerful one---see Samasource), but in terms of the gains from specialization.

When I taught my "online work" sophomore economics seminar at Harvard, I started the class by posting a job looking to translate the Wikipedia article on the TV show Madmen into Tagalog (chosen because I knew lots of Filipino contractors would be online at that time). By the end of the hour, we had (1) posted a job (2) hired a contractor (her first job) (3) taken delivery of the translation and (3) paid for the work and (3) given feedback to the contractor.

Anyway, so here's what I did:

Writing a job description

Here's what my job post looked like to potential contractors---not much detail, but then again, this is a pretty well-defined task:


Inviting applicants; Screening the results

I was thinking that Hungarian is a fairly uncommon language on oDesk, so I suspected that I would have to go recruit a translator. I did this & hired the first person who had a reasonable profile. However, in retrospect, I could have posted & waited: I got 12 applicants in less than 24 hours.

How did my non-invited applicants look?


So we have some very promising candidates (from Romania or Hungary & focus on writing/translation). There are also a handful of candidates from elsewhere that don't seem relevant---they don't focus on translation and their country makes me doubt they could do a good hungarian-to-english translation.

This brings up a point about one the challenges of our marketplace. Many oDesk employers complain about application "spam" which they regard as applications from workers that are inappropriate for a job or who haven't read the job's instructions. We try to reduce this problem by giving contractors a "job application quota" (so that they will be more choosy in the jobs they apply to & invest more in their applications) but it's far from perfect and getting the "supply" of applications to match the demand is tricky since applications are an externality (and whether they are positive or negative depends on the job & the worker---one man's spam is another man's treasure).

The data definitely reflects what employer's say: here is a plot showing the relationship between cover letter novelty (defined as the fraction of words in a current cover letter that are common with that contractor's most previous application) and the probability of that worker getting an interview:


Obviously this isn't causal evidence, but the pattern is pretty clear. Customized, job-specific applications are much more likely to lead to interviews (and then hires).

Ok, getting back to the translation job. I hired the first applicant and had a few back and forth messages with her about whether to translate the whole thing & how to treat the handwritten stuff. Once she had the details, she started working (in fact, she's working right now, as I write this---note the green "Working Now" in the image below:


Monitoring her progress

Let's take a deeper look at what's she doing:

You can see her screen (!)?
So this is the part of oDesk that often wigs people out---I can actually look at screenshots from her computer, taken approximately every 10 minutes at random intervals (so long as she's using the team room client, which she needs to do to bill hours). This strikes people as incredibly big-brotherish at first. But this is actually what makes oDesk work for hourly contracts. If employers cannot monitor what their workers are doing, they will not hire workers on an hourly basis because it is too easy to pad hours. (An aside: I once hired an offline editor that billed me for 3 hours when the largest time delta in the edits (which I can see from the time-stamped tracked changes) was about 45 minutes).

The lack of trust in online relationships (at least at first) is what drives many people to use fixed price contracts (e.g., see Amazon Mechanical Turk), since it deals with the padding issue, but using fixed price contracts creates other strategic issues, one of which is quality---workers have an incentive to cut corners. More importantly, with a fixed-price contract, the parties have to over-invest in writing nearly complete contracts (see Steven Tadelis's paper with Patrick Bajari on this fixed-price vs. cost contract issue---this paper really influenced my thinking about contracting). Mechanical Turk "solves" the fixed price hold-up problem by just letting the employer decide whether or not to pay, with no recourse for the worker. This clearly doesn't work as the stakes get beyond the penny range.

Practically speaking, once you get comfortable with a contractor, you rarely check the screen-shot work diary in a monitoring sort of way---more in just a "what's everyone up to?" sort of way.

Unscrupulous Employers and Online Work
One thing that is obvious with a translation job is that the screenshot work diary would let an unscrupulous employer steal a worker's output. Note that I can zoom in and actually see the in-progress translation:


The fact that the screen shots are images limits this problem to an extent, but for something like programming where you are asking coders to commit their work to a repository daily, stealing becomes far easier if employers can decide later not to pay. For this reason, oDesk guarantees that contractors will get paid for hourly work.

Getting the Translation
So as I write this, she's still working, but let's check out the first paragraph:


That's a pretty interesting idea, though I wonder if the government running special shops (if that's what this network of shops idea actually implies) is a good idea. Surprisingly, the prime minister is from the conservative party(?):
http://en.wikipedia.org/wiki/Viktor_Orb%C3%A1n

Anyway, perhaps I could have gotten all of this information from László Sándor if I had asked him nicely, but at some point he'd get tired of satisfying my idle curiosity and I'd rather wait until I have some request or question that I couldn't get answered for about $10 and 10 minutes of my time. When I get the full translation, I'll post a link.

Update: Here are the results--- http://dl.dropbox.com/u/420874/Overhead%20maximal.docx.