tag:blogger.com,1999:blog-15971862608742806672024-02-19T04:55:28.548-08:00Online LaborThoughts on the economic and technological implications of people working online.Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.comBlogger29125tag:blogger.com,1999:blog-1597186260874280667.post-50414699324607924892013-11-30T07:45:00.004-08:002013-11-30T07:45:45.350-08:00Would a job by any other name pay as much? I'm working on a project where it would be useful to know what an oDesk job is likely to pay at the time it is posted. Although there are plenty of structured predictors available (e.g., the category, skills, estimated duration etc.), presumably the job description and the job title contain lots of wage-relevant information. The title in particular is likely to identify the main skill needed, the task to be done and perhaps the quality of the person the would-be employer is looking for (e.g., "beginner", or "senior").<br />
<br />
Unfortunately, I haven't done any natural language processing before, so I'm a bit out of my element. However, there are good tutorials online as well as R packages that can guide you through the rough parts. I thought writing up my explorations might be useful to others that want to get started with this approach. A gist of the code I wrote is available <a href="https://gist.github.com/johnjosephhorton/7720452" target="_blank">here</a>.<br />
<br />
What I did:<br />
<br />
1) I took 20K recent hourly oDesk jobs that where the freelancer worked at least 5 hours. I calculated the log wage over the course of the contract. Incidentally, oDesk wages---like real wages---are pretty well approximated by a normal distribution. <br />
<br />
2) I used the <a href="http://www.rtexttools.com/" target="_blank">RTextTools</a> package to create a document term matrix from the job titles (this is just a matrix of 1 & 0 where the rows are jobs and the columns are relatively frequent words that are not common English words---if the job title contained that word, it gets a 1, otherwise a 0).<br />
<br />
3) I fit a linear model using the <a href="http://statweb.stanford.edu/~tibs/lasso.html" target="_blank">lasso</a> for <a href="http://en.wikipedia.org/wiki/Regularization_%28mathematics%29" target="_blank">regularization</a> (using the <a href="http://cran.r-project.org/web/packages/glmnet/index.html" target="_blank">glmnet package</a>). I used cross validation to select the best lambda. A linear model probably isn't ideal for this, but at least it gives nicely interpretable coefficients. <br />
<br />
So, how does it do? Here are a sample of the coefficients that didn't get set to zero by the lasso, ordered by magnitude (point sizes are scaled by the log number of times that word appears in the 10K training sample): <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6rh-jOXjgFTYnWHiETz7gymxAbJ1yNBmhQbLvd2pWBpjm6Z0j02s1IJRmkSh0joaACn_UuU_XxeCIwHzIeYYOEin3CHP9KYn1dtp16XPHKezftrPhen55DXlNZG2FJnNDJATSuacwBKdd/s1600/title_log_wage_prediction.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6rh-jOXjgFTYnWHiETz7gymxAbJ1yNBmhQbLvd2pWBpjm6Z0j02s1IJRmkSh0joaACn_UuU_XxeCIwHzIeYYOEin3CHP9KYn1dtp16XPHKezftrPhen55DXlNZG2FJnNDJATSuacwBKdd/s1600/title_log_wage_prediction.png" /></a></div>
The coefficients can be interpreted as % changes from the mean wage in the sample when that corresponding word (or word fragment) is present in the title. Nothing too surprising I think: at the extremes, SEO is a very low paying job, whereas developing true applications is high paying.<br />
<br />
In terms of out of sample prediction, the R-squared was a little over 0.30. I'll have to see how much of an improvement can be obtained from using some of the structured data available, but explaining 30% of the variation just using the titles is a higher than I would have expected before fitting the model. <br />
<br />
<br /><br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-31292226120731408162013-09-22T11:46:00.000-07:002013-09-22T11:46:38.146-07:00Economics for skeptical social scientistsI recently gave a talk at the "<a href="http://www.um.edu.mt/events/is1202training2013" target="_blank">Training school on Virtual Work</a>," which was held at the University of Malta. The participants were mostly graduate students and junior faculty at European universities studying some aspect of virtual work e.g., Wikipedia editors, gold farmers, Current TV contributors, MTurk workers etc. Most were coming from very different methodological background than my own and the people I usually work with---sociology, anthropology, media studies, gender studies etc. I think it is fair to say that most participants have a fairly dim view of economics.<br />
<br />
One of the organizers felt that few participants would have encountered the economic perspective on online work. I was asked to present a kind of non straw man version of economics and present the basic tools for how economists think about labor markets. Below is the result---a kind of apologia for economics, combined with a smattering of basic labor economics. I'm not the best judge obviously, but I think it was reasonably well received.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="356" marginheight="0" marginwidth="0" mozallowfullscreen="" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/26433234" style="border-width: 1px 1px 0; border: 1px solid #CCC; margin-bottom: 5px;" webkitallowfullscreen="" width="427"> </iframe> <br />
<div style="margin-bottom: 5px;">
<strong> <a href="https://www.slideshare.net/johnjosephhorton/economics-for-othersocialscientistsrelease" target="_blank" title="Economics and Online Work (a slightly misleading title though - see description)">Economics and Online Work (a slightly misleading title though - see description)</a> </strong> from <strong><a href="http://www.slideshare.net/johnjosephhorton" target="_blank">John Horton</a></strong> <br />
<br />
<br />
PS - I should write more about the school later, but one of the main take-aways for me was how (a) pervasive the acceptance of the <a href="http://en.wikipedia.org/wiki/Labor_theory_of_value" target="_blank">labor theory of value</a> was among participants and (b) how this leads to very different conclusions about almost everything that matters with respect to online work. It would be interesting to try to analyze a couple of different online work phenomena using the LTV and the marginalist approach to value. <br />
<div>
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-8169324920954456462013-09-22T02:35:00.003-07:002013-09-22T02:46:17.500-07:00Impressions from a visit to a large call centerPaul Krugman has a piece of research advice, which is to "listen to the gentiles." What he means is to pay attention to what smart practitioners say about their business in order to get economic ideas and insights. I recently had the chance to take a tour of large call center for a major global financial institution. It was interesting throughout and I thought I would share some of my notes.<br />
<br />
<b>Recruiting and Training</b><br />
<i>The company uses a tiered screening approach</i>. They start with an online test. Some proceed to phone interviews and the finally round is in-person interviews. The HR director felt that the simulated work environment test was primary predictor of future job success---this matches up quite well with the industrial psychology literature on employer screening. He also felt that the best predictor of retention was how comfortable a worker seemed up front with the demands of shift-work.<br />
<br />
<i>The company made extensive use of their existing employees to recruit new ones</i>. Referrals were highly valued because they were more likely to bring in candidates who understood the reality of shift-based call center work (and thus were less likely to turn over). It seemed to be less about bonding or reducing formal recruitment costs.<br />
<br />
<i>Long company-specific training period, but no general training (as Becker would have predicted)</i>. However, there are several other call centers in this region and the company does lose employees to them. The company-specific training period was surprisingly long (on the order of 3 months) and was conducted by more senior employees during low call volume periods. <br />
<div>
<br /></div>
<div>
<b>Compensation</b></div>
<div>
<i>The company acts like a price taker with respect to wages</i>. Compensation for new employees was determined by doing yearly market research into what competitors were paying. The company did have very high turn-over (though about in line with the industry), but there was no mention of raising wages as a solution. Their approach seems to be to wait for people to "sort out" of the job that find that they cannot handle shift work. I meant to ask about explicit performance incentives but didn't get a chance to. However, my impression was that rewards came through promotion and one-off bonuses rather than through relating payment to specific actions, despite performance being quite measurable. </div>
<div>
<br /></div>
<div>
<i>They had surprisingly rich amenities</i>. Although pay was not high, amenities were reminiscent of a Silicon Valley start-up: pleasant office, cheap and free food, free gym, concierge service etc. Some of these things seemed like amenities the firm could more cheaply offer than their competitors because of their larger size, since some were club goods. In other words, they could amortize a concierge over many more employees. </div>
<div>
<br /></div>
<div>
<b>Operations </b></div>
<i>Customers are segmented by value and routed accordingly</i>. The company is multi-national and has call centers in several locations, including Europe, Southwest Asia and East Asia. The company's clients are segmented based on value and routed to the call center that roughly corresponds to the skill level of the workers at the call center e.g., the best customers get the European call center and low-tier do not. <br />
<br />
<i>They are highly sophisticated at demand and supply management</i>. Perhaps unsurprisingly, they are good at forecasting call volumes and staffing accordingly---with all of this done semi-automatically. They can adjust supply on the fly by calling off training, meetings etc. if demand spikes via building-wide announcements of status changes.<br />
<br />
<i>There was little evidence that much technologically-driven productivity improvement was on the horizon</i>. Although the tasks are highly structured, there was no evidence that significant technology-driven productivity gains were on the horizon. All the big gains from automation already occurred many years ago (e.g., the ubiquitous "Press 1 for "Accounts"). There was no talk of Watson-like automation of responses to customer queries. The one technology they really wanted---and that would radically reduce their costs---was some easy way to verify customer identities over the phone. This alone would increase their productivity by about 20-30%.<br />
<br />
<i>There was little evidence that this would could be easily distributed</i>. Most of the firm's workplace policies seemed to be driven by concerns about regulatory compliance and fear of losing sensitive customer information and required a great deal of monitoring and control. It is difficult to imagine a substantial chunk of this work being done by a geographically distributed workforce. <br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com4tag:blogger.com,1999:blog-1597186260874280667.post-39310598388157243032013-07-10T21:50:00.004-07:002013-07-12T04:41:59.284-07:00You Can Sometimes Trust Research Done on Mechanical Turk, But It Depends on the Research Question<a href="http://www.law.yale.edu/faculty/DKahan.htm" target="_blank">Dan Kahan</a> has an <a href="http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-problems-with-mechanical-turk-stud.html" target="_blank">interesting post</a> on some of the validity problems with research conducted on Mechanical Turk (MTurk). I think I largely agree with his main point, which is that the evolution of the marketplace has been such that it's become less useful for conducting certain kinds of research. However, I do worry there's a potential baby/bathwater problem if researchers decide that "unrepresentative" or "experiment-savvy" means a useless subject pool (e.g., Andrew Gelman titled his blog post about Kahan's article "Don't Trust the Turk").<br />
<br />
I haven't done MTurk research in several years, but the external validity issue raised by the blog post is something I thought about quite a bit when I was running experiments on the platform. I wrote a section about external validity in my <a href="http://john-joseph-horton.com/papers/the_online_laboratory.pdf" target="_blank">ExpEcon paper</a> with Richard Zeckhauser and Dave Rand). They key portion is excerpted below (the source code and data for that paper are available <a href="https://bitbucket.org/johnjosephhorton/online_laboratory/wiki/Home" target="_blank">here</a>):<br />
<blockquote class="tr_bq">
<span style="color: #333333;"><b>Representativeness</b></span> </blockquote>
<blockquote class="tr_bq">
<span style="color: #333333;">People who choose to participate in social science experiments </span><span style="color: #333333;">represent a small segment of the population. The same is true of </span><span style="color: #333333;">people work online. Just as the university students who make up the </span><span style="color: #333333;">subjects in most physical laboratory experiments are highly selected </span><span style="color: #333333;">compared to the U.S. population, so too are subjects in online </span><span style="color: #333333;">experiments, although along different demographic dimensions.<br />
</span><span style="color: #333333;"><br /></span><span style="color: #333333;">The demographics of MTurk are in flux, but surveys have found that </span><span style="color: #333333;">U.S.-based workers are more likely to be younger and female, while </span><span style="color: #333333;">non-U.S. workers are overwhelmingly from India and are more likely to </span><span style="color: #333333;">be male </span><span style="color: #004080;">(</span><span style="color: #333333;">Ipeirotis, 2010)</span><span style="color: #333333;">. However, even if subjects "look like" </span><span style="color: #333333;">some population of interest in terms of observable characteristics, </span><span style="color: #333333;">some degree of self-selection of participation is unavoidable. As in </span><span style="color: #333333;">the physical laboratory, and in almost all empirical social science, </span><span style="color: #333333;">issues related to selection and "realism'" exist online, but these </span><span style="color: #333333;">issues do not undermine the usefulness of such research (Falk, 2009).<br /> </span><span style="color: #333333;"><b><br /></b></span><span style="color: #333333;"><b>Estimates of changes versus estimates of levels</b></span> </blockquote>
<blockquote class="tr_bq">
<span style="color: #333333;">Quantitative research in the social sciences generally takes one of </span><span style="color: #333333;">two forms: it is either trying to estimate a level or a change. For </span><span style="color: #333333;">"levels" research (for example, what is the infant mortality in the </span><span style="color: #333333;">United States? Did the economy expand last quarter? How many people </span><span style="color: #333333;">support candidate X?), only a representative sample can guarantee a </span><span style="color: #333333;">credible answer. For example, if we disproportionately surveyed young </span><span style="color: #333333;">people, we could not assess X's overall popularity.<br />
</span><span style="color: #333333;"><br /></span><span style="color: #333333;">For "changes" research (for example, does mercury cause autism? Do </span><span style="color: #333333;">angry individuals take more risks? Do wage reductions reduce output?), </span><span style="color: #333333;">the critical concern is the sign of the change's effect; the precise </span><span style="color: #333333;">magnitude of the effect is often secondary. Once a phenomenon has been </span><span style="color: #333333;">identified, "changes'" research might make “levels” research </span><span style="color: #333333;">desirable to estimate magnitudes for the specific populations of </span><span style="color: #333333;">interest. These two kinds of empirical research often use similar </span><span style="color: #333333;">methods and even the same data sources, but one suffers greatly when </span><span style="color: #333333;">subject pools are unrepresentative, the other much less so.<br />
</span><span style="color: #333333;"><br /></span><span style="color: #333333;">Laboratory investigations are particularly helpful in "changes" </span><span style="color: #333333;">research that seeks to identify phenomena or to elucidate causal </span><span style="color: #333333;">mechanisms. Before we even have a well-formed theory to test, we may </span><span style="color: #333333;">want to run experiments simply to collect more data on phenomena. This </span><span style="color: #333333;">kind of research requires an iterative process of generating </span><span style="color: #333333;">hypotheses, testing them, examining the data and then </span><span style="color: #333333;">discarding hypotheses. More tests then follow and so on. Because the </span><span style="color: #333333;">search space is often large, numerous cycles are needed, which gives </span><span style="color: #333333;">the online laboratory an advantage due to its low costs and speedy </span><span style="color: #333333;">accretion of subjects.</span></blockquote>
<span style="color: #333333;">
</span>
<span style="color: #333333;"> </span><br />
<span style="color: #333333;">
</span>
<span style="color: #333333;">
</span>
<br />
<pre style="background-color: white; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 12px; line-height: 1.4;"><a href="http://www.blogger.com/blogger.g?blogID=1597186260874280667&pli=1" name="cl-1075" style="color: #3b73af;"></a></pre>
<pre style="background-color: white; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 12px; line-height: 1.4;"></pre>
Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com0tag:blogger.com,1999:blog-1597186260874280667.post-25952039204233575012013-05-24T11:25:00.001-07:002013-05-24T11:25:21.812-07:00Platforms can tax externalities and generate costly signals <blockquote class="twitter-tweet">
The word "URGENT" should cost at least $100 per usage.<br />
— Merlin Mann (@hotdogsladies) <a href="https://twitter.com/hotdogsladies/status/337993350294691840">May 24, 2013</a></blockquote>
<br />
One thing that's great about platforms is that socially efficient, signal-generating <a href="http://en.wikipedia.org/wiki/Pigovian_tax" target="_blank">Pigovian taxation</a> like the kind proposed in this tweet is not a joke---you can actually do things like this, which may be one of the great advantages of markets mediated by a powerful third party.<br />
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com1tag:blogger.com,1999:blog-1597186260874280667.post-24409741344479124402013-05-21T10:24:00.000-07:002013-05-21T10:24:19.949-07:00Country-Specific Minimum Wage Data, Courtesy of Wikipedia I was looking for some data on minimum wages in various countries and found that Wikipedia (perhaps unsurprisingly) has a very nice, <a href="http://en.wikipedia.org/wiki/List_of_minimum_wages_by_country" target="_blank">well-annotated table</a>. After downloading the data & cleaning it a bit (harder than it should be), I made several plots. There were too many countries for one plot, so I made one for each (approximate) quartile. At the end of the blog post is the R code I used for fetching the data & making the plots. <br />
<br />
<h2>
<span style="font-size: x-large;">Fourth Quartile </span></h2>
<div>
NB: Some countries have exemption policies for worker or occupation characteristics, so for a more complete understanding, of say, why Australia appears to have a minimum wage more than 2x the US minimum wage, check the Wikipedia table. </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD7Dax1_FiDzCSCo5iYiDDz6X5el8hArm78GyAMFuQy4NhGqdYJ-G5uf1bHTkKnTX25XKqxfwLuBswdj0S7M89EiEsoj8rHDatXabT9gcXqeyRD60vZWecmEcjI-vXdApXXWdMPmGSeUgA/s1600/quartile_75.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD7Dax1_FiDzCSCo5iYiDDz6X5el8hArm78GyAMFuQy4NhGqdYJ-G5uf1bHTkKnTX25XKqxfwLuBswdj0S7M89EiEsoj8rHDatXabT9gcXqeyRD60vZWecmEcjI-vXdApXXWdMPmGSeUgA/s640/quartile_75.png" width="640" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h2>
<span style="font-size: x-large;">Third Quartile </span></h2>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ9URgO-F2X4DBS4Ho71-pVe45el_Lc5-wABkH2WEC0qJEnsPSDj6e96-YPNF5AgMFQLg7EWzSUaqHsX1j9CPce5faFP2NMBZCI03FyN-lr0iVjpoXV6SvLzb3wG5zyP7fEwzrsH3qyhCP/s1600/quartile_50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ9URgO-F2X4DBS4Ho71-pVe45el_Lc5-wABkH2WEC0qJEnsPSDj6e96-YPNF5AgMFQLg7EWzSUaqHsX1j9CPce5faFP2NMBZCI03FyN-lr0iVjpoXV6SvLzb3wG5zyP7fEwzrsH3qyhCP/s640/quartile_50.png" width="640" /></a></div>
<br />
<br />
<h2>
<span style="font-size: x-large;">Second Quartile </span></h2>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTOTMaD4wh_TiYqqtUHW5AiJEe8FlicB5O8kvazINYKfv95HgHpTcaJUY_X7Gy6qPEyaz5QBFKoiFeOXbRhK8FlmjLAe7qLn-JewLTwKuUnsioJhGynd1JMGXfd0rFLniM1UTH8gd2Bv4K/s1600/quartile_25.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTOTMaD4wh_TiYqqtUHW5AiJEe8FlicB5O8kvazINYKfv95HgHpTcaJUY_X7Gy6qPEyaz5QBFKoiFeOXbRhK8FlmjLAe7qLn-JewLTwKuUnsioJhGynd1JMGXfd0rFLniM1UTH8gd2Bv4K/s640/quartile_25.png" width="640" /></a></div>
<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">First Quartile </span></h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2fVBQxMkMjKEZ-GzSB5mHnw5HShpb3c8sYnnE4FDSw6UoffXFG6_r2xFN7Wn7ozVuzYLr2lxzkmMSBUt2Q3iXAN30JmUtFsFVK-3IH8nMhx1vSdpuiMioMGE3ALpkpGG8EpR1qNS_2br8/s1600/quartile_0.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2fVBQxMkMjKEZ-GzSB5mHnw5HShpb3c8sYnnE4FDSw6UoffXFG6_r2xFN7Wn7ozVuzYLr2lxzkmMSBUt2Q3iXAN30JmUtFsFVK-3IH8nMhx1vSdpuiMioMGE3ALpkpGG8EpR1qNS_2br8/s640/quartile_0.png" width="640" /></a></div>
<h2>
<span style="font-size: x-large;">Distribution of Minimum Wages</span></h2>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlgJ_pzboGnKqGwefJjx3nbjd7pGfflN9y_cpYR0zHlaHcrZHweUJrANdlq6ukeX0dLK7p0BQdDwNi-oyqNi-tiFvZvCPF23KVe7CT2s2drVtZNZf71NAyRTC7895Cz7WxuOuFzKkByEJj/s1600/distr.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlgJ_pzboGnKqGwefJjx3nbjd7pGfflN9y_cpYR0zHlaHcrZHweUJrANdlq6ukeX0dLK7p0BQdDwNi-oyqNi-tiFvZvCPF23KVe7CT2s2drVtZNZf71NAyRTC7895Cz7WxuOuFzKkByEJj/s640/distr.png" width="640" /></a></div>
<br />
Below is a some R code for grabbing the table of country-specific minimum wages from Wikipedia.<br />
<div>
<script src="https://gist.github.com/johnjosephhorton/5621448.js"></script>
</div>
<div>
<br />
<div>
<div>
<br /></div>
<div>
<br /></div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com3tag:blogger.com,1999:blog-1597186260874280667.post-91391075096269698252012-07-31T21:58:00.003-07:002012-08-03T22:15:22.991-07:00The Indian blackouts & oDeskA <a href="http://www.washingtonpost.com/world/india-blackout-on-second-day-leaves-600-million-without-power/2012/07/30/gJQA7j1LMX_story.html" target="_blank">nationwide blackout</a> in India has left some 600 million people without electricity. Given that a large number of the contractors on oDesk are from India, I assumed that effects of the blackout would show up readily in the oDesk data. This evening, I wrote a query to get the hours worked each day by Indian contractors during the last month and the number of applications sent. I divided these counts by the respective totals for that day for all of oDesk. From this time series, we can get a sense of what was <i>supposed</i> to happen today and compare it to what actually happened. The time series for applications (top) and hours worked (bottom) are plotted below [1], with today annotated in red. Each percentage estimate has a 95% confidence interval. <br />
<div>
<br /></div>
<div>
<b>Some observations</b></div>
<div>
<ul>
<li>There is a very easy to detect drop-off in the hours worked---my eyeball calculation says they should have been responsible for around 22% of the hours worked today, while the actual number is closer 17.5%. This is far less of a fall-off than we would naively predict from the "1/2 of Indians without power" headline. Presumably many contractors have access to private generators, or perhaps oDesk is over-represented in parts of the country that were less affected by the blackout. </li>
<li>There is no corresponding obvious drop-off in the fraction of applications. I don't have a good explanation for this, but perhaps non-affected Indian contractors have made up the difference and exploited the now-thinner market. If I can get some data on what parts of the country are actually being affected by the blackout, I could test this notion since I do have contractor locations down to the city level. </li>
<li>Indian contractors take weekends off, both in terms of working and job finding (or at least more so than their oDesk counter-parts from other countries). Remember that this time series is the fraction for a given day, so there's no reason for a strong weekend/weekday pattern. See <a href="https://www.odesk.com/blog/2012/07/visualizations-of-odesk-oconomy/" target="_blank">oDesk Country Explorer</a> for more of this kind of data. </li>
<li>Indian contractors are generally over-represented in the application pool, making up ~25% of applications but only about ~20% of hours worked, though this could easily reflect differences in the kinds of categories Indian contractors work in---there is a great deal of variance in the average number of applications per opening across the different job categories. </li>
</ul>
</div>
<div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAJr9BJXCvHn4cbfpRlfmubVcho8fhRL4YWeqxtKaac1wG6WxFK601p4bCqIe7Rgru94RW6WRvdqYxFxLftDNQG-7OyZOvkZxQN3qmZIZnkqXvmWbcuU0h9ExQQzJsEiidJqgK15nQ4S0D/s1600/india_blackout.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="512" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAJr9BJXCvHn4cbfpRlfmubVcho8fhRL4YWeqxtKaac1wG6WxFK601p4bCqIe7Rgru94RW6WRvdqYxFxLftDNQG-7OyZOvkZxQN3qmZIZnkqXvmWbcuU0h9ExQQzJsEiidJqgK15nQ4S0D/s640/india_blackout.png" width="640" /></a></div>
<div>
</div>
</div>
</div>
<div>
Code for the plots (done in ggplot2):</div>
<div>
<script src="https://gist.github.com/3223763.js?file=india_blackout.R">
</script>
</div>
<div>
<br /></div>
<div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com5tag:blogger.com,1999:blog-1597186260874280667.post-90025154957183547512012-07-25T21:35:00.001-07:002012-07-25T22:05:55.985-07:00Digitization of the supply side of the labor market<span style="background-color: yellow;">Note: This blog post also contains a short review of Google's new Consumer Surveys service. See the end of the blog post for details. </span><br />
<br />
On most electronic commerce sites, information about the supply side is digitized and publicly available while information about the demand side is generally not: <a href="http://www.amazon.com/" target="_blank">Amazon</a>, <a href="http://www.expedia.com/" target="_blank">Expedia</a>, <a href="http://www.apple.com/itunes/" target="_blank">iTunes</a>, <a href="http://www.etsy.com/" target="_blank">Etsy</a> etc., all collect and display detailed data about the items for sale, but there is generally little or no information about the consumers with the demands. <span style="background-color: white;">If we look at the labor market, the reverse us true, in that it is the demand side that's digitized. </span><span style="background-color: white;">On online job boards like <a href="http://www.careerbuilder.com/" target="_blank">CareerBuilder</a>, <a href="http://monster.com/">Monster.com</a>, <a href="http://www.indeed.com/" target="_blank">Indeed</a>, <a href="http://www.simplyhired.com/" target="_blank">SimplyHired</a> etc., vacancies are described via detailed textual descriptions about the nature of the work, skills required, location and approximate salary, but the job seekers---the sellers---generally do not create profiles that describe themselves to the marketplace. </span><br />
<br />
<br />
While we might think that there are some fundamental reason for this difference, I don't think this is the case for the simple reason that in the case of labor markets, the supply side <b>is</b> being digitized, primarily though LinkedIn (in a big way) and through sites like <a href="http://www.odesk.com/" target="_blank">oDesk</a> (in a comparatively smaller, but more comprehensive way). On these sites, workers create permanent, searchable profiles for employers that containe rich, employment-relevant data about themselves.<br />
<div>
<br /></div>
<div>
<div>
<span style="background-color: white;">With the rise of LinkedIn, we are witnessing an </span><span style="background-color: white;">unprecedented, voluntary data collection and digitization of the </span><span style="background-color: white;">supply side of the labor market. On LinkedIn, individuals can create </span><span style="background-color: white;">public profiles and list their education, professional credentials, </span><span style="background-color: white;">associations, skills, current and past work experiences and, </span><span style="background-color: white;">critically, their other professional connections (indicated by </span><span style="background-color: white;">approved links to other LinkedIn users). As of yesterday (July 24th, 2012), </span><span style="background-color: white;">approximately 19% of the US-based Internet using population had a </span><span style="background-color: white;">LinkedIn </span><span style="background-color: white;">profile [* see note below for interesting background for this 19% figure]. </span><span style="background-color: white;"><a href="http://press.linkedin.com/about" target="_blank">According to LinkedIn</a>, as of March 12, 2012, over 160 million people </span><span style="background-color: white;">have created profiles, </span><span style="background-color: white;">and in </span><span style="background-color: white;">many industries, a LinkedIn profile is expected of all </span><span style="background-color: white;">applicants. I talked recently to oDesk's corporate recruiter, asking her how many candidates had LinkedIn profiles. She responded: </span></div>
</div>
<div>
<span style="background-color: white;"><br /></span></div>
<blockquote class="tr_bq">
<span style="font-family: arial, sans-serif; font-size: 12.499999046325684px;">I'd say it is close to 100% (and certainly 100% for viable candidates). I can't think of an example of someone who I have screened who didn't have a profile on LinkedIn. </span></blockquote>
<div>
<br /></div>
<div>
<div>
I think this supply digitization is likely to prove consequential, because once the supply side of the labor market is digitized, platforms can begin making data-driven, highly contextualized recommendations to both sides of the market. <span style="background-color: white;">The recommendations made by a platform can have the advantage of being </span><span style="background-color: white;">potentially informed by the platform's holistic perspective on the </span><span style="background-color: white;">marketplace. In computer-mediated marketplaces, by necessity </span><span style="background-color: white;">essentially every piece of data that goes into or is generated by the </span><span style="background-color: white;">marketplace is captured in an electronic database that could </span><span style="background-color: white;">conceivably used to make recommendations. </span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<span style="background-color: white;">Of course, job board do try to make recommendations by suggesting vacancies to workers, </span><span style="background-color: white;">but they are limited to conditioning </span><span style="background-color: white;">those recommendations on whatever search terms and perhaps geographic </span><span style="background-color: white;">and/or salary constraints a job-seeker enters in a relatively brief </span><span style="background-color: white;">search session. The platform cannot condition its recommendations on a </span><span style="background-color: white;">worker's employment history, educational background, skills, current </span><span style="background-color: white;">employment status, professional connections, certifications, </span><span style="background-color: white;">personality, test scores and other match-relevant factors, nevermind try to balance recommendations to navigate the twin shoals of market thinness and market congestion. </span></div>
</div>
<div>
<div>
<br /></div>
<div>
Unfortunately, I think a lot of this work on recommendations will happen within companies in a state of semi-secrecy, but hopefully enough will be made public that others can contribute, ala the Netflix challenge. It's a little sad that to date <span style="background-color: white;">society has expended more machine learning research effort trying to predict taste in moves rather than fit for jobs, despite the enormous welfare consequences of the labor market. However, I predict this will change and expect a lot more work on this topic from computer scientists and market designers in the coming years. </span></div>
<div>
<br /></div>
</div>
<h3>
[*] The Origin of "19% of the US Population has a LinkedIn profile" Number</h3>
<div>
In writing this blog post, I wanted to get an accurate number for what fraction of the US population has a LinkedIn profile. This number was proving hard to come by, so I decided to try a relatively new service launched by Google called <a href="http://www.google.com/insights/consumersurveys/home" target="_blank">Google Consumer Surveys</a>. For 10 cents an answer, you can pose questions to a supposedly representative sample of US-based Internet users. You also get some of the respondent's basic demographics, such as inferred age, gender and income. I launched a one question survey and got 1511 responses in less than a day. The screenshot below shows the main results, but it also includes some neat tools for looking at the data in different ways. I made the survey public---check it out <a href="https://www.google.com/insights/consumersurveys/view?survey=7gvhbjkcf6lxm&question=1&filter=&rw=1" target="_blank">here</a>. I'm quite pleased with the service and plan to use it again. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi92VYNXt9chDrx5pVtIy9HtIx84WQtnfvOamAI_VRs2FPPvn8Dh3xJN2ICdEY1iXRB4j0UFLxP-B_weQdjBQejQrb0rjHxDlXPvxh7wFR3GK2x6brzlMTFhupTBrifyoYHlAS4faBWsu48/s1600/linked_in_penetration.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="283" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi92VYNXt9chDrx5pVtIy9HtIx84WQtnfvOamAI_VRs2FPPvn8Dh3xJN2ICdEY1iXRB4j0UFLxP-B_weQdjBQejQrb0rjHxDlXPvxh7wFR3GK2x6brzlMTFhupTBrifyoYHlAS4faBWsu48/s640/linked_in_penetration.png" width="640" /></a></div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com1tag:blogger.com,1999:blog-1597186260874280667.post-15508497183378917512012-07-05T09:33:00.003-07:002012-07-05T09:36:12.064-07:00Shrimponomics, Complements & BPOs<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://www.ixbt.com/short/2k5-03/ShrimpDisk.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://www.ixbt.com/short/2k5-03/ShrimpDisk.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Most relevant image available from doing a <br />
Google Image search for "Shrimp using a computer"</td></tr>
</tbody></table>
<br />
A few years ago, there was a <a href="http://www.freakonomics.com/2007/08/30/shrimponomics/" target="_blank">Freakonomics post</a> about how people reason about economic situations and phenomena. The phenomenon in question was shrimp consumption: t<span style="background-color: white;">he amount of shrimp people eat in the US <i>per capita</i> tripled between 1982 and 2007. </span><span style="background-color: white;">When asked to explain this rise, non-economists mainly give demand reasons (changes in preferences), while economists are more likely to also give supply reasons (improved fishing efficiency, rise of aquaculture etc.). </span><br />
<div>
<br /></div>
<div>
If I had to offer an explanation for this focus on demand explanations, my guess it that demand explanations come more easily to us because it is the side of the market that is more familiar to us : most of us have eaten shrimp & bought shrimp---very few of us have worked in commercial fishing. So when asked "why are people consuming more shrimp?" we start with "why might <i>I</i> consume more shrimp?" and although price is certainly a reason (and a path of thought that would help lead to a demand explanation), it's not as salient or even as interesting as things like changing tastes, health trends, exciting new shrimp-based dishes etc. </div>
<div>
<br /></div>
<div>
So this blog post isn't about shrimp and it isn't about supply & demand. It's about complements and substitutes. I think there is a similar psychological tendency to focus on goods-as-substitutes than goods-as-complements<span style="background-color: white;">. </span><span style="background-color: white;">At the individual level where we are making choices, we are usually thinking in terms of substitutes: do I want coffee or tea? Should I take a vacation to Las Vegas or Hawaii? Mac or PC? It's a bit more subtle to think about "if I had X, would it make Y more useful to me" which is at the heart of all complementarity stories. </span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
This is a long-winded introduction to my real topic, which is that in my <a href="http://onlinelabor.blogspot.com/2012/06/will-online-labor-markets-disrupt.html" target="_blank">last blog post</a>, I made the argument that online work could disrupt the BPO industry by serving as a substitute for what BPOs offer. A point I didn't think of---but in retrospect seems pretty obvious---is how just as easily complementarity could be the dominate effect. After my blog post, my CTO at oDesk, <span style="background-color: white;"><a href="https://plus.google.com/109694920571648066264/posts" target="_blank">Odysseas</a>, emailed me with his thoughts</span><span style="background-color: white;">: </span></div>
<br />
<blockquote class="tr_bq">
<div style="font-family: arial, sans-serif; font-size: 13px;">
The primary benefit of BPOs is not that of labor cost arbitrage. Thats typically the motive/benefit for offshore staff augmentation firms - but BPOs are business process outsourcers. BPO is ADP [Automated Data Processing] that outsources your payroll or a business that outsource your HR process etc... We often tend to think of BPOs as an offshore firm that does a little bit of everything having as sole pivot point its lower cost of labor - thats true, but its an abuse of the term and I would agree there that the particular type of business is going to be affected in the years to come from online labor.</div>
</blockquote>
This part is basically my substitutes story---now the complements part: <br />
<blockquote class="tr_bq">
<div style="font-family: arial, sans-serif; font-size: 13px;">
However, the more interesting effect would be the effect of online labor to the real BPOs..</div>
<div style="font-family: arial, sans-serif; font-size: 13px;">
There BPOs will not be negatively affected - the opposite. The availability of online labor would allow BPOs to <span style="background-color: white;">become more flexible lower their overall fixed costs force them to become more automated and streamline (their virtual nature will require that), allowing them to lower even the cost per customer, allowing them to focus on smaller projects, smaller customers allowing to address smaller/different market segments. They will become less relying on an enterprise sales force customer acquisition model which is dramatically affecting their cost structure.</span></div>
<div style="font-family: arial, sans-serif; font-size: 13px;">
We are seing examples of what the new BPOs will become in companies that outsource the process of testing (uTest) of seo writing (Mediapiston) etc.</div>
</blockquote>
He's of course exactly right---and he's a CS PhD, not an economist, so shame on me :). If you think of true BPOs in the sense that Odysseas is talking about, then the complementarity story becomes more important. These true BPOs would be big buyers in the inputs market and would benefit greatly from a liquid, efficient market for labor. <br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com1tag:blogger.com,1999:blog-1597186260874280667.post-85152602627322226442012-06-26T21:02:00.002-07:002012-06-27T07:36:41.964-07:00Will online labor markets disrupt the traditional BPO firm?Today I spoke on a panel on something called "impact sourcing" at the <a href="http://www.worldbpoforum.com/event_speakers.aspx" target="_blank">BPO World Forum</a>. The idea of impact sourcing, in a nutshell, is that online work is a tool for development and that for-profit firms outsourcing some part of their business should look beyond traditional <a href="http://en.wikipedia.org/wiki/Business_process_outsourcing" target="_blank">BPO firms</a> and consider non-profits like <a href="http://samasource.org/" target="_blank">Samasource</a> and <a href="http://www.digitaldividedata.org/" target="_blank">Digital Divide Data</a>. It was a good audience for this pitch, as many of attendees were CIOs from big companies that are accustomed to signing multi-million dollar IT outsourcing deals with the likes of traditional BPO firms like Wipro, Infosys, Tata Consultancy etc. <br />
<div>
<div>
<br /></div>
<div>
<div>
After the panel, I was at a reception where I talked to someone fairly high up in a traditional BPO. <span style="background-color: white;">When I described my elevator pitch version of oDesk's business---clients post jobs, contractors make bids, clients make a hire, we intermediate the work and take a percentage---he said, literally "what are you doing here at this conference? You guys are like the Antichrist." What he meant (in a half joking, half serious way) is that oDesk and similar companies threaten the model of the BPO. </span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<span style="background-color: white;">My perception is that the traditional BPO model is </span><span style="background-color: white;">possible because of two facts: (a) the enormous, purely placed-based differences in wages and (b) the difficulty of actually arbitraging those differences without help. BPOs stand ready to help companies reap the benefits of (a) by giving the help necessitated by (b). The word is still very far away from (a) no longer being true, but if oDesk and similar companies can radically lower the barriers to arbitraging differences by making it easy to hire, manage and pay workers regardless of geography, then (b) starts to become less true. </span><span style="background-color: white;">If we get to the point where the qualitative differences of online remote and in-person work diminish and assessing and hiring workers is simple and easy, it would </span><span style="background-color: white;">obviate the need for much of what the BPO firm is selling</span><span style="background-color: white;">. </span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<span style="background-color: white;">This is not to say that there isn't still a huge space for IT consulting---outsourcing an entire process is hard and BPOs with lots of experience have something very valuable to offer. Furthermore, besides purely cost level, one of the motivations for business process outsourcing is ability to change cost structure, namely by turning a fixed cost into a variable cost. But these caveats aside, on the margin, the mediation aspect of the BPO role</span><span style="background-color: white;"> seems likely to get less attractive over time as technology improves and online labor markets mature. </span></div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-72751015755438411842012-06-06T21:16:00.001-07:002012-06-07T08:26:18.437-07:00Resources for online social scienceThe Economist recently had <a href="http://www.economist.com/node/21555876" target="_blank">an article</a> about the growing use of online labor markets as subject pools in psychology research; ReadWriteWeb wrote a <a href="http://www.readwriteweb.com/archives/crowd-sourced-studies-offer-advantages-challenges-for-researchers.php" target="_blank">follow-up</a>. If you've been following this topic, there wasn't very much new, but if you're a researcher that would like to use these methods, the articles were pretty light on useful links. This blog post is an attempt to point out some of the resources/papers available. This is my own very biased, probably idiosyncratic view of the resources, so hopefully people will send me corrections/additions and I can update this post.<br />
<br />
<div>
To start, let's have this medium pay tribute to itself by running through some blogs and their creators. </div>
<h3>
Blogs </h3>
<ul>
<li>There is the "<a href="http://crowdresearch.org/blog/" target="_blank">Follow the Crowd</a>" blog which I believe is associated with HCOMP conference. It's definitely more CS than Social Science, but I think it's filled with good examples of high-quality research done w/ MTurk and with other markets. </li>
<li>There's <a href="http://venus.unive.it/paolacci/" target="_blank">Gabriel Paolacci</a>'s (now at Erasmus University) "<a href="http://experimentalturk.wordpress.com/" target="_blank">Experimental Turk</a>" blog which was mentioned in the article and is probably the best resource for examples of social and psychological science research being done with MTurk. </li>
<li><a href="http://people.stern.nyu.edu/panos/" target="_blank">Panos Ipeirotis </a>(at NYU and who is now academic-in-residence at oDesk) has a great blog "<a href="http://www.behind-the-enemy-lines.com/" target="_blank">Behind-enemy-lines</a>" that's basically all things relating to online work </li>
<li>The defunct "<a href="http://groups.csail.mit.edu/uid/deneme/" target="_blank">Deneme blog</a>" by <a href="http://glittle.org/" target="_blank">Greg Little</a> (who also works at oDesk) and <a href="http://www.hmslydia.com/" target="_blank">Lydia Chilton</a> (at University of Washington). </li>
</ul>
<h3>
Guides / How-To (Academic Papers)</h3>
A number of researchers have written guides to using MTurk for research. I think the first stop for social scientists should be the paper by <a href="http://jessechandler.com/" target="_blank">Jesse Chandler</a>, <a href="http://venus.unive.it/paolacci/" target="_blank">Gabriel Paolacci</a> and <a href="http://people.stern.nyu.edu/panos/" target="_blank">Panos Ipeirotis</a>:<br />
<br />
Chandler, J. Paolacci, G. and Iperiotis, I. Running Experiments on Mechanical Turk,<br />
<i>Judgement and Decision Making</i> (<a href="http://journal.sjdm.org/10/10630a/jdm10630a.html" target="_blank">paper</a>) (<a href="http://article%7Bpaolacci2010running%2C%20%20%20title%3D%7Brunning%20experiments%20on%20amazon%20mechanical%20turk%7D%2C%20%20%20author%3D%7Bpaolacci%2C%20g.%20and%20chandler%2C%20j.%20and%20ipeirotis%2C%20p.g.%7D%2C%20%20%20journal%3D%7Bjudgment%20and%20decision%20making%7D%2C%20%20%20volume%3D%7B5%7D%2C%20%20%20number%3D%7B5%7D%2C%20%20%20pages%3D%7B411--419%7D%2C%20%20%20year%3D%7B2010%7D%2C%20%20%20publisher%3D%7Bsociety%20for%20judgment%20and%20decision%20making%7D%20%7D/" target="_blank">bibtex</a>)<br />
<br />
My own contribution is a paper with <a href="http://www.people.fas.harvard.edu/~drand/" target="_blank">Dave Rand</a> (who will still be starting as new assistant professor at Yale) and <a href="http://www.hks.harvard.edu/fs/rzeckhau/" target="_blank">Richard Zeckhauser</a> (at Harvard). The paper paper contains a few replication studies, but the real meat and the part I think is most important is the part discussing precisely why/how you can do valid causal inference online (I'm stealing this write-up/links of the paper from Dave's website):<br />
<br />
<span style="background-color: white; text-align: -webkit-left;">Horton JJ, </span><span style="background-color: white; text-align: -webkit-left;">Rand DG</span><b style="background-color: white; text-align: -webkit-left;">,</b><span style="background-color: white; text-align: -webkit-left;"> Zeckhauser RJ. (2011) The Online Laboratory: Conducting Experiments in a Real Labor Market. </span><i style="background-color: white; text-align: -webkit-left;">Experimental Economics</i><span style="background-color: white; text-align: -webkit-left;">. </span><b style="background-color: white; text-align: -webkit-left;">14</b><span style="background-color: white; text-align: -webkit-left;"> 399-425. </span><b style="background-color: white; text-align: -webkit-left;"><a href="http://www.people.fas.harvard.edu/~drand/horton_rand_zeckhauser_2011.pdf">(PDF)</a> (</b><a href="http://scholar.google.com/scholar.bib?q=info:WvyFC3n6_N4J:scholar.google.com/&output=citation&hl=en&as_sdt=0,5&ct=citation&cd=0" target="_blank">bibtex</a>)<br />
<br />
<i style="background-color: white; text-align: -webkit-left;">Press: <a href="http://marketplace.publicradio.org/display/web/2010/12/27/tech-report-are-you-part-of-an-online-psychology-experiment/" target="_blank">NPR's Morning Edition Marketplace [audio]</a>, <a href="http://www.theatlantic.com/technology/archive/2010/11/how-amazon-speeds-up-economics-experiments/65590/" target="_blank">The Atlantic</a>, <a href="http://cyber.law.harvard.edu/events/luncheon/2010/11/rand" target="_blank">Berkman Luncheon Series [video]</a>, <a href="http://www.nationalaffairs.com/blog/detail/findings-a-daily-roundup/a-working-class-of-their-own" target="_blank">National Affairs</a>, <a href="http://blog.crowdflower.com/2010/01/altruism-on-amazon-mechanical-turk/" target="_blank">Crowdflower</a></i><span style="background-color: white; text-align: -webkit-left;">, </span><a href="http://www.marginalrevolution.com/marginalrevolution/2010/05/assorted-links-8.html" style="background-color: white; text-align: -webkit-left;"><i>Marginal Revolution</i></a><span style="background-color: white; text-align: -webkit-left;">,</span><a href="http://experimentalturk.wordpress.com/2010/02/11/altruism/" style="background-color: white; text-align: -webkit-left;" target="_blank"> <i>Experimental Turk</i></a><span style="background-color: white; text-align: -webkit-left;">, </span><i style="background-color: white; text-align: -webkit-left;"><a href="http://www.ethanzuckerman.com/blog/2010/11/02/david-rand-social-science-experiments-with-mechanical-turk/" target="_blank">My Heart's in Accra</a></i><span style="background-color: white; text-align: -webkit-left;">, </span><i style="background-color: white; text-align: -webkit-left;"><a href="http://www.hyperorg.com/blogger/2010/11/02/berkman-dave-rand-on-social-science-on-the-net/" target="_blank">Joho </a></i><a href="http://www.hyperorg.com/blogger/2010/11/02/berkman-dave-rand-on-social-science-on-the-net/" style="background-color: white; text-align: -webkit-left;" target="_blank"><i>blog</i></a><i style="background-color: white; text-align: -webkit-left;">, <a href="http://veratevelde.blogspot.com/2011/01/online-labor-markets.html" target="_blank">Veracities blog</a></i><br />
<br />
<div>
<h3>
Software </h3>
<div>
Unfortunately there hasn't been too much sharing of software for doing online experiments. Since a lot of the experimentation is done by computer scientists who do not feel daunted by making their own one-off, ad hoc applications, there are a lot of one-off, ad hoc applications. Hopefully people know of other tools that are out there that the can open source / they can share links to. <br />
<br />
"<a href="https://github.com/johnjosephhorton/randomizer" target="_blank">Randomizer</a>"<br />
<br />
Basically, it lets you provide subjects one link that will automatically redirect them (at random) to a collection of URLs you've specified.I made the first really crummy version of this and then got a real developer to re-do it so it runs on Google App Engine.<br />
<br />
<br />
<br />
"<a href="https://github.com/johnjosephhorton/quicklime" target="_blank">QuickLime</a>"<br />
This is a tool for quickly setting up an Limesurvey (an open source alternative to Qualtrics & Surveymonkey) on a new EC2 machine. This was made courtesy of oDesk research. I haven't fully tested it yet, so as with all this software, caveat oeconomus.<br />
<br />
<br />
"<a href="http://developers.odesk.com/w/page/12364003/FrontPage" target="_blank">oDesk APIs</a>"<br />
There haven't been lot of experiments done on <a href="http://www.odesk.com/" target="_blank">oDesk</a> by social scientists, but there's no reason it cannot be done. While it currently is not as convenient or as low-cost as doing experiments on MTurk, I think long-term oDesk workers would make a better subject pool since you can more carefully control experiments, it's easier to get everyone online at the same time to participate in an experiment, there are no spammers etc. If you're looking for some ideas or pointers, feel free to email me.<br />
<br />
"<a href="https://github.com/boto/boto" target="_blank">Boto</a>"<br />
This is a python toolkit for working with Amazon Web Services (AWS). It's fantastic and saved me a lot of time when I was doing lots of MTurk experiments. <br />
<br />
"<a href="http://groups.csail.mit.edu/uid/projects/seaweed/ChiltonThesis-Seaweed-19May2009.pdf" target="_blank">Seaweed</a>"<br />
This was Lydia Chilton's masters thesis. The idea was to create tools for conducing economics experiments online. I don't think it ever moved beyond the beta stage, but if you (a) have some grant money and (b) are thinking about porting z-tree to the web, you should email Lydia and see where the codebase is & if anyone is working on it.<br />
<br />
Here's a little <a href="https://gist.github.com/676230" target="_blank">javascript snippet</a> I wrote for doing randomization within the page of an MTurk task. </div>
</div>
<h3>
</h3>
<h3>
People </h3>
I'm not doing to try to do a who-is-who of Crowdsourcing, but if you're looking for some contacts of other people (particularly those in CS) who are doing work in this field, you can check out the list of recent participants at "<a href="http://crowdresearch.org/crowdcamp/" target="_blank">CrowdCamp</a>" which was a workshop prior to HCI.<br />
<br />
<h3>
History</h3>
Probably the <a href="http://www-users.cs.umn.edu/~echi/papers/2008-CHI2008/2008-02-mech-turk-online-experiments-chi1049-kittur.pdf" target="_blank">first paper </a>I'm aware of that pointed out that experiments (i.e., user studies) were possible on MTurk was by <a href="http://www-users.cs.umn.edu/~echi/" target="_blank">Ed Chi</a>, <a href="http://kittur.org/" target="_blank">Niki Kittur</a> and <a href="http://www2.parc.com/istl/groups/uir/people/bongwon/bongwon.htm" target="_blank">Bongwon Suh</a>. As far as I know, the first social science done on MTurk was <a href="http://everythingisobvious.com/" target="_blank">Duncan Watts</a> and <a href="http://www.stevens.edu/news/content/winter-mason" target="_blank">Winter Mason</a>'s paper on <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.173.8632" target="_blank">financial incentives</a> and the performance of crowds.<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com0tag:blogger.com,1999:blog-1597186260874280667.post-68416040293783971982012-06-01T23:35:00.001-07:002012-06-01T23:35:48.691-07:00The Innovation of StackOverflowSo as I write this, there is an egg timer ticking away next to me, set with 10 minutes of time. What am I waiting for? 10 minutes is how much time I predicted it would take to get my programming question answered on StackOverflow (SO):<br />
<br />
<br />
<a href="http://stackoverflow.com/questions/10860020/output-a-vector-in-r-in-the-same-format-used-for-inputting-it-into-r">http://stackoverflow.com/questions/10860020/output-a-vector-in-r-in-the-same-format-used-for-inputting-it-into-r</a><br />
<br />
The back story was that I was writing some R code and I got to a point where I was stuck: there was something I wanted to do and I remembered that there was a built-in function that could accomplish my goal. Unfortunately, I couldn't remember that function's name. After some fruitless googling, I posted the question on SO.<br />
<div>
<br /></div>
<div>
So, how long did it actually take to get the right answer? About 6 1/2 minutes. As I write this sentence, I'm waiting for some more time to elapse so I can actually approve the answer: </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgycrCERp4FpqLPXQYSB-bV4gJ6ZXujZKZapZ9oruJK3Zxr3F6wvZxuRhFt309G7mOiRvtnkufX22nxZ_1TmeVq6Uc5QiR1T9XBZg69a2KD5_uKJwAY6N1nrXu9r0K8z3UF5gTQax9ZqIuE/s1600/stackoverflow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="329" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgycrCERp4FpqLPXQYSB-bV4gJ6ZXujZKZapZ9oruJK3Zxr3F6wvZxuRhFt309G7mOiRvtnkufX22nxZ_1TmeVq6Uc5QiR1T9XBZg69a2KD5_uKJwAY6N1nrXu9r0K8z3UF5gTQax9ZqIuE/s640/stackoverflow.png" width="640" /></a></div>
<div>
</div>
<div>
This has been my general experience with SO---amazingly high-quality answers delivered almost immediately. I feel sheepish that I haven't been able to answer as many questions as I've asked, but one of the animating ideas of the community is that asking high-quality, answerable questions is a way of contributing. </div>
<div>
<br /></div>
<div>
What's interesting to me is that SO is an example of a primarily social---as opposed to technological---innovation. There's nothing really technically innovative about SO: the site is fast, search works well, tagging works well etc., but lots of sites have those things. What's special about SO is that through a carefully designed system of incentives and policies, they have created a community that is literally---and I think profoundly---changing how people program computers. </div>
<div>
</div>
<div>
The reason I point about the social nature of the innovation is that it's become <a href="http://www.theatlantic.com/business/archive/2012/05/the-golden-age-of-silicon-valley-is-over-and-were-dancing-on-its-grave/257401/" target="_blank">popular to lament</a> the shallowness or perceived frivolity of many start-ups that are built around social rather than technological innovations (e.g., Facebook, Twitter, Instagram etc.). The idea seems to be that if you aren't making solar panels or cancer-curing drugs, you're not doing something socially useful. I personally don't share that bias, but if we are going to judge companies on the basis of some more "serious" metric like productivity or social surplus, then SO is a great example how a purely social innovation can succeed spectacularly on those metrics. </div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com6tag:blogger.com,1999:blog-1597186260874280667.post-63181722272944030062012-05-22T10:40:00.000-07:002012-05-22T13:26:21.968-07:00Data openness by private firmsThe New York Times has a story today about social scientists working with <a href="http://www.nytimes.com/2012/05/22/science/big-data-troves-stay-forbidden-to-social-scientists.html?partner=rss&emc=rss" target="_blank">company data</a> and being unable or unwilling to make it public. The story begins:<br />
<blockquote class="tr_bq">
<span style="background-color: white; font-family: georgia, 'times new roman', times, serif; font-size: 15px; line-height: 22px; text-align: left;">When scientists publish their research, they also make the underlying data available so the results can be verified by other scientists.</span></blockquote>
I think the first sentence is probably more a description of how we'd <i>like</i> the world to be than how it actually is right now, especially in the social sciences. The main so-what of the story is that private companies are collecting enormous amounts of high quality data that lets you do fascinating social science, but companies are understandably reluctant to make this data public, primarily for privacy reasons (and probably also because they are afraid of giving up some competitive advantage).<br />
<br />
I think the options for any organization that does or might do research are:<br />
<br />
1) Do research for business purposes. Make neither the findings nor the data public.<br />
2) Do research for business purposes. Make the findings but not the full data public.<br />
3) Do research for business purposes. Make the findings and data public. <br />
4) Do research. Make findings and data public.<br />
<br />
<div>
<br /></div>
<div>
Most companies probably aren't interested in (4) and this is probably academia's biggest comparative advantage. Barring (4), I think from a social perspective, privacy issues aside, the best outcomes in order are (3) > (2) > (1). I can understand (1) in some cases, but at least in the kind of companies I'm familiar with, the advantages of keeping everything secret probably aren't that great. </div>
<div>
<br /></div>
<div>
The advantages of (2) or (3) over (1): </div>
<div>
<br /></div>
<div>
a) If you're a software company and you release a feature that works, it will probably get copied anyway, regardless of whether you publish a paper, so you might as well get the thought leadership credit for coming up with the idea in the first place. This <a href="http://ilpubs.stanford.edu:8090/422/" target="_blank">paper</a> is/was the basis for Google's secret sauce---posting it to the InfoLab servers back in 1999 didn't doom the company and probably did a lot to increase the perceptions that they were doing something smarter (even though there were antecedents of this idea going back many years---including in Economics, by my academic <a href="http://en.wikipedia.org/wiki/Wassily_Leontief" target="_blank">grandfather</a>). </div>
<div>
<br /></div>
<div>
b) If you give them access and them publish, you can get outside academics to work on your problems for free (the Netflix prize is an obvious example). You can recruit those academics to come work for you, or at least get their grad students to come work for you. </div>
<div>
<br /></div>
<div>
c) If you let your internal researchers publish, you can get them to work at reduced cost or get researchers you otherwise wouldn't be able to attract (see Scott Stern's paper on scientists <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.6981" target="_blank">"paying" to do science</a>).<br />
<br />
On (2) versus (3), I think there is a real dilemma: openness and privacy concerns are in tension. Furthermore, just releasing more aggregated or somehow obfuscated versions of the data is not risk free: there's actually an emerging literature in Computer Science on how to release data in ways that are guaranteed to still have the right privacy properties (<strike>CMU</strike> UPenn professor Aaron Roth recently taught a <a href="http://www.cis.upenn.edu/~aaroth/courses/privacyF11.html" target="_blank">course</a> on the topic). The fact that smart people are working on it is exciting, since they might figure out provably risk-free ways to release data publicly, but it's also evidence that this isn't a trivially easy problem---seemingly innocuous data disclosures would let someone unravel the obfuscation. <br />
<br />
<br /></div>
<div>
As a coda, I have a personal anecdote to share about this story. One of the people discussed in the article is Bernardo Huberman: </div>
<blockquote class="tr_bq">
<span style="background-color: white; font-family: georgia, 'times new roman', times, serif; font-size: 15px; line-height: 22px; text-align: left;">The chairman of the conference panel — Bernardo A. Huberman, a physicist who directs the social computing group at HP Labs here — responded angrily. In the future, he said, the conference should not accept papers from authors who did not make their data public. He was greeted by applause from the audience.</span> </blockquote>
<div>
When I was a grad student, I taught a course to Harvard sophomore economics majors called "Online Labor" (<a href="http://dl.dropbox.com/u/420874/Syllabus%20for%20EC%20970.pdf" target="_blank">syllabus</a>). I assigned some of Huberman's papers on motivation. I emailed him to ask for the data from one of his papers. He wrote back: </div>
<blockquote class="tr_bq">
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Dear Dr. Horton:</span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Thank you for your interest in my work and I certainly feel pleased when I learn that you liked my paper enough to assign it to your class.</span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">As to your request, let me talk with the person who now handles the youtube data (we lately used it to uncover the persistence paradox) and I'll get back to you.</span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Incidentally if you are interested in the role that attention and status (its marker) play among people I could send you a paper that reports on a experiment (as opposed to observational data) that elucidates it quite cleanly across cultures.</span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Best,</span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Bernardo</span></blockquote>
<br />
I got the data within days---I can state that he privately practices what he preaches publicly.<br />
<br />
<b>Update:</b> I incorrectly stated that <a href="http://www.cis.upenn.edu/~aaroth/" target="_blank">Aaron Roth</a> was a professor at CMU---he did his PhD at <i>CMU</i>. He's a professor at <i>UPenn</i>. Apologies. <br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-73852264771110251582012-03-06T13:38:00.001-08:002012-03-06T13:40:33.432-08:00Location of India-Based Contractors on oDeskMy favorite R package, <a href="http://had.co.nz/ggplot2/" target="_blank">ggplot2</a>, recently introduced enhanced support for <a href="http://en.wikipedia.org/wiki/Choropleth_map" target="_blank">choropleth maps</a>. I'd like to make some of these kinds of maps with oDesk data, but as a first step, I thought I'd just plot the locations of all of our India-based contractors by city. In the plot below, dots size as log-scaled by # of contractors reporting that city. The massive light blue dot near Delhi is default coordinate when we're missing the city.<br />
<br />
For those of you who know India, anything surprising/interesting here? <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0VRobZcLP0K7XCru-wH2kaZihLgGeQH0gd6GFupcHGena5j0QB0RMfeJNYGXfwRJNJRJgdhWRmOzffRGw9dykycVVe5Z5okCjDAyh4sqfDRXUZKncSRfzSgh5q1ZrYEJnMljnHykk3bLY/s1600/india.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0VRobZcLP0K7XCru-wH2kaZihLgGeQH0gd6GFupcHGena5j0QB0RMfeJNYGXfwRJNJRJgdhWRmOzffRGw9dykycVVe5Z5okCjDAyh4sqfDRXUZKncSRfzSgh5q1ZrYEJnMljnHykk3bLY/s320/india.png" width="320" /></a></div>
<br />
Here's the associated R code to make this figure:
<script src="https://gist.github.com/1989087.js?file=india.R">
</script>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com4tag:blogger.com,1999:blog-1597186260874280667.post-42171882416569834202012-02-21T09:36:00.001-08:002012-02-21T23:25:45.827-08:00Economics of the Cold Start Problem in Talent Discovery<a href="http://www.flickr.com/photos/nlscotland/4700564268/" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" title="Supply train steaming into a railhead by National Library of Scotland, on Flickr"><img alt="Supply train steaming into a railhead" height="248" src="http://farm2.staticflickr.com/1283/4700564268_6f453c0204.jpg" width="320" /></a>
<br />
Tyler Cowen <a href="http://marginalrevolution.com/marginalrevolution/2012/02/why-is-there-a-shortage-of-talent-in-it-sectors-and-the-like.html" target="_blank">recently</a> highlighted <a href="http://marginalrevolution.com/marginalrevolution/2012/02/why-is-there-a-shortage-of-talent-in-it-sectors-and-the-like.html" target="_blank">this</a> paper by <a href="http://hse-econ.fi/tervio/" target="_blank">Marko Terviö</a> as an explanation for labor shortages in certain areas of IT. The gist of the model is that in hiring novices, firms cannot fully recoup their hiring costs if the novices' true talents will become common knowledge post-hire. It's a great paper, but what people might not know is that the theory it proposes has been tested and found to perform very well. For her job market paper, <a href="http://www.economics.harvard.edu/faculty/pallais" target="_blank">Mandy Pallais</a> conducted a large experiment on <a href="http://www.odesk.com/" target="_blank">oDesk</a> where she essentially played the role of the talent-revealing firm.<br />
<br />
Here's the abstract from her <a href="http://scholar.harvard.edu/apallais/publications/inefficient-hiring-entry-level-labor-markets" target="_blank">paper</a>:<br />
<blockquote class="tr_bq">
... I formalize this intuition in a model of the labor market in which positive hiring costs and publicly observable output lead to inefficiently low novice hiring. I test the models relevance in an online labor market by hiring 952 workers at random from an applicant pool of 3,767 for a 10-hour data entry job. In this market, worker performance is publicly observable. Consistent with the models prediction, novice workers hired at random obtain significantly more employment and have higher earnings than the control group, following the initial hiring spell. A second treatment confirms that this causal effect is likely explained by information revelation rather than skills acquisition. Providing the market with more detailed information about the performance of a subset of the randomly-hired workers raised earnings of high productivity workers and decreased earnings of low-productivity workers. </blockquote>
In a nutshell, as a worker, you can't get hired unless you have feedback, and you can't get feedback unless you've been hired. This "<a href="http://en.wikipedia.org/wiki/Cold_start" target="_blank">cold start</a>" problem is one of the key challenges of online labor markets, where there are far fewer signals about a worker's ability and less common knowledge about what different signals even mean (quick: what's the MIT of Romania?). I would argue that scalable talent discovery and revelation is the most important applied problem in online labor/crowdsourcing.<br />
<br />
Although acute in online labor markets, the problem of talent discovery and revelation is no cake walk in traditional markets. Not surprisingly, several new start-ups (e.g., <a href="http://smarterer.com/" target="_blank">smarterer</a> and <a href="http://www.gild.com/" target="_blank">gild</a>) are focusing on scalable skill assessment, and there is excitement in the tech community about using talent revealing sites like <a href="http://stackoverflow.com/" target="_blank">StackOverflow</a> and <a href="https://github.com/" target="_blank">Github</a> as replacements for traditional resumes. It is not hard to imagine these low-cost tools or their future incarnations being paired with scalable tools to create human capital, like the automated training programs and courses offered by <a href="http://www.udacity.com/" target="_blank">Udacity</a>, <a href="http://www.khanacademy.org/" target="_blank">Kahn Academy</a>, <a href="http://www.codecademy.com/" target="_blank">codeacademy</a> and <a href="http://web.mit.edu/newsoffice/2011/mitx-education-initiative-1219.html" target="_blank">MITx</a>. Taken together, they could create a kind of substitute for the combined training/signaling role that traditional higher education plays today.<br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">Like what you read? </span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Why not follow me on <a href="https://twitter.com/#!/johnjhorton" target="_blank">twitter</a> or subscribe to this blog via <a href="http://onlinelabor.blogspot.com/feeds/posts/default" target="_blank">RSS</a>?</span><br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-91315249850325044792012-02-20T10:34:00.000-08:002012-02-20T10:34:48.153-08:00Solvate joins the deadpool<a href="http://techcrunch.com/2012/02/19/solvate-shutting-down/" target="_blank">Techcrunch</a> and <a href="http://www.betabeat.com/2012/02/20/freelance-marketplace-solvate-shuts-down-after-four-years-and-6-3-m/" target="_blank">Betabeat</a> are reporting that <a href="http://www.solvate.com/" target="_blank">Solvate</a>, a platform for remote work, is shutting down. Unlike oDesk, Elance, Freelancer etc., they were not trying to create a true marketplace: they were trying to do more of a high-touch, human-in-the-loop matching service. <br />
<br />
In the email Solvate sent to their users about the shutdown, they explicitly cited scalability issues, which I'm guessing refers to the non-sustainable effort and cost of hand-matching buyers and sellers. I wouldn't say this is definitive proof that the high-touch matching business model doesn't work (my outsider impression is that <a href="http://www.glgresearch.com/" target="_blank">GLG</a> is killing it), but it is a reminder that the value-added from your human-in-the-loop matching has to be sufficiency high that you can re-coup your costs: you can't take a hit on every unit sold but make it up on volume. <br />
<br />
I think it's too bad they are shutting down---I would have liked to see how their approach to online labor would have evolved. That being said, I personally found their emphasis (at least in their marketing copy) on US-based workers off-putting. Solvate's CEO was <a href="http://gigaom.com/collaboration/solvate-ceo-most-labor-platforms-undermine-american-workers/" target="_blank">quoted</a> extensively in a Gigaom article, in which he claimed that online labor markets were undermining US workers. He also suggested that by relying only on US-based workers, Solvate could promise a higher level of talent and expertise. All online labor markets have to find ways to help workers credibly demonstrate their talents, and using crude geography-based proxies for talent is an approach, but not a particularly admirable one. To me, the whole ethical/moral "so what" of online work is that geography and nationality doesn't <i>have</i> to matter. <br />
<br />
A a coda, here is my response to the original <a href="http://gigaom.com/collaboration/are-online-marketplaces-driving-down-web-worker-salaries/" target="_blank">Gigaom</a> article:<br />
<br />
<br />
<div style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; font-size: 15px; line-height: 24px; margin-bottom: 18px; outline-color: initial; outline-style: initial; outline-width: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; vertical-align: baseline;">
<span style="font-family: 'Courier New', Courier, monospace;">Full disclosure: I’m the staff economist at oDesk and these opinions represent my own views.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; font-size: 15px; line-height: 24px; margin-bottom: 18px; outline-color: initial; outline-style: initial; outline-width: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; vertical-align: baseline;">
<span style="font-family: 'Courier New', Courier, monospace;">A couple of thoughts:</span></div>
<div style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; font-size: 15px; line-height: 24px; margin-bottom: 18px; outline-color: initial; outline-style: initial; outline-width: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; vertical-align: baseline;">
</div>
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">Like any competitive market, the forces of supply and demand are going to determine prices in these online markets. With the opening up of new countries that have large, reasonably well-educated, internet savvy populations, supply increases which will tend to drive down wages. On the other hand, these markets (and the ability to break work up into small, outsourceable bits) also make it possible to outsource more work, increasing demand, and hence prices.</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;"> At least within oDesk, we haven’t seen strong trends in wages, though presumably this article is talking about freelancers in general and we obviously don’t have visibility on their wages.</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">As a practical matter, I don’t think workers in developed countries like the US can’t compete in these markets—they actually have a lot of advantages: perfect english, same time-zone, familiarity with US business culture/expectations etc. Further, price matters, but it’s not the only thing. For what it’s worth, I work with many oDesk contractors and the break-down is 1 x US, 1 x Italy, 1 x Russia, 1 x Pakistan and 2 x Philippines.</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">The efficiency and distributional effects of information and communications technology are complex and the evidence is ambiguous, so I’d be skeptical of anyone offering a definite answer to these kinds of questions. There was an interesting Quora thread on this <a href="http://www.quora.com/Will-oDesk-and-mfg-com-be-the-demise-of-the-American-middle-class" target="_blank">topic</a>.</span><span style="font-family: 'Courier New', Courier, monospace;"> </span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">I think focusing on what these markets do for relatively well-paid workers in developed countries misses one of the most important moral facts about these markets, which is that they generate new, relatively well-paid, meaningful work opportunities for people in developing countries. It’s obviously not a random sample of our workers, but If you spend a few minutes on oDesk’s Facebook fanpage and look at the comments and stories, it’s clear that online work is improving lives in a pretty dramatic way.</span></li>
</ul>
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com0tag:blogger.com,1999:blog-1597186260874280667.post-17589685347159356962012-02-18T09:59:00.000-08:002012-02-19T13:15:40.967-08:00High-wage skills on oDesk (or why you might want to learn Clojure if you're not a lawyer)<br />
<b><span style="font-family: 'Courier New', Courier, monospace;">Update: </span></b><span style="font-family: 'Courier New', Courier, monospace;">Hello HackerNews readers. One thing that I discussed but probably didn't emphasize enough is that this data show the <i>correlation </i>between listed skills and <i>offered</i> wages---you absolutely cannot infer a causal relationship (my cheeky title notwithstanding). Unless I get to create and run a massive skills training program experiment, it's going to be hard to get at causality. But I can do something about the offered/earned distinction. If you don't want to miss my follow-on blog post where I explore the relationship between skills and actual <i>earned</i> wages from actual projects, follow me on <a href="https://twitter.com/#!/johnjhorton" target="_blank">twitter</a>. </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> </span><span style="font-size: x-small;"> </span><br />
<br />
<a href="https://www.odesk.com/">oDesk</a> recently introduced a controlled, centralized vocabulary of about 1,400 skills for buyers and contractors to use when posting jobs and creating profiles. The primary motivation for the change was to make it easier for buyers and sellers to find each other: without a standardized vocabulary, would-be traders can fail to match simply because they use different terms for the same skill.<br />
<br />
A side effect of this transition is that high quality data on the relationships between skills and wages are now available. I recently built a dataset of contractors' hourly wages by skill: for each skill, I identified all contractors listing that skill on their profiles and averaged their offered hourly wages. Although contractors are free to offer any hourly wage they like, in my experience, wages offered closely map to actual earnings. However, to reduce the influence of outliers, I restricted the sample to contractors offering between 50 cents and 100 dollars per hour. I also only included skills for which there were 30 or more observations.<br />
<br />
In the bar chart below (made using the very cool <a href="http://cran.r-project.org/web/packages/googleVis/index.html">googleVis</a> package for R), I plotted the top 50 skills, ordered by average hourly wage (<a href="http://dl.dropbox.com/u/420874/skills.html">here</a> is a "live" version with mouse-over). The top of the list is dominated by high-end consulting areas (e.g., patents and venture capital consulting) or hot newer technologies (e.g., <a href="http://redis.io/">redis</a> and <a href="http://aws.amazon.com/rds/">Amazon RDS</a>). The programming language that commands the highest wage is <a href="http://en.wikipedia.org/wiki/Clojure">Clojure</a>, which is a rather esoteric skill: it's a lisp dialect that compiles to the Java Virtual Machine (JVM). Perhaps this is the market reflecting Paul Graham's <a href="http://paulgraham.com/pypar.html"> "Python Paradox"</a>:
<br />
<br />
<code>"if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the language to learn, if you want to get a good job, is a language that people don't learn merely to get a job."</code><br />
<br />
At the time Graham wrote this, Python was a far less mainstream language, probably analogous to how Clojure is regarded today. It's an interesting pattern, and although they'd cut up my economist membership card if I made a causal claim between knowing Clojure and being able to command hire wages, I'm intrigued by the idea of using online labor markets as a bellwether to help guide human capital choices.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxH4x8nDtkoFN8Bc50-uZwetxY4N3tT0Q2R-MX5doODq8jLdFEtfkMOlxwy-6SE1BZEwNdhykSUw3RfIRkTAdSACtu3oO5sWZ9rFBSqHwpGKWeOscC5T7Dh0huHAD60kOWi7hyiE_vHQJW/s1600/odesk_skills.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxH4x8nDtkoFN8Bc50-uZwetxY4N3tT0Q2R-MX5doODq8jLdFEtfkMOlxwy-6SE1BZEwNdhykSUw3RfIRkTAdSACtu3oO5sWZ9rFBSqHwpGKWeOscC5T7Dh0huHAD60kOWi7hyiE_vHQJW/s1600/odesk_skills.png" /></a></div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com12tag:blogger.com,1999:blog-1597186260874280667.post-851261107702656122012-02-16T09:35:00.000-08:002012-02-16T09:35:54.364-08:00Why aren't we all freelancers?<a href="http://www.flickr.com/photos/nationaalarchief/5897341040/" title="Eendenfokkerij / Duck farm by Nationaal Archief, on Flickr"><img alt="Eendenfokkerij / Duck farm" height="500" src="http://farm6.staticflickr.com/5301/5897341040_8027094185.jpg" width="367" /></a>
<br />
<br />
Investors typically hold diverse portfolios of assets, with the goal of reducing risk. While diversification is commonplace in investing, most of us have no diversification in our labor income streams: we work at one job at a time, for a single employer. However, the "returns" to a job vary like returns on investments, especially on non-financial dimensions (e.g., engagement, learning, co-workers, working conditions). As in investing, there is also a significant amount of direct financial risk in holding one job---the firm may impose layoffs or go out of business. Given the similarities between jobs and assets, why isn't there a similar impetus to diversify, i.e., why don't we all hold a portfolio of small jobs at the same time, with many different employers [0]?<br />
<br />
Some workers---freelancers and independent consultants---do follow this diversified model, but it's hardly the norm of workers generally. Below, I lay out a laundry list of potential economic explanations for why the portfolio/freelancing approach is not more common. What's interesting to me both academically and as someone working at oDesk is that many of these points are not set-in-stone attributes of the productive process but are instead things that smart features or policies might change. <br />
<br />
<div>
<b>Non-linearity in costs of searching/vetting/bargaining</b><br />
Hiring a freelancer for a small project is like picking out a fancy restaurant; hiring a full time employee is more like buying a house. The effort of searching and vetting (and thus the cost) is related to the stakes of the hire. However, there is no guarantee that those costs scale linearly with the stakes. Suppose it takes nearly as much effort to find a small job as it does to find a large job---then a portfolio approach will generate larger search costs per dollar earned in wages [1]. <br />
<br />
<b>Non-linearity in job size and productivity </b><br />
If you can make X widgets <i>or</i> Y schwidgets in 1 hour, it doesn't mean you can make X/2 widgets <i>and</i> Y/2 schwidgets in 1 hour. Every job has some fixed set-up costs---getting out the materials, remembering the key details, etc. The larger the costs, the less attractive the small job. On the other hand, productivity eventually wanes from boredom, physical fatigue, etc. ("I'm really getting bored with this TPS report---time for some Facebook"). The optimal size job (from a productivity standpoint) might be near or above the current 40 hours per week, 50 weeks a year paradigm, in which case going smaller means getting less efficient. <br />
<br />
<b>Complementarities with team members that grow over time</b><br />
One of the advantages of team production is that workers can share knowledge with each other, motivate each other and generally create an environment where everyone is more productive than they would be working alone. There's no reason teams of freelancers working together cannot achieve the same complementarities with each other, but if these complementarities take time to develop, larger jobs become more attractive. <br />
<br />
<div>
<b>Firm-specific human capital </b></div>
<div>
If a job requires lots of firm-specific human capital, the per-job learning requirement is high, which tends to encourage larger jobs [2].<br />
<br /></div>
</div>
<b>Monitoring & policing costs</b><br />
Once you get a sense of the character and reputation of some trading partner, you don't need to constantly monitor that person/firm. After some level of trust has been established, these costs would fall. This again pushes for larger jobs. This is probably clearer in terms of firms monitoring workers, since the big fear is shirking, but it does go both ways: workers need to make sure their checks don't bounce, that their employers aren't skimming from the 401K, using malk for the coffee service instead of milk, etc. <br />
<br />
<b>Employer concerns about IP (broadly defined)</b><br />
I do not think it is likely to find workers working simultaneously for direct competitors [3], the interests of most firms are fairly orthogonal to each other. <br />
<br />
<b>Existing public policy </b><br />
At least in the US, at the present time, certain realities (health insurance, getting financial credit etc.) are full-time employee advantaged.<br />
<br />
[0] Note that this isn't a <a href="http://en.wikipedia.org/wiki/Theory_of_the_firm#Team_production" target="_blank">theory of the firm</a> argument or discussion. I'm assuming that one can be a full employee and reap all the benefits of firm organization / team production even with fractional employment. <br />
<br />
[1] One of the reasons mechanical turk is semi-dysfunctional is that when problems arise (about the scope of work, payment terms etc.), all the surplus generated by the relationship is quickly destroyed: one minute thinking, talking and haggling about a task paying pennies is likely to be economically wasteful. This was one motivation for hagglebot. <br />
<div>
<br /></div>
<div>
<div>
<div>
[2] I think this is why ideal use of online labor is not so much a 1 for 1 replacement of some traditional job, but a decomposition of jobs into easily outsource-able pieces and pieces that require deep firm-specific knowledge. </div>
<div>
<br /></div>
<div>
[3] McKinsey excepted. </div>
<div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com1tag:blogger.com,1999:blog-1597186260874280667.post-25539243904188595882012-02-07T22:00:00.000-08:002012-02-08T07:49:18.615-08:00Writing Smell Detector (WSD) - a tool for finding problematic writing<span style="background-color: yellow;">tl;dr version: WSD is a python tool to help find problems in your writing. Here's the <a href="https://github.com/utapyngo/WritingSmellDetector" target="_blank">source</a> and here's example <a href="http://dl.dropbox.com/u/420874/test.html" target="_blank">output</a>. </span><br />
<br />
In grad school, I wrote a program that used a series of regular expressions to detect "writing smell" (analogous to<a href="http://en.wikipedia.org/wiki/Code_smell" target="_blank"> code smell</a>), i.e., telltale signs of bad writing and mistakes. The rules for smelliness were loosely based on one of my favorite writing how-to's: <a href="http://www.amazon.com/gp/product/0226899152/ref=as_li_tf_tl?ie=UTF8&tag=onlin0c-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0226899152">Style: Toward Clarity and Grace</a> by Joseph Williams.<br />
<br />
The program took as input a text file and output was an annotated report with snippets of the offending bits. I used it for all my papers and found it really helpful, but the coding was very, um, <i>academic</i> (i.e., written for use by the person who wrote it) and it was written in Mathematica [1], which was the language I knew best at the time. FWIW, <a href="http://dl.dropbox.com/u/420874/Writing%20Tools%20Based%20on%20Style%20Book.nb" target="_blank">here</a> is my original version. <br />
<br />
For a long time, I've wanted to port it to some other language and make it accessible and capable of receiving new rule contributions and explanations. To this end, I recently commissioned an oDesk contractor (<a href="https://github.com/utapyngo" style="background-color: white; color: #4183c4; font-family: helvetica, arial, freesans, clean, sans-serif; letter-spacing: -1px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">utapyngo</a>) to make a more polished, modular version in Python. I think he totally outdid himself. It's got a nice modular model now that lets you easily incorporate new rules and he greatly improved upon my often-flawed regular expressions. Be forewarned---the documentation is non-existent and the rules aren't explained, but I plan to take fix this over time, while I'm using it.<br />
<br />
It's open source (courtesy of <a href="http://www.odesk.com/" target="_blank">oDesk</a>, who paid the bills) and available <a href="https://github.com/utapyngo/WritingSmellDetector" target="_blank">here</a> on github (live <a href="http://dl.dropbox.com/u/420874/test.html" target="_blank">example output</a>). To use it, just clone it, install the python package <a href="http://jinja.pocoo.org/docs/" target="_blank">jinja2</a> and then do:<br />
<span style="font-family: monospace;"><b><br /></b></span><br />
<span style="font-family: monospace;"><b>$ python wsd.py -o output_file.html your_masterpiece.tex</b></span><br />
<span style="font-family: monospace;"><b><br /></b></span><br />
<div>
Here's a screenshot of what the HTML output looks like, illustrating the a/an rule (i.e., that it's "an ox" but "a cat"):<br />
<br /></div>
<div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOgKKY1aOrQEjCZ05CMn70oWcV6ZqWUyViC8IsiDBRYtrJdpISvDBWvGFlSJFawPO2T-ONVi-ze1zyiLtSWtd8cWxkqjDeQTe5_GFEuGxkx3gQKRpRFC4rMK_CpxLFKpQLK6RoVsyp7BD/s1600/word_smell.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOgKKY1aOrQEjCZ05CMn70oWcV6ZqWUyViC8IsiDBRYtrJdpISvDBWvGFlSJFawPO2T-ONVi-ze1zyiLtSWtd8cWxkqjDeQTe5_GFEuGxkx3gQKRpRFC4rMK_CpxLFKpQLK6RoVsyp7BD/s640/word_smell.png" width="640" /></a></div>
<br />
Note the statement of the rule, the patterns that it looks for and the snippets. It also has a hyperlink to the full text, which is available at the bottom of the document.<br />
<br />
A few thoughts:<br />
<ol>
<li>If you're interested in contributing (rules or features), let me know. </li>
<li>It might be nice to turn this into a web-service, though my instinct is that someone interested in algorithmically evaluating their LaTeX/structured text isn't going to find cloning the repository & then running a script to be a big obstacle. And they probably don't want to make their writing public. </li>
<li>A few weeks ago, I read this <i>usethis</i> profile of CS professor <a href="http://matt.might.usesthis.com/">Matt Might</a>. In the software section of the interview, he said that he had some <a href="http://matt.might.net/articles/shell-scripts-for-passive-voice-weasel-words-duplicates/" target="_blank">shell scripts</a> that do something similar. I haven't really investigated, but maybe there's ideas here worth incorporating. </li>
</ol>
[1] When I told the other members of the oDesk Research / Match Team that I had code for doing this writing smell thing, they were impressed and wanted a copy; when I told them it was written in Mathematica, they thought this was hilarious and mocked me for several minutes. I tried to explain that Mathematica actually has great tools for pattern matching, but this fell on deaf ears.Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com7tag:blogger.com,1999:blog-1597186260874280667.post-34296416895049834082012-02-06T08:42:00.000-08:002012-02-06T08:42:39.204-08:00Minimum Viable Academic Research<a href="http://www.flickr.com/photos/vanf/6548427219/" title="remember clippy by pj_vanf, on Flickr"><img src="http://farm8.staticflickr.com/7020/6548427219_2205d37b11.jpg" width="500" height="375" alt="remember clippy">
<br>
<b>A non-viable product in minimum form, courtesy of <a href="http://www.flickr.com/photos/vanf/6548427219/">Flickr</a> </b>
</a>
<div>
<br />
<br />
<br />
<br />
One of the most talked about ideas in the world of start-ups is the notion of the <a href="http://en.wikipedia.org/wiki/Minimum_viable_product" target="_blank">minimum viable product</a> (MVP). The rationale for MVP is clear: you don't want to build products that customers don’t want, never mind waste time polishing and optimizing those unwanted products. "Minimally viable" doesn't even require the product to exist yet---the viability refers to whether it will give you the feedback you need to see if the project has potential. For example, you might do an A/B test where you buy keywords for some new feature, but then just have a landing page where people can enter their email address, thereby gauging interest. The important thing is that it is market feedback, not just opinions of people near you.<br />
<br />
<br />
In academia, a big part of the the day to day work is getting feedback on ideas. Each new paper or project is like a product you’re thinking of making. So you float ideas with colleagues, your advisers, your spouse, etc., and you might present some preliminary ideas at a workshop or seminar. The problem is that in most workshops and seminars, where you could potentially get something close to a sample of what the research community will think of the final product, the feedback is usually friendly and limited to implementation (e.g., "How convincing is the answer you are providing to the question you've framed?"), instead of "market" feedback on how much "value" your work is creating.<br />
<br />
<br />
The academia analogue to market feedback on value will come later, in two forms: (a) journal reviews / editor decisions and (b) citations. By value, I mean something like (importance of question) x (usefulness of your answer). At least in economics, knowing what is important is difficult. There is no Hilbert's list of big and obvious open questions. A few such questions do exist, but they tend to be sweeping in nature---e.g., "Why are some countries rich and some countries poor?" and "Why do vacancies and unemployed workers co-exist?"---that no single work can decisively answer. To do real research, you need to pick some important part of a question and work on that. <br />
<br />
<br />
A fundamental problem is that the institutional framework in some disciplines (economics being one example, though not all---see <a href="http://www.nytimes.com/2012/01/29/opinion/sunday/the-perils-of-bite-size-science.html" target="_blank">this</a> recent NYTimes op-ed on scientific works being too short; see <a href="http://chrisblattman.com/2012/01/29/brevity-is-the-soul-of-science/" target="_blank">here</a> for an economist's take on the topic) requires you to do lots and lots of polishing before you know (via journal rejection/acceptance) whether even the most polished form of your work is going to score high enough on the importance-of-question measure. At seminars, people are usually too polite to say, "Why are you working on this?" or "Even if I believed your answer, I wouldn't care" or "So what?" But that's the kind of painful feedback that would be most useful at early stages. There are some academics that will give that kind of "Why are you doing this?" critique, and while they are notorious and induce fear in grad students, the world needs more of them. (I once gave a seminar talk where an audience member asked, "How does this study have any external validity?" And I had to admit he was right---it had none. I dropped the project shortly thereafter, after spending the better part of 3 months working on it.)<br />
<br />
<br />
It's not that people won't be critical in seminars. You'll generally get lots of grief about your modeling assumptions, econometrics, framing etc. But those are easy critiques (and they let the critics show off a little). It's the more fundamental critiques about importance/significance that are both rare and useful. In academia, you really, really need the importance/significance critique because you can work on basically anything you want, literally for years, without anyone directly questioning your judgment and choices. And while this gives you tons of freedom and flexibility, you might waste significant fractions of your career on marginalia. I also don't think it's the case that if you're good, you'll simply know: I've heard from several super-star academics that their most cited paper is one they didn't think much of when they wrote it and Their favorite paper has languished in relative obscurity. One interpretation (beyond <a href="http://blog.american.com/2009/07/summers-law-and-the-stimulus/" target="_blank">Summer's law</a>) is that you aren't the best judge of what's important.<br />
<br />
<br />
<b>How does one get more importance-of-question feedback?</b><br />
<br />
<br />
In economics, there's a tendency (need?) to write papers that are 60 page behemoths, filled with robustness checks, enormous literature reviews, extensive proofs that formalize somewhat obvious things, etc. This long, polished version really is the minimally viable version of the paper, in that you can't safely distribute more preliminary, less polished work (people might think you don't know the difference). I think on the whole, this is probably a good thing. But it's often not the minimally viable version of an idea. Often the "so what" of a paper is summarized by the abstract, a blog post, a single regression, etc. <br />
<br />
<br />
I'm not sure what the solution is, but one intriguing bit of advice I recently received from a very successful (albeit non-traditional) researcher was to essentially live-blog my research. There's actually very little chance of being "scooped”; if anything, being public about what you're doing is likely to deter others. And, because it's "just" a blog post, you nullify the "they don't know the difference between polished and unpolished work." The flip side is that I think there's a kind of folk wisdom in academia that blogging pre-tenure is a bad idea (I imagine the advice is even stronger for a grad student pre-job market). But if you were doing it for MVP reasons / feedback reasons, the slight reputation hit you'd take might be offset by the superior "so what" feedback you might get from doing such a thing. Anyway, still thinking about this strategy.*<br />
<br />
* Beyond the purely professional strategic concerns, it might actually move science along a little faster and make research a bit more democratic and <a href="http://en.wikipedia.org/wiki/Open_research" target="_blank">open</a>.</div>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com3tag:blogger.com,1999:blog-1597186260874280667.post-75175107742156940682012-02-04T13:07:00.000-08:002012-02-04T13:07:37.104-08:00Stereotypes about animals (and children) as revealed by Google auto-suggestI saw this <a href="https://twitter.com/#!/m_sendhil/status/165870948015284224/photo/1" target="_blank">tweet</a> by @m_sendhil, which had a screenshot of Google's auto-suggest for "why are indians so," which contained a collection of (often contradictory) stereotypes (e.g., fat and skinny). I began doing the same exercise for other nationalities and ethnic groups, products, animals etc.<br />
<br />
Here is the screenshot for turtles (which apparently have lots of fans):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVXtVXsF8Y0H7qdY3WGYq7Y6QzOHzcdRtROo-ai1BUvsq04zKQaze5DtzEQzxfswiPDP0OBIIj1B_nrDmn5MYEwmnIZ_Lbd_sJfZNB2YMvjD43dXtFw44n9mzkoe860D9NKz3a8van-YTz/s1600/turtles.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="124" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVXtVXsF8Y0H7qdY3WGYq7Y6QzOHzcdRtROo-ai1BUvsq04zKQaze5DtzEQzxfswiPDP0OBIIj1B_nrDmn5MYEwmnIZ_Lbd_sJfZNB2YMvjD43dXtFw44n9mzkoe860D9NKz3a8van-YTz/s640/turtles.png" width="640" /></a></div>
<br />
<br />
It was interesting to me how many of the supposed attributes showed up repeatedly across entities. This gave me an idea: I should turn this procrastination/time-wasting into something more useful, which was to learn how to make graph/network plots with the python package <a href="http://networkx.lanl.gov/" target="_blank">networkx</a> (code below). Here is the result, using the top 4 auto-suggests for cats, children, cows, dogs, frogs, goldfish, hamsters, mice, turtles and pigs. Entities are in blue, attributes in red. Edges are drawn if that attribute was auto-suggested for that entity. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOuChZmWPIJ_kWcog5BfitHne2Yb5q-h0plWpMtd_d1afsGCYxi0Mr72UmZDuWoWSxRvRmOYFxONvwmJH1ldFU7mnpxt1wN9-39EMvqnLesfLZJx0WtyUtCHpJFYhRBgF3fYsbGHfRY_Rz/s1600/relationships.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOuChZmWPIJ_kWcog5BfitHne2Yb5q-h0plWpMtd_d1afsGCYxi0Mr72UmZDuWoWSxRvRmOYFxONvwmJH1ldFU7mnpxt1wN9-39EMvqnLesfLZJx0WtyUtCHpJFYhRBgF3fYsbGHfRY_Rz/s640/relationships.png" width="640" /></a></div>
<b>Some observations</b><br />
I'm guessing the "addicting" and"good" attributes of goldfish refers to the cheesy snack cracker and not the actual fish. People seem to be rather ambivalent about children. I'm kind of surprised that people were not wondering why dogs are smart. Finally, are pigs actually salty (this seems unlikely), or this just how pork is usually prepared? <br />
<br />
<b>The code: </b><br />
<br />
<script src="https://gist.github.com/1740131.js?file=stereotypes_about_animals.py">
</script>Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-8042888810118878482012-02-01T08:59:00.000-08:002012-02-01T09:03:44.216-08:00Employer recruiting intensityI was reading/skimming <a href="http://faculty.chicagobooth.edu/steven.davis/pdf/w16265.pdf" target="_blank">this</a> paper by Davis et al. and in the abstract, they write: <br />
<br />
"<i>This paper is the first to study vacancies, hires, and vacancy yields at the establishment level in the </i><i>Job Openings and Labor Turnover Survey, a large sample of U.S. employers. ... </i><i>We show that (a) employers rely heavily on other </i><i>instruments, in addition to vacancy numbers, as they vary hires, (b) the hiring technology exhibits </i><i>strong increasing returns to vacancies at the establishment level, or both. <b>We also develop evidence </b></i><i><b>that effective recruiting intensity per vacancy varies over time, accounting for about 35% of movements in aggregate hires.</b></i>"<br />
<br />
<br />
In a nutshell, they document that recruiting intensity varies across time and that this variation has a big effect on the number of aggregate hires. What's interesting is that the labor literature tends to focus on search intensity by workers, with firm search intensity comparatively understudied, but this paper suggests that ignoring employer efforts is likely to give a (very) incomplete impression. My guess is that this bias in the literature comes from the comparative lack of employer data on matching, though JOLTS (which this paper uses) is ameliorating the problem.<br />
<br />
On oDesk, we've got excellent visibility on employer recruiting. Below is the "so what" plot from a recent experiment where we "recommended" contractors to employers (based on our analysis of what the job consisted of). The recommendations came immediately after the employer posted the job. We also made it easier for that employer to invite those recommended contractors to apply. The y-axis is the fraction of jobs where the employer made at least one invitation; treatment and control are side-by-side. We can see that regardless of category, the treatment was generally effective in increasing the number of invitations. But I think the striking thing is how much variation there is in "levels" of recruiting by category: in the control admin group, less than 10% of employers recruited, while in sales, it's almost 25%.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://img.skitch.com/20120201-1gmcjig3acmucgh1b65syxkff9.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="514" src="https://img.skitch.com/20120201-1gmcjig3acmucgh1b65syxkff9.jpg" width="640" /></a></div>
<br />
Presumably the difference depends on a number of factors: how many applicants the job will get organically, how close a substitutes are the different applicants, the value to the firm of filling the vacancy to the firm and so on. It also clearly matters how easy it is to search/recruit, given the effectiveness of our pretty lightweight intervention. From a welfare standpoint, this last point about the role of search/recruiting cost is potentially interesting, as reducing employer search frictions/costs technologically is, at least in online labor markets, a highly scalable proposition.Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com0tag:blogger.com,1999:blog-1597186260874280667.post-22602517263221093182011-11-26T11:18:00.001-08:002011-11-26T11:51:54.265-08:00What do contractors in machine learning charge by the hour?I saw <a href="http://www.quora.com/Machine-Learning/What-do-contractors-in-machine-learning-charge-by-the-hour">this</a> question on Quora this morning: "What do contractors in machine learning charge by the hour?"<br />
<br />
Obviously the answer depends on the skill level, but based on the few machine learning contractors I've worked with on oDesk, I would guess about $30-$40/hour is average. However, I wanted to check with some real data. I have access to our full database, but a more useful answer would allow people to see how to do this for other skills by simply scraping our search results (we have an API, but I don't know how to use it <i>yet </i>and I wanted to try the <a href="http://www.crummy.com/software/BeautifulSoup/" target="_blank">BeautifulSoup</a> package).<br />
<br />
First step was to reverse-engineer our search syntax and the HTML for profile rates. A contractor search of "machine learning" gives this:<br />
<br />
"<a href="https://www.odesk.com/contractors?nbs=1#q=machine+learning">https://www.odesk.com/contractors?nbs=1#q=machine+learning</a>"<br />
<br />
and if I click on the next 10 results, the url is:<br />
<br />
"<a href="https://www.odesk.com/contractors?nbs=1#q=machine+learning&skip=10">https://www.odesk.com/contractors?nbs=1#q=machine+learning&skip=10</a>"<br />
<br />
I then checked to see what happens if I set "skip=0"---it returns that same thing as the first search URL, so now know that I can just write one function that returns the query URL, with parameters of "q" and "skip."<br />
<br />
Next, I needed to find out how the rates are stored. Using Chrome's awesome "Inspect element" feature, I found that the rates for the 10 returned results as listed as "rate_1", "rate_2", and so on:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://img.skitch.com/20111126-nhx4fa3scy5gpdq818u268b2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="450" src="https://img.skitch.com/20111126-nhx4fa3scy5gpdq818u268b2.jpg" width="640" /></a></div>
<br />
The rest was pretty easy---I just wrote two loops to collect up wages. I saw that we had some clear false positives, which I filtered out. This actually brings up a big problem on oDesk that we've been working on---namely that until recently, we had no standardization of skills, which made it hard to match people or do really good, highly specific queries. We've now moved to a closed (but expandable) vocabulary of skills, ala StackOverflow which in the long run will make it much easier to do matching and recommendations (and little data projects like this). That's a topic for another blog post. So, returning to the original question, my answer based on a pretty tiny sample: <br />
<br />
<br />
<b>Min: 16.67</b><br />
<b>Max: 100.0</b><br />
<b>Mean: 39.6983333333</b><br />
<br />
<br />
And here's my pretty crappy but for-the-moment-functional code on GitHub:<br />
<br />
<script src="https://gist.github.com/1396169.js?file=get_machine_learning_wages_on_odesk.py">
</script>
<br />
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com3tag:blogger.com,1999:blog-1597186260874280667.post-67942379095179299262011-10-29T11:44:00.000-07:002013-12-01T14:26:47.785-08:00Should Online Labor Markets Set a Minimum Wage?Some critics of online labor markets mistakenly believe that the platform creators have an incentive to keep wages low. <i>Employers</i> have that incentive, but all else equal, the creators of online labor markets want to see see wages rise, since almost all of them take a percentage of earnings. At least from a revenue standpoint, they should be indifferent between more work at a lower price or less work at a higher price, so long as the wage bill stays the same. It would be a different story if the platforms could somehow tax employers on the value-added by the platform, but so far, that's not possible.<br />
<br />
One tool for raising wages might be for the platform to impose a minimum wage. It's certainly possible that imposing a binding minimum wage would increase platform revenue---it depends on the relative labor supply and demand elasticities, as well as the marginal cost of intermediating work. A platform's costs of good sold (i.e., intermediation service) is not precisely zero---there are server costs, customer service costs, fraud risks etc, so there is some wage at which the platform would be better off not allowing parties to contract.<br />
<br />
Moving from generalities, let's look at workers from the Philippines, who (a) make a big chunk of the workforce on oDesk and (b) generally do relatively low-paid work (e.g., data entry, customer service, writing etc.) and thus would be most affected by a minimum wage imposition. If we look from about 2009 on (when the Philippines first started to become important), we can see that wages are basically flat, perhaps with a slight rise in some categories.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://img.skitch.com/20111029-q66wn3fikckjryyxs4xpssa968.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="500" src="https://img.skitch.com/20111029-q66wn3fikckjryyxs4xpssa968.jpg" width="640" /></a></div>
<br />
<div>
We can see that mean hourly wages range from $3 (for low-skilled data entry work) to about $8/hour for software development. By US standards, $3/hour is quite low---it's less than half the US federal minimum wage. However, let's look at where $3/hour wage puts someone in the the Philippine <b>household</b> income distribution, assuming they work 40 hours a week, 50 weeks a year: </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://img.skitch.com/20111029-s322de6pcpnbm5a8w1qk78gu8.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="464" src="https://img.skitch.com/20111029-s322de6pcpnbm5a8w1qk78gu8.jpg" width="640" /></a></div>
<div>
<br />
Unfortunately the I couldn't get a more refined measure of income, but my eye-ball estimate is that $3.00/hour is at about the 50th percentile of the distribution. The equivalent hourly wage for median household income in the US is about $31/hour (<a href="http://en.wikipedia.org/wiki/Household_income_in_the_United_States">using 2006 measure from Wikipedia</a>) using the same 50 weeks a year, 40 hours a week formulation. It's important to note that this is household income, meaning that in many cases it is the combined income of a husband, wife and working-age children. And although online work does require a computer and a good internet connection, it does not require spending money on transportation, work clothes, food prepared outside the home etc. It also probably lets workers economize on child care e.g., I might be willing to let a 10 year old watch a 3 year old if I'm in the next room, but not if I'm across town.<br />
<br />
So, what's the conclusion?<br />
<br />
From a platform perspective, I can concede that imposing a minimum wage could be revenue-increasing, but it depends on some pretty hard to estimate factors: how well do we know the elasticities? Are the long and short-term elasticities the same? What happens if we can get our intermediation costs down? Implementation-wise, enforcement might be very hard---I could easily imagine workers giving under-the-table rebates.<br />
<br />
From a worker/welfare perspective, a minimum wage would clearly help some but hurt others. Any binding minimum wage is going to price some workers out of the market. How do we weigh their lost opportunities against the increased wages paid to those that see a bump? This starkly highlights one of the real drawbacks of a minimum wage as social policy, which is that it might be globally progressive and yet highly locally regressive for workers on the bad side of the cut-off. </div>
<br />
I'd love to hear both employer and worker perspectives on this---feel free to comment here & I'll respond.<br />
<br />
<i><span class="Apple-style-span" style="color: #999999;">Like this? Follow me on <a href="https://twitter.com/#!/johnjhorton">twitter</a>. </span></i><br />
Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com2tag:blogger.com,1999:blog-1597186260874280667.post-60529834328387902362011-10-27T20:50:00.000-07:002011-10-27T20:59:07.653-07:00Workers-as-Bundled-Goods<div>
<a href="http://www.flickr.com/photos/simonmiller/169060848/" target="_blank"><img alt="Bigger Big Mac by Simon Miller, on Flickr" border="0" src="http://farm1.static.flickr.com/55/169060848_f73e2fa85e_m.jpg" title="Bigger Big Mac by Simon Miller, on Flickr" /></a><br />
<a href="http://creativecommons.org/licenses/by-nc-sa/2.0/" target="_blank"><img align="left" alt="Creative Commons Attribution-Noncommercial-Share Alike 2.0 Generic License" border="0" src="http://i.creativecommons.org/l/by-nc-sa/2.0/80x15.png" title="Creative Commons Attribution-Noncommercial-Share Alike 2.0 Generic License" /></a> Image by <a href="http://www.flickr.com/people/simonmiller/" target="_blank">Simon Miller</a><a href="http://www.imagecodr.org/" target="_blank"> </a><br />
<br /></div>
A standard pricing strategy in many industries is <a href="http://en.wikipedia.org/wiki/Product_bundling">bundling goods</a>, e.g., productivity "suites" like Microsoft Office, value meals at fast food restaurants, hotel and flight combos, etc. In the labor market, we also see a kind of bundling, though not by design: each worker is a collection of skills and attributes that can't be broken apart and purchased separately by the firm. For example, by hiring me, my company gets my writing, meeting attendance, programming, etc.; they can't choose to not buy my low-quality expense-report-filing service.<br />
<br />
Good mangers deal with this bundling by keeping workers engaged at their highest value activity. However, every activity has decreasing marginal returns, so even activities that start out as high-value eventually reach the "flat of the curve" where the marginal benefit of more of X gets pretty small. This phenomena gives large firms an advantage, in that their (generally) larger problems give workers more runway to ply their best skills (by the same token, small firms have to worry much more about "fit" within their existing team).<br />
<br />
While pervasive, this flat-of-the curve dynamic and the resulting small-firm handicap is not a fundamental feature of organizations or labor markets--it springs from the binary nature of employment. It goes away or it least is diminished if a worker can instead being partly employed (i.e., freelance) at a number of firms, each paying the worker to do what they do best. To date, the stated value proposition of most freelancing sites has been that they allow for global wage arbitrage. Obviously that's important, but I suspect this "unbundling" efficiency gain will, in the long term, have a more profound effect on how firms organize and how labor markets function.Anonymoushttp://www.blogger.com/profile/14545375829531307836noreply@blogger.com0