Impressions from a visit to a large call center

Paul Krugman has a piece of research advice, which is to "listen to the gentiles." What he means is to pay attention to what smart practitioners say about their business in order to get economic ideas and insights. I recently had the chance to take a tour of large call center for a major global financial institution. It was interesting throughout and I thought I would share some of my notes.

Recruiting and Training
The company uses a tiered screening approach. They start with an online test. Some proceed to phone interviews and the finally round is in-person interviews. The HR director felt that the simulated work environment test was primary predictor of future job success---this matches up quite well with the industrial psychology literature on employer screening. He also felt that the best predictor of retention was how comfortable a worker seemed up front with the demands of shift-work.

The company made extensive use of their existing employees to recruit new ones. Referrals were highly valued because they were more likely to bring in candidates who understood the reality of shift-based call center work (and thus were less likely to turn over). It seemed to be less about bonding or reducing formal recruitment costs.

Long company-specific training period, but no general training (as Becker would have predicted). However, there are several other call centers in this region and the company does lose employees to them. The company-specific training period was surprisingly long (on the order of 3 months) and was conducted by more senior employees during low call volume periods.

Compensation

The company acts like a price taker with respect to wages. Compensation for new employees was determined by doing yearly market research into what competitors were paying. The company did have very high turn-over (though about in line with the industry), but there was no mention of raising wages as a solution. Their approach seems to be to wait for people to "sort out" of the job that find that they cannot handle shift work. I meant to ask about explicit performance incentives but didn't get a chance to. However, my impression was that rewards came through promotion and one-off bonuses rather than through relating payment to specific actions, despite performance being quite measurable.    

They had surprisingly rich amenities. Although pay was not high, amenities were reminiscent of a Silicon Valley start-up: pleasant office, cheap and free food, free gym, concierge service etc. Some of these things seemed like amenities the firm could more cheaply offer than their competitors because of their larger size, since some were club goods. In other words, they could amortize a concierge over many more employees.    

Operations 

Customers are segmented by value and routed accordingly. The company is multi-national and has call centers in several locations, including Europe, Southwest Asia and East Asia. The company's clients are segmented based on value and routed to the call center that roughly corresponds to the skill level of the workers at the call center e.g., the best customers get the European call center and low-tier do not.

They are highly sophisticated at demand and supply management. Perhaps unsurprisingly, they are good at forecasting call volumes and staffing accordingly---with all of this done semi-automatically. They can adjust supply on the fly by calling off training, meetings etc. if demand spikes via building-wide announcements of status changes.

There was little evidence that much technologically-driven productivity improvement was on the horizon. Although the tasks are highly structured, there was no evidence that significant technology-driven productivity gains were on the horizon. All the big gains from automation already occurred many years ago (e.g., the ubiquitous "Press 1 for "Accounts"). There was no talk of Watson-like automation of responses to customer queries. The one technology they really wanted---and that would radically reduce their costs---was some easy way to verify customer identities over the phone. This alone would increase their productivity by about 20-30%.

There was little evidence that this would could be easily distributed. Most of the firm's workplace policies seemed to be driven by concerns about regulatory compliance and fear of losing sensitive customer information and required a great deal of monitoring and control. It is difficult to imagine a substantial chunk of this work being done by a geographically distributed workforce.

You Can Sometimes Trust Research Done on Mechanical Turk, But It Depends on the Research Question

Dan Kahan has an interesting post on some of the validity problems with research conducted on Mechanical Turk (MTurk). I think I largely agree with his main point, which is that the evolution of the marketplace has been such that it's become less useful for conducting certain kinds of research. However, I do worry there's a potential baby/bathwater problem if researchers decide that "unrepresentative" or "experiment-savvy" means a useless subject pool (e.g., Andrew Gelman titled his blog post about Kahan's article "Don't Trust the Turk").

I haven't done MTurk research in several years, but the external validity issue raised by the blog post is something I thought about quite a bit when I was running experiments on the platform. I wrote a section about external validity in my ExpEcon paper with Richard Zeckhauser and Dave Rand). They key portion is excerpted below (the source code and data for that paper are available here):

Representativeness 

People who choose to participate in social science experiments represent a small segment of the population. The same is true of people work online. Just as the university students who make up the subjects in most physical laboratory experiments are highly selected compared to the U.S. population, so too are subjects in online experiments, although along different demographic dimensions.

The demographics of MTurk are in flux, but surveys have found that U.S.-based workers are more likely to be younger and female, while non-U.S. workers are overwhelmingly from India and are more likely to be male (Ipeirotis, 2010). However, even if subjects "look like" some population of interest in terms of observable characteristics, some degree of self-selection of participation is unavoidable. As in the physical laboratory, and in almost all empirical social science, issues related to selection and "realism'" exist online, but these issues do not undermine the usefulness of such research (Falk, 2009).

Estimates of changes versus estimates of levels 

Quantitative research in the social sciences generally takes one of two forms: it is either trying to estimate a level or a change. For "levels" research (for example, what is the infant mortality in the United States? Did the economy expand last quarter? How many people support candidate X?), only a representative sample can guarantee a credible answer. For example, if we disproportionately surveyed young people, we could not assess X's overall popularity.

For "changes" research (for example, does mercury cause autism? Do angry individuals take more risks? Do wage reductions reduce output?), the critical concern is the sign of the change's effect; the precise magnitude of the effect is often secondary. Once a phenomenon has been identified, "changes'" research might make “levels” research desirable to estimate magnitudes for the specific populations of interest. These two kinds of empirical research often use similar methods and even the same data sources, but one suffers greatly when subject pools are unrepresentative, the other much less so.

Laboratory investigations are particularly helpful in "changes" research that seeks to identify phenomena or to elucidate causal mechanisms. Before we even have a well-formed theory to test, we may want to run experiments simply to collect more data on phenomena. This kind of research requires an iterative process of generating hypotheses, testing them, examining the data and then discarding hypotheses. More tests then follow and so on. Because the search space is often large, numerous cycles are needed, which gives the online laboratory an advantage due to its low costs and speedy accretion of subjects.

Platforms can tax externalities and generate costly signals

The word "URGENT" should cost at least $100 per usage.
— Merlin Mann (@hotdogsladies) May 24, 2013

One thing that's great about platforms is that socially efficient, signal-generating Pigovian taxation like the kind proposed in this tweet is not a joke---you can actually do things like this, which may be one of the great advantages of markets mediated by a powerful third party.

Country-Specific Minimum Wage Data, Courtesy of Wikipedia

I was looking for some data on minimum wages in various countries and found that Wikipedia (perhaps unsurprisingly) has a very nice, well-annotated table. After downloading the data & cleaning it a bit (harder than it should be), I made several plots. There were too many countries for one plot, so I made one for each (approximate) quartile. At the end of the blog post is the R code I used for fetching the data & making the plots.

Fourth Quartile 

NB: Some countries have exemption policies for worker or occupation characteristics, so for a more complete understanding, of say, why Australia appears to have a minimum wage more than 2x the US minimum wage, check the Wikipedia table. 

Third Quartile 

Second Quartile 

First Quartile 

Distribution of Minimum Wages

Below is a some R code for grabbing the table of country-specific minimum wages from Wikipedia.

The Indian blackouts & oDesk

A nationwide blackout in India has left some 600 million people without electricity. Given that a large number of the contractors on oDesk are from India, I assumed that effects of the blackout would show up readily in the oDesk data. This evening, I wrote a query to get the hours worked each day by Indian contractors during the last month and the number of applications sent. I divided these counts by the respective totals for that day for all of oDesk. From this time series, we can get a sense of what was supposed to happen today and compare it to what actually happened. The time series for applications (top) and hours worked (bottom) are plotted below [1], with today annotated in red. Each percentage estimate has a 95% confidence interval.  

Some observations

• There is a very easy to detect drop-off in the hours worked---my eyeball calculation says they should have been responsible for around 22% of the hours worked today, while the actual number is closer 17.5%. This is far less of a fall-off than we would naively predict from the "1/2 of Indians without power" headline. Presumably many contractors have access to private generators, or perhaps oDesk is over-represented in parts of the country that were less affected by the blackout. 
• There is no corresponding obvious drop-off in the fraction of applications. I don't have a good explanation for this, but perhaps non-affected Indian contractors have made up the difference and exploited the now-thinner market. If I can get some data on what parts of the country are actually being affected by the blackout, I could test this notion since I do have contractor locations down to the city level. 
• Indian contractors take weekends off, both in terms of working and job finding (or at least more so than their oDesk counter-parts from other countries). Remember that this time series is the fraction for a given day, so there's no reason for a strong weekend/weekday pattern. See oDesk Country Explorer for more of this kind of data.   
• Indian contractors are generally over-represented in the application pool, making up ~25% of applications but only about ~20% of hours worked, though this could easily reflect differences in the kinds of categories Indian contractors work in---there is a great deal of variance in the average number of applications per opening across the different job categories. 

Code for the plots (done in ggplot2):

Digitization of the supply side of the labor market

Note: This blog post also contains a short review of Google's new Consumer Surveys service. See the end of the blog post for details. 

On most electronic commerce sites, information about the supply side is digitized and publicly available while information about the demand side is generally not: Amazon, Expedia, iTunes, Etsy etc., all collect and display detailed data about the items for sale, but there is generally little or no information about the consumers with the demands. If we look at the labor market, the reverse us true, in that it is the demand side that's digitized. On online job boards like CareerBuilder, Monster.com, Indeed, SimplyHired etc., vacancies are described via detailed textual descriptions about the nature of the work, skills required, location and approximate salary, but the job seekers---the sellers---generally do not create profiles that describe themselves to the marketplace. 


While we might think that there are some fundamental reason for this difference, I don't think this is the case for the simple reason that in the case of labor markets, the supply side is being digitized, primarily though LinkedIn (in a big way) and through sites like oDesk (in a comparatively smaller, but more comprehensive way). On these sites, workers create permanent, searchable profiles for employers that containe rich, employment-relevant data about themselves.

With the rise of LinkedIn, we are witnessing an unprecedented, voluntary data collection and digitization of the supply side of the labor market. On LinkedIn, individuals can create public profiles and list their education, professional credentials, associations, skills, current and past work experiences and, critically, their other professional connections (indicated by approved links to other LinkedIn users).  As of yesterday (July 24th, 2012), approximately 19% of the US-based Internet using population had a LinkedIn profile [* see note below for interesting background for this 19% figure]. According to LinkedIn, as of March 12, 2012, over 160 million people have created profiles, and in many industries, a LinkedIn profile is expected of all applicants. I talked recently to oDesk's corporate recruiter, asking her how many candidates had LinkedIn profiles. She responded: 

I'd say it is close to 100% (and certainly 100% for viable candidates).   I can't think of an example of someone who I have screened who didn't have a profile on LinkedIn. 

I think this supply digitization is likely to prove consequential, because once the supply side of the labor market is digitized, platforms can begin making data-driven, highly contextualized recommendations to both sides of the market. The recommendations made by a platform can have the advantage of being potentially informed by the platform's holistic perspective on the marketplace. In computer-mediated marketplaces, by necessity essentially every piece of data that goes into or is generated by the marketplace is captured in an electronic database that could conceivably used to make recommendations. 

Of course, job board do try to make recommendations by suggesting vacancies to workers, but they are limited to conditioning those recommendations on whatever search terms and perhaps geographic and/or salary constraints a job-seeker enters in a relatively brief search session. The platform cannot condition its recommendations on a worker's employment history, educational background, skills, current employment status, professional connections, certifications, personality, test scores and other match-relevant factors, nevermind try to balance recommendations to navigate the twin shoals of market thinness and market congestion. 

Unfortunately, I think a lot of this work on recommendations will happen within companies in a state of semi-secrecy, but hopefully enough will be made public that others can contribute, ala the Netflix challenge. It's a little sad that to date society has expended more machine learning research effort trying to predict taste in moves rather than fit for jobs, despite the enormous welfare consequences of the labor market. However, I predict this will change and expect a lot more work on this topic from computer scientists and market designers in the coming years. 

[*] The Origin of "19% of the US Population has a LinkedIn profile" Number

In writing this blog post, I wanted to get an accurate number for what fraction of the US population has a LinkedIn profile. This number was proving hard to come by, so I decided to try a relatively new service launched by Google called Google Consumer Surveys. For 10 cents an answer, you can pose questions to a supposedly representative sample of US-based Internet users. You also get some of the respondent's basic demographics, such as inferred age, gender and income. I launched a one question survey and got 1511 responses in less than a day. The screenshot below shows the main results, but it also includes some neat tools for looking at the data in different ways. I made the survey public---check it out here. I'm quite pleased with the service and plan to use it again.  

Shrimponomics, Complements & BPOs

Most relevant image available from doing a
Google Image search for "Shrimp using a computer"

A few years ago, there was a Freakonomics post about how people reason about economic situations and phenomena. The phenomenon in question was shrimp consumption: the amount of shrimp people eat in the US per capita  tripled between 1982 and 2007. When asked to explain this rise, non-economists mainly give demand reasons (changes in preferences), while economists are more likely to also give supply reasons (improved fishing efficiency, rise of aquaculture etc.). 

If I had to offer an explanation for this focus on demand explanations, my guess it that demand explanations come more easily to us because it is the side of the market that is more familiar to us : most of us have eaten shrimp & bought shrimp---very few of us have worked in commercial fishing. So when asked "why are people consuming more shrimp?" we start with "why might I consume more shrimp?" and although price is certainly a reason (and a path of thought that would help lead to a demand explanation), it's not as salient or even as interesting as things like changing tastes, health trends, exciting new shrimp-based dishes etc.     

So this blog post isn't about shrimp and it isn't about supply & demand. It's about complements and substitutes. I think there is a similar psychological tendency to focus on goods-as-substitutes than goods-as-complements.  At the individual level where we are making choices, we are usually thinking in terms of substitutes: do I want coffee or tea? Should I take a vacation to Las Vegas or Hawaii? Mac or PC? It's a bit more subtle to think about "if I had X, would it make Y more useful to me" which is at the heart of all complementarity stories. 

This is a long-winded introduction to my real topic, which is that in my last blog post, I made the argument that online work could disrupt the BPO industry by serving as a substitute for what BPOs offer. A point I didn't think of---but in retrospect seems pretty obvious---is how just as easily complementarity could be the dominate effect. After my blog post, my CTO at oDesk, Odysseas, emailed me with his thoughts:  

The primary benefit of BPOs is not that of labor cost arbitrage. Thats typically the motive/benefit for offshore staff augmentation firms - but BPOs are business process outsourcers. BPO is ADP [Automated Data Processing] that outsources your payroll or a business that outsource your HR process etc... We often tend to think of BPOs as an offshore firm that does a little bit of everything having as sole pivot point its lower cost of labor - thats true, but its an abuse of the term and I would agree there that the particular type of business is going to be affected in the years to come from online labor.

This part is basically my substitutes story---now the complements part:

However, the more interesting effect would be the effect of online labor to the real BPOs..

There BPOs will not be negatively affected - the opposite. The availability of online labor would allow BPOs to become more flexible lower their overall fixed costs force them to become more automated and streamline (their virtual nature will require that), allowing them to lower even the cost per customer, allowing them to focus on smaller projects, smaller customers allowing to address smaller/different market segments.  They will become less relying on an enterprise sales force customer acquisition model which is dramatically affecting their cost structure.

We are seing examples of what the new BPOs will become in companies that outsource the process of testing (uTest) of seo writing (Mediapiston) etc.

He's of course exactly right---and he's a CS PhD, not an economist, so shame on me :). If you think of true BPOs in the sense that Odysseas is talking about, then the complementarity story becomes more important. These true BPOs would be big buyers in the inputs market and would benefit greatly from a liquid, efficient market for labor.