Restaurant Hygiene and Social Media: How to Improve Regulatory Disclosures in the Digital Age

Editor's note: The following is the first chapter of Retooling Metropolis (2016) published by the Manhattan Institute.

Online review platforms have dramatically changed the way people learn about the quality of products and services. Yelp and TripAdvisor, for example, are now dominant information sources that exert considerable influence over consumer behavior. But social media has broad implications for the improvement of government regulatory policy as well. Consider that health officials now regularly assign hygiene scores to restaurants and increasingly require that the scores be publicly displayed. What role might social media play in making the results of these restaurant inspections—and other inspections—more widely known and useful? Drawing on my academic research and fieldwork, I argue that: (1) online review platforms can be useful vehicles for government regulatory disclosures; (2) the regulatory regimes that generate these disclosures can themselves be made more efficient by the user-generated content on these social media; and (3) government disclosure policies should take into consideration the information that is already readily available to customers on social media.

Health departments in nearly every developed country inspect restaurants to identify and reduce practices and conditions, such as storage of food at unsafe temperatures, that can lead to illness. While the worst offenders are forced to close, restaurants can fall into the “pass” category with a range of health-code violations. A central question faced by regulators is whether—and how—to inform the public about the results of their health inspections. The disclosure of such scores can help people decide where they might, and might not, eat; and it can also create incentives for restaurants to improve their food-safety practices. This mirrors the policy challenges faced in a variety of settings, ranging from food labeling to gas mileage.[1]

Since the late 1990s, there has been a vocal movement to require restaurants to prominently post their hygiene scores. Research by Ginger Jin and Phillip Leslie show that Los Angeles’s mandatory posting policy, passed in December 1997, led to improved hygiene scores and a 20% reduction in rates of food-borne illnesses. Based largely on their research, hygiene-score posting is frequently cited as a success, and other cities, including New York, have implemented similar requirements.[2]

When “scores on doors” began, there was no Facebook, Twitter, or Yelp. Today Yelp has amassed about 100 million reviews, while TripAdvisor, Chowhound, and other websites have amassed similar figures. Reviewers on these platforms don’t typically (if at all) carry clipboards or charts to eateries, but they do comment on issues that interest health inspectors. And my research indicates that consumers increasingly turn to these types of social-media platforms for guidance.[3]

Public reporting and inspections should draw on and interact with social media, not ignore it. Moving toward digital disclosure can potentially reduce costs and produce reports that are more in line with the way millions of people learn about products and services. Moreover, regulators and other public officials should leverage the data that already exist on social media to improve the efficiency of government operations. While this essay focuses on restaurant hygiene, the principles and methods described can be applied in a variety of settings.

Although restaurant inspections differ by city, the main elements are similar. San Francisco’s Retail Food Safety Program, for example, is overseen by the city’s Department of Public Health (SFDPH), which administers inspections of “restaurants, markets, and all other retail food operations,”[4] such as bars, pushcarts, and bakeries. Unannounced inspections are used to calculate food-safety scores based on an establishment’s compliance with health and safety regulations.

Each inspection results in a score from 0 to 100. Deductions from the perfect score of 100 are made for each violation, with more serious infractions receiving larger deductions. SFDPH classifies violations as high-, moderate-, or low-risk.

Major safety violations that endanger public health can result in a restaurant’s closure, and restaurants are supposed to be inspected two or three times per year, depending on their scores. In many cities, however, lack of resources means that inspections are not as frequent. Cities could improve their inspection and disclosure policies by working with social media in three ways: as a disclosure outlet; as a means for leveraging online information; and as a tool for determining optimal disclosure practices.

Social media as an outlet for disclosure

I recently collaborated on a project to incorporate the SFDPH’s hygiene scores onto Yelp’s website and analyze the results. For the initial posting, public-inspection results for roughly 4,000 San Francisco restaurants were matched with Yelp’s records. (The results became publicly visible on March 25, 2013.) Since then, my collaborators and I have monitored and analyzed the relationship between hygiene scores and Yelp reviews. Records are now updated after each inspection and automatically posted to Yelp; after our initial matching, new restaurants are matched upon the first inspection.

When Yelp users now research a restaurant, they see reviews, ratings, and hygiene scores (“health scores”). If a user clicks on the last, she is brought to a screen, as in Figure 1, displaying the violations that led to the score, as well as results from previous inspections.

The initial consumer reaction was limited, in part because scores were given little space on the main page, and a click of a button was required to see the full history. In response, we focused on San Francisco restaurants most in need of improvement (specifically, those that scored in the bottom 5%) by creating a “consumer alert” that popped up when visitors arrived at these restaurants’ Yelp pages. (For all other restaurants, no alert would appear, though the original, less prominent, hygiene-score display remained.) This modification produced roughly a 20% drop in reservations and deliveries for restaurants with the lowest scores. This suggests the importance of design for digital disclosures of government regulatory information.[5]  Disclosure, in other words, requires more than just data dumps. Collaborations with review platforms would allow governments to make data-driven design changes to better inform the public. This can help to shift the mind-set from “Does disclosure work?” to “How should disclosure be designed?” And online platforms provide an important tool in understanding what works.

Leveraging online information to improve regulatory regimes

Apart from basic guidelines—such as conducting more frequent inspections at sushi restaurants (because of their raw menu offerings) than, say, burger joints—health officials typically assign inspectors in a fairly random manner. But social media can help health inspections become more efficient.

Suppose you were looking for restaurants that were causing food poisoning. What if you were to search Yelp for terms like “sick” or “dirty”? You would probably find a few culprits. But the entire history of Yelp reviews and ratings can be merged with the history of hygiene violations to create an algorithm that identifies the likelihood of finding a hygiene violation. The algorithm would “learn” from the history of past reviews and predict the likelihood of finding violations on the basis of recent reviews. Inspectors could then be reallocated to the restaurants most likely to have violations. Similar methods could be applied to inspections of other kinds of facilities, such as parks, hospitals, dentists, and so on.

A 2013 research study (which I coauthored) explored the feasibility of using natural language processing to predict hygiene violations using Yelp reviews.[6] Figure 2 makes clear that Yelp scores can help predict hygiene scores. Indeed, after applying our algorithm, we were able to accurately predict—using only Yelp reviews and ratings—more than 80% of hygiene scores for San Francisco restaurants that scored in the bottom half. A 2014 study implemented a similar analysis for New York City, based on searches for terms such as “sick” and “vomit.”[7]  These findings suggest that cities can use private, user-generated data to develop algorithms to better target inspections, reduce administrative costs, predict potential problems, and provide immediate quality-improvement efforts.

Cities have been slow to incorporate algorithms and predictive analytics into their operations, and one problem may be that their staffs lack the technical skills to do so. One solution might be to turn to consulting companies. Another might be to have cities run prediction tournaments on platforms such as Kaggle, Topcoder, or DrivenData that host data-science contests. These tournaments might allow cities to acquire the technical skills necessary to create city-specific algorithms; they might also increase the diffusion of new methods and insights.[8]

Last year, I and my collaborators Ed Glaeser, Andrew Hillis, and Scott Kominers partnered with Yelp, Boston (which does employ data scientists), and DrivenData to run a contest to create a city-specific algorithm. The tournament attracted more than 500 contestants, with 55 submitting at least one set of predictions. The winning algorithm positioned Boston to cut the number of restaurant inspections by 40% while identifying the same number of hygiene violations by switching to a targeted inspection process. We worked with Boston on an experiment to explore the value of using this type of algorithm for targeting health inspections. Our research found that an algorithm outperformed not only random inspections but also a list of restaurant targets created by inspectors, who have some degree of discretion about which restaurants to prioritize. Boston is now implementing an algorithm using Yelp reviews to figure out which restaurants to prioritize.

The assignment of restaurant inspections includes predicting, on the basis of past history and other factors, where health problems might occur. Effective policing involves predicting—and preventing—crime. Successful forecasting and planning involve predicting economic conditions. Fostering a thriving school district involves predicting which teachers will add the most value to a classroom. All of these are prediction problems. And all of these can be improved through better predictive analytics in conjunction with a more deliberate approach to data collection.

Using online information to determine optimal disclosure policies

Policymakers should consider whether to increase, decrease, or maintain overall investment in public reporting. The answer relates to classic research on designing incentives in settings with multiple quality dimensions, and depends on whether the quality dimensions are substitutes or complements.[9]

To understand the difference, consider a public-health department that is focusing on two problems: rodents in the dining room and in the kitchen. Yelp creates an incentive for restaurants to remove rodents from the former. A restaurant might, hypothetically, respond by laying traps in the dining room, which would encourage rodents to live in the kitchen instead. In this case, the two problems are substitutes, and the increased social reporting on dining-room rodents might lead to more problems in the kitchen.

But if the rodents leave the restaurant because of dining-room traps, Yelp reporting that led to more dining-room traps would help with both problems; it would be a complement. When quality dimensions are substitutes—and there are numerous dimensions that customers cannot see or report on social media—policymakers should increase disclosure and enforcement efforts. Where quality dimensions are complements, and most areas of interest are readily observable to customers, policymakers could reduce disclosure and enforcement efforts. 

The principles that inform how social media may improve government regulatory regimes are not limited to restaurant inspections. They may be applicable to hospital and physician ratings, plumber and electrician licensing, landlord audits, and elsewhere. Overall, government disclosures, certifications, and inspection processes need to be updated to better reflect—and leverage—the complicated, information-rich, social media ecosystem.

1. See Alan Mathios, “The Impact of Mandatory Disclosure Laws on Product Choices: An Analysis of the Salad Dressing Market,” Journal of Law and Economics 43, no. 2 (2000): 651–77; and David Dranove and Ginger Jin, “Quality Disclosure and Certification: Theory and Practice,” Journal of Economic Literature 48, no. 4 (2010): 935–63.
2. See Ginger Jin and Phillip Leslie, “The Effect of Information on Product Quality: Evidence from Restaurant Hygiene Grade Cards,” Quarterly Journal of Economics 118, no. 2 (2003): 409–51; and idem, “Reputational Incentives for Restaurant Hygiene,” American Economic Journal: Microeconomics 1, no. 1 (2009): 236–67.
3. See Michael Luca, “Reviews, Reputation, and Revenue: The Case of Yelp.com,” Working Paper, 2011.
4. See SFDPH, Food Safety Program: Restaurant Safety Scores.
5. See Weijia Dai and Michael Luca, “Digitizing Disclosure: The Case of Restaurant Hygiene Scores,” Working Paper, 2016.
6. See Jun Seok Kang et al., “Where Not to Eat? Improving Public Policy by Predicting Hygiene Inspections Using Online Reviews,” Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013.
7. Cassandra Harrison et al., “Using Online Reviews by Restaurant Patrons to Identify Unreported Cases of Foodborne Illness—New York City, 2012–2013,” Centers for Disease Control and Prevention, Morbidity and Mortality Weekly Report 63, no. 20 (May 23, 2014).
8. See Edward Glaeser et al., “Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy,” American Economic Review 106, no. 5 (2016): 114–18.
9. See Bengt Holmström and Paul Milgrom, “Multitask Principal-Agent Analyses: Incentives Contracts, Asset Ownership, and Job Design,” Journal of Law, Economics and Organization 7 (1991): 24–52; and George Baker, “Incentive Contracts and Performance Measurement,” Journal of Political Economy 100, no. 3 (1992): 598–614.