Big Data is transforming government and civic activism in New York
New York City has been a Big Data pioneer for decades. In the early 1990s, the city launched the CompStat data-driven policing system, so that, in the words of former NYPD chief Lou Anemone, officers could stop “just running around answering 911 calls” and start analyzing patterns to prevent crime. Thanks in part to CompStat, major crimes in the city have since fallen by 80 percent.
“In the 21st century, cities are getting the "networked revolution" of large-scale data collection, both human and automated...”
During the Michael Bloomberg mayoral years, the city used data to pinpoint dangerous intersections and driving habits, cutting traffic deaths by nearly a third. Today, thanks to advances in data-storage capacity as well as the ubiquity of smartphones and broadband access, New York has an unprecedented number of facts to analyze and act upon, CompStat-style, across all areas of government — from building inspection to noise reduction.
But while it can shine a brighter light on problems and give citizens and government new tools with which to understand them, Big Data can’t solve the problems themselves. For that, we still need old-fashioned political will.
Cities are in the middle of what Daniel Doctoroff, a Bloomberg-era deputy mayor, calls their fourth modern revolution.
In the 18th century, cities got the first: the steam engine, which made possible the industrial age. In the 19th century, electricity gave cities the subways and elevators they needed to fit more people into small spaces. In the 20th century, the automobile made it easier for residents and workers to leave dense urban areas. And now, in the 21st century, cities are getting the “networked revolution” of large-scale data collection, both human and automated, as well as continuous connectivity to transmit and store those data.
And with better transparency laws, much of this Big Data becomes open data: information that everyone can see and use. Consider how one area of data reporting and collection has exploded in just a decade: New Yorkers’ complaints, questions and observations about their city.
Mobilizing hackers
Last year, more than 30 million people called or went online to 311, the city’s information and complaint system. That’s nearly four times the city’s population, and four times the number of people who called in 2003, the first year New York offered the service. Every call or click makes a point of data for the city or for outside observers — who can easily access much of this information — to follow.
During the Bloomberg years, the city began using these data to figure out, CompStat-style, where and when to send officials out — not to prevent crimes but to address other issues.
“New York’s worst natural disaster in recent memory — Superstorm Sandy — spurred the city not only to use data more aggressively but also to improve the quality of those data.”
Mike Flowers served as New York’s first data-analytics chief. One basic problem of even highly functional cities, Mr. Flowers notes, is “too big a body, too little blanket,” and the traditional answer to that problem, he says, is “let me hire more people.” Instead, Mr. Flowers used data to help him perform critical tasks, such as protect New Yorkers from firetraps.
“We were getting 20,000 complaints a year,” he says, many via 311, “about illegal [apartment] conversions” — that is, landlords or tenants subdividing their properties, creating dangerous conditions for poor tenants. But building inspectors, “treat[ing] the Empire State Building the same as a small apartment building,” regarded all buildings equally, deploying resources unwisely.
Working with the fire department and buildings department to analyze historical information, Mr. Flowers’ team discovered which buildings were more likely to have critical violations resulting in destruction and death — and made it a priority to inspect those buildings. Now, every building gets a risk grade and is inspected accordingly. Fire deaths and injuries have never been so low.
New York’s worst natural disaster in recent memory — Superstorm Sandy — spurred the city not only to use data more aggressively but also to improve the quality of those data.
Noel Hidalgo heads BetaNYC, a civic “hacking” organization that relies on open data to try to help government perform better. Mr. Hidalgo says that, after Sandy in 2012, the Bloomberg administration grasped how bad some of the city’s information was: Locations had multiple different addresses, for instance, and the buildings department had no idea what the fire department knew about a particular building. The administration then made sure that every location in the city has a single, consistent address, now available to the public.
Half a decade ago, the Metropolitan Transportation Authority started making data feeds available for public use. Private-sector entrepreneurs have drawn on them to create dozens of apps to help New Yorkers do everything from find out when the next bus is coming to learn about a piece of artwork in a particular train station.
The MTA is exploring other applications; it held a “hackathon” in early March so that volunteers vying for $2,000 in prizes could crunch data to help the authority speed up bus times.
Open data can be deployed not only to improve government but also to help business owners cope with the government agencies that regulate them. Aileen Gemma Smith, a local entrepreneur, saw that, after Sandy, small-business owners wanted to know such things as when a certain street would reopen. She noted that this lack of information was chronic even in normal times. Business owners showing up for work would be surprised that the city had closed their street for repairs—meaning lost customers and thus lost revenue.
Ms. Smith then launched an app called Mind My Business, which crawls through hundreds of city data sets to provide practical information: “The MTA is closing the subway stop near your store this weekend,” or “the city is repairing the sidewalk that goes by your shop this week,” or “the previous owner of the restaurant space you own got fined four times for the following violations.”
“Data aren’t just for privileged folks doing research,” Ms. Smith says. “Open data is how I help the local bodega guy, how I help the diner that’s been there for 25 years.”
“More data can make more people aware that the city has a problem — whether with slow ambulance-response times or a rising homeless population.”
Amateurs can mine Big Data to improve the quality of life in the city, too. Bicyclist Paul Vogel got a camera for his bike a few years back because he “had a couple of bad run-ins” with reckless car and truck drivers, “and I’m really bad at remembering license plates.” When he gets home after a ride, he sends pictures of taxis and other for-hire cars whose drivers have violated laws to 311. “I was surprised that the 311 system for [taxi complaints] is so efficient . . . that I could get someone fined,” he says.
Over a year or so, his self-described “hobby” has earned the city about $30,000 in revenue. More important, he may have saved some lives by deterring drivers he got fined from parking in bike lanes or running red lights again.
Wanted: information
One Big Data problem is that data is not available unless information is made public, and freedom of information laws are far from perfect. Government agencies sometimes charge onerous fees for documents, and they often interpret their mandate to protect privacy too broadly, forcing petitioners to spend money and time suing for what they should get for free. But the laws have given a generation of civic researchers a valuable new tool.
In the 1990s, Steven Romalewski, a veteran of data collection from his days with the New York Public Interest Research Group, used information gleaned via the laws to map toxic-waste sites so people would know “here’s how close they are to parks, here’s how close they are to schools.” And in the city, during the Rudolph Giuliani years, Mr. Romalewski fought to get data on children whom the city had tested for lead-poisoning exposure. After Mr. Bloomberg became mayor in 2002, the city released the data, which Mr. Romalwski tested by zip code and displayed by council district on a map.
Traffic-safety advocates have similarly been fighting for traffic-crash data and presenting them to the public for years. And some city agencies—the department of planning, for one — have long had websites featuring sophisticated, accessible data sets on zoning and population
During the Bloomberg years, civic-minded citizens got a legal update for the connectivity age: the 2012 open-data law, which requires the city to make certain data sets available on a free online portal. Today, city agencies offer 1,400 data sets to download and examine, with no tools needed besides spreadsheet software — now free, thanks to Google — and some patience. If you’re up late, upset about a barking dog, you can find out within an hour or so how many other people have had the same complaint over the last six years and where they live — DIY data collection and analysis that were unfathomable even 15 years ago.
Those responsible for providing the data, however, often resist its dissemination. The issue isn’t cost, which is minimal, but accountability. “Data can equal embarrassment,” says John Kaehny of the city’s Transparency Working Group. More data can make more people aware that the city has a problem — whether with slow ambulance-response times or a rising homeless population.
“Despite many advances, we still need more data — and better data — to improve New York’s quality of life.”
So we still need freedom-of-information laws, as well as journalists and activists to ask for data that the city hasn’t yet collected or won’t give out because no one has asked for them. At some point, though, Big Data and open data run into an old-fashioned problem: The city knows what the information suggests that it should do — but it won’t do it.
For instance, residents of the lower Hudson waterfront for years have suffered incessant noise from tourist helicopters. They collected flight and noise data, showed them to the city and inundated officials with complaining emails, but the only relief they could get was a reduction in flights to every two minutes from every one minute.
Despite many advances, we still need more data — and better data — to improve New York’s quality of life. Take a common scenario: A resident calls to complain that construction plates over underground work in the street aren’t secured properly. The website says that inspectors found nothing wrong. “You wonder, did they ever inspect it? You don’t know,” says Mr. Kaehny. The city should let the public see specific details of government agencies’ responses to such complaints.
The future of data, though, is not only 311 calls but also sensors. This year, a private vendor is erecting free Wi-Fi kiosks around the city to let New Yorkers access high-speed Internet as they walk along the street. The city could work with the vendor to outfit these kiosks with sensors to monitor everything from noise-pollution levels to carbon monoxide. High levels of carbon monoxide could direct traffic officers to a congested street with double-parked trucks.
The city could also combine human reporting with sensor enforcement, forcing a building site with a long history of noise violations, say, to deploy a permanent sensor at its site. The city could learn when a municipal garbage can is full and needs emptying, or count how many people walk through a particular intersection at particular times or count how many people, precisely, get into a subway car — and use that information to provide better services.
To fully capitalize on Big Data’s potential, however, we’ll need more and smarter investments in the assets we inherited from our second modern urban revolution — electricity.
This piece originally appeared in Pittsburgh Post-Gazette, adapted from the Spring 2016 Issue of City Journal
This piece originally appeared in Pittsburgh Post-Gazette