Since dawn of iPhone, many of the smart functions of smartphones have come from elsewhere: corporate computers known as cloud. Mobile apps sent user data to the cloud for useful tasks like transcribing speeches or suggesting replies to messages. Now Apple and Google say smartphones are smart enough to do critical and sensitive things machine learning tasks like those on their own.
At Apple’s WWDC event this month, the company said its virtual assistant Siri will transcribe speech without using the cloud in some languages on recent and future iPhones and iPads. During its own Developer I / O event last month, Google said that the latest version of its Android The operating system has a feature dedicated to the secure handling of sensitive data on the device, called Private Compute Core. Its initial uses include powering the company’s version of the Smart Reply feature built into its mobile keypad that can suggest replies to incoming messages.
Apple and Google both say machine learning on the device offers more privacy and faster apps. Not transmitting personal data reduces the risk of exposure and saves time waiting for data to cross the Internet. At the same time, the retention of data on devices aligns with the long-term interest of tech giants in keeping consumers connected to their ecosystems. People who hear that their data can be treated more privately might become more willing to agree to share more data.
The recent promotion by companies of on-device machine learning comes after years of working on the technology to restrict the data their clouds can “see”.
In 2014, Google started collecting data on Chrome browser usage through a technique called differential privacy, which adds noise to the data collected so as to restrict what these samples reveal about individuals. Apple has used the technique on data collected from phones to inform emoji and typing predictions and for web browsing data.
More recently, the two companies have adopted a technology called federated learning. It allows a cloud-based machine learning system to be updated without recovering raw data; instead, individual devices process data locally and only share digested updates. As with differential privacy, companies have only discussed the use of federated learning in limited cases. Google used this technique to keep its mobile typing predictions up to date with language trends; Apple has published research on its use for update speech recognition models.
Rachel Cummings, an assistant professor at Columbia who was previously a privacy consultant for Apple, says the rapid shift to machine learning on phones has been striking. “It’s incredibly rare to see something go from first conception to full scale deployment in such a few years,” she says.
These advancements have required not only advancements in computing, but also enterprises to address the practical challenges of processing data on consumer-owned devices. Google said its federated learning system only taps users’ devices when they’re plugged in, inactive, and on a free internet connection. The technique was made possible in part by improving the power of mobile processors.
More powerful mobile hardware has also contributed to the 2019 Announcement that his virtual assistant’s voice recognition on Pixel devices would be entirely on the device, without the cloud kickstand. Apple’s new on-device voice recognition for Siri, announced at WWDC this month, will use the company’s “neural engine” added to its mobile processors to power machine learning algorithms.
The technical prowess is impressive. It is questionable to what extent they will significantly change the relationship of users with the tech giants.
Apple’s WWDC presenters said Siri’s new design was a “major privacy update” that addressed the risk associated with accidental streaming of audio to the cloud, saying it was the greater user concern about voice assistants. Some Siri commands, such as setting timers, can be recognized entirely locally, allowing for quick response. Yet in many cases, commands transcribed to Siri, possibly including from accidental recordings, will be sent to Apple’s servers for the software to decode and respond. Siri voice transcription will still be cloud-based for HomePod smart speakers commonly installed in bedrooms and kitchens, where accidental recording may be of more concern.
Google is also promoting the handling of data on the device as a privacy victory and has indicated that it will expand the practice. The company expects partners like Samsung who use its Android operating system to embrace the new Privacy Compute Core and use it for features that rely on sensitive data.
Google has also made local analysis of browsing data a feature of its proposal for reinventing online advertising targeting, called FLoC and claimed to be more private. Academics and some rival tech companies have said the design will likely help Google consolidate its dominance of online ads by making targeting more difficult for other companies.