Better data, safer roads
“About one-and-a-half years ago, we started thinking about how to get more information out of our data,” explains Thomas Mahlberg, a data engineer at Bosch. “We were looking at big volumes of data—it was all quite abstract. The data science team and the back-end operations team used different data storage and data analysis technologies. Evaluating a huge amount of files was time-consuming and analyzing the whole picture was difficult, as was getting an understanding of regional or time-based effects.”
At that time, Bosch was visualizing the data on a map by using a search engine platform. “It was working quite well—to get a first impression of what our data actually meant,” says Mahlberg. But when Bosch began incorporating data not only from Germany, but all of Europe, the system began to show its limitations. In addition, it required continuous maintenance efforts.
“We are currently developing a service to predict the road conditions facing the driving vehicle,” explains Stefan Manzer, Product Owner of the data science team at Bosch. “This allows automated driving vehicles to drive safely, but also provides additional information and warnings to non-automated vehicles. Our goal is to reduce the number of accidents that happen due to severe road and weather conditions like ice, water on the road, or fog.”
To run the kind of analysis Bosch was looking for, the company needed a query engine able to produce geographical aggregations—summarized by Geohash/S2 cells—from which metrics could then be calculated. “We needed this feature because our Europe implementation had roughly 860,000 forecasts per time slot,” explains Mahlberg. “But bringing a data load like that up on a map without aggregation was tantamount to killing the browser.”
Building on its trusted, long-term partnership with Microsoft, the Bosch team homed in on Azure Data Explorer. The changeover meant that Bosch could move away from a cost-intensive infrastructure as a service (IaaS)–based solution to a cost-effective native platform service. The change to the Kusto query language also paved the way for greater efficiency, being highly intuitive for both time series data and geospatial analytics.
“The initial system was set up within a week,” explains Mahlberg. “We saved data into the blob storage and from the blob storage we inserted it into ADX automatically. Bringing the data to ADX was simple.” Having used a REST API for running queries from the front end to the back end under the company’s former setup, Bosch replaced the connector to the previous platform with a connector to ADX, writing a few additional queries. “It was just replacing the data source—that works quite well and fast,” explains Mahlberg.
Soon, the Bosch team was reaping the benefits of more powerful analysis and more intuitive Kusto query language. “You can run the queries for much bigger timeframes,” says Mahlberg.
“Before we used ADX, we needed to think, ‘What's our use case?’ In-depth analyses for a few hours of joined sensor and weather data required data to be stored differently compared to analyzing a whole year of Europe-wide weather data on a high level. Now, we just put it in ADX and later we see what queries we will run on the data. We don't have to examine beforehand how to optimally store the data in order to query it.”
“It's dramatically changed how we work, making things that used to be complex, much simpler and quicker,” Manzer says. “As a product owner, it's become easy to just write queries and get basic information. And by using a managed service on Azure we could reduce our operations’ cost drastically, within a couple of months.” Visualizations are also possible directly within ADX. “You can do everything on the web UI, only more conveniently. It's a combination of transparent language and a really rich feature set.” Mahlberg agrees. “It’s easy for non-data scientists to get insight into the data and use their own tools to work with the data without having to set up a Jupyter Notebook or other data science–related tools.”
Stepping on the gas brings business acceleration
“The main benefit of ADX is development speed: queries and evaluations that took days, now take 10 minutes,” says Mahlberg. “Plus, you can handle great volumes of data quite easily. It’s really impressive to see the rate at which you can develop with ADX and the ease with which you can access the data and pull it out again in the right format. Non-data engineers can write a query and get the data they want in a few minutes; meanwhile, data scientists are able to run very complex queries in an equally short amount of time.”
Mahlberg and Manzer are already looking into ways to incorporate even more data into their forecast models, while taking the time to introduce the capabilities of Azure and ADX to other teams at Bosch. “A lot of people have been asking about Azure and ADX,” Mahlberg notes, "And I think we're providing them with extremely relevant feedback."
