ABOUT NORCOM
Analysis of globally distributed data in the vehicle endurance test
The task
In the vehicle endurance test, more and more measurement data are recorded in ever shorter periods of time; these must be evaluated promptly.
The challenge
Since the vehicles are in use around the world, it is becoming increasingly difficult to transfer the resulting measurement data to the headquarters for analysis.
our solution
With the “Distributed Query Engine” component, DaSense applies the Hadoop guiding principle “Bring the algorithm to the data” to a global network of data centers. Scalable data loading sections ensure that newly loaded data is automatically quality-checked, preprocessed and converted into a big data analysis format. Initial evaluations are available on decentralized measurement data stations within fixed time periods. By networking DaSense instances, the evaluations can be aggregated across the stations for global analysis.
In the background, a specially designed data mover is used to continuously transfer the data for backup in the central data storage in accordance with data governance rules.
The customer benefit
The immediate provision of data for evaluation means that test runs can be planned in a more agile manner and development costs are reduced. The intelligent data management ensures that analyzes are optimized with regard to data locality and that existing resources are used more efficiently.
PROJECT PROFILE
Our role
Customer support from data scientists, data engineers, software developers and architects
Our activities
Installation of several Hadoop environments with DaSense
Set up big data workflows to convert data
Creation and production of measurement data analyzes across the Hadoop environments
Technologies & methods
Applications: DaSense
Data / databases: Hbase, HDFS, MF4, Parquet, ORC, Avro
Languages / Frameworks: Python (Anaconda Stack), Java, Hadoop / Mapr / Hortonworks, Airflow, Spark, Yarn, Oozie, Nifi
Methods: time series analysis, outlier detection, machine learning