top of page

ABOUT NORCOM

Analysis of globally distributed data in the vehicle endurance test

The task

In the vehicle endurance test, more and more measurement data are recorded in ever shorter periods of time; these must be evaluated promptly.

The challenge

Since the vehicles are in use around the world, it is becoming increasingly difficult to transfer the resulting measurement data to the headquarters for analysis.

our solution

With the “Distributed Query Engine” component, DaSense applies the Hadoop guiding principle “Bring the algorithm to the data” to a global network of data centers. Scalable data loading sections ensure that newly loaded data is automatically quality-checked, preprocessed and converted into a big data analysis format. Initial evaluations are available on decentralized measurement data stations within fixed time periods. By networking DaSense instances, the evaluations can be aggregated across the stations for global analysis.

In the background, a specially designed data mover is used to continuously transfer the data for backup in the central data storage in accordance with data governance rules.

The customer benefit
The immediate provision of data for evaluation means that test runs can be planned in a more agile manner and development costs are reduced. The intelligent data management ensures that analyzes are optimized with regard to data locality and that existing resources are used more efficiently.

PROJECT PROFILE

Our role

Customer support from data scientists, data engineers, software developers and architects

Our activities

  • Installation of several Hadoop environments with DaSense

  • Set up big data workflows to convert data

  • Creation and production of measurement data analyzes across the Hadoop environments

Technologies & methods

  • Applications: DaSense

  • Data / databases: Hbase, HDFS, MF4, Parquet, ORC, Avro

  • Languages / Frameworks: Python (Anaconda Stack), Java, Hadoop / Mapr / Hortonworks, Airflow, Spark, Yarn, Oozie, Nifi

  • Methods: time series analysis, outlier detection, machine learning

bottom of page