top of page

ABOUT NORCOM

Operational data for product improvement

The task

The evaluation of operating data is relevant for companies for the further development and targeted improvement of products. The goals are the optimization of maintenance times and a return of knowledge for product improvement. The devices, which are distributed around the world, first transfer their data to the cloud, where initial evaluations are carried out. From there, they are transported to an on-premise big data environment for consolidation with existing data.

The data is analyzed interactively on pre-aggregated tables; the aggregations are updated at regular intervals using big data workflows.

 

The challenge

In addition to the huge, distributed amount of data, a particular challenge is the handling of personal data, which requires the flexible establishment of rules for storage, use and deletion.

 

our solution

The cloud data was merged with existing data and external sources (weather data, geodata) in a big data analytics environment via scalable data loading routes. There they were made available for BI analyzes. For this purpose, the data was aggregated in advance with big data analyzes and patterns extracted with advanced analytics. A tailor-made interactive analysis app was also implemented for special displays that could not be implemented with BI standard tools.

 

The customer benefit
The connection to Big Data can be used as a service by the entire BI team; the data is automatically updated daily to weekly. The established advanced analytics workflows with interactive visualization enable new data-supported insights into device usage. Virtualization and templating enable fast, efficient adaptation to growing amounts of data.

Project-

Characteristics

Our role

Support / training of the customer by data scientists, data engineers, software developers and architects

Our activities

  • Conception of the big data / database architecture

  • Setup and operation of a Hadoop environment

  • Setting up big data workflows for converting and analyzing data

  • Interactive analysis and visualization on mass data

Technologies & methods

  • Applications: DaSense, Powerbi, Qlik

  • Databases: Hbase, MySQL, Hive on Tez (LLAP), GIS, DarkSky

  • Languages / frameworks: Python (Anaconda Stack, bokeh), Java, Javascript, Hadoop / Hortonworks / Azure, Spark, Yarn, Oozie, Docker Swarm, Kerberos, Ranger, Xen, Overpass, Kylin, Hue

  • Methods: geodata, time series analysis, machine learning

bottom of page