About

The Client is a large jewelry manufacturer and retailer that distributes its products in online and offline stores across the US.

Challenge

The Client had a legacy Informatica solution for data analytics, which started showing subpar performance as the companys business grew. As the legacy solution was unable to handle the growing data volumes, the Client initiated in-house development of a new Incorta-based enterprise data warehouse. The new DWH was to enable enterprise-wide analytics of the data coming from the companys business-critical systems (e.g., CRM, ERP, SCM) to facilitate informed decision-making for the companys management.

When implementing the Incorta Spark layer, the Client faced the lack of big data skills in its in-house team. The team members needed expert guidance to boost the rebuilding of legacy ETL processes on the new Incorta platform. The project had strategic importance for the Client as each ETL pipeline migrated to Incorta allowed the company to run tens to hundreds of reports much faster than in the legacy system. To speed up the project and ensure full reliability of the new ETL pipelines, the Client started looking for a reliable big data consultant.

Solution

Trusting VolgoTechnologies nine years in big data services and a solid portfolio of successful big data projects, the Client chose us as the consultant for the project.

VolgoTechnologies senior data engineer conducted an in-depth analysis of the solution under development. Also, he interviewed the Clients team about the difficulties they faced when rewriting the ETL business logic for the new solution. He discovered that the developers were highly proficient in SQL but lacked Python and Spark skills, which was slowing down the project pace.

Staging

Data Ware House

Data Ware House

Desktop Application

Results

By reaching out to VolgoTechnologies, the Client significantly sped up the delivery of the new ETL pipelines for its enterprise data warehouse. Thanks to the expert guidance and knowledge transfer from VolgoTechnologies big data consultant, the Clients in-house developers are showing significant improvement in their Python and Spark skills and confidently rewriting the business logic of ETL pipelines for the new EDW solution.

Technologies and Tools

Development: Incorta, Python, Apache Spark, PySpark, SQL, Bash.