GEMA – Central process control with Camunda
GEMA orchestrates more than 580 cross-team processes with millions of records centrally using Camunda. cimt developed the process control and supported the migration to Google Cloud.
GEMA – New data strategy with Databricks
Customer
GEMA
Industry: Media
Headquarters: Berlin, Munich
Your contact
Oliver FrommMember of the Executive Board, cimt Group
Consulting focus
- Databricks
- Delta Lake
- Lakehouse
- Unity Catalog
- Google BigQuery
- Tableau
Measurable benefits
- Massive time savings
- Higher efficiency through automation
- Reduced costs
- Improved data quality & transparency
- More flexibility & self-service
About GEMA
GEMA is the association of composers, lyricists and music publishers as the authors of musical works. It manages the rights of music creators in Germany and of more than two million further rights holders worldwide, ensuring that authors and music publishers receive a fair share of the revenue generated from the use of their works.
In a world where data-driven decisions have become a decisive competitive advantage, an efficient and scalable data processing workflow is essential. Yet in many organisations, legacy structures, manual checks and inflexible workflows are still a reality.
The GEMA project shows how the targeted use of Databricks within a lakehouse architecture turned a previously fragmented setup of SQL, Excel and manual reconciliation into a modern data platform with a high degree of automation. As a result, numerous business units now receive their data with high reliability, quality and timeliness – while costs have fallen and transparency has increased.
The starting point: reporting with many manual steps
Initially, all data management processes at GEMA relied heavily on manual work and numerous individual steps. Data was checked exclusively in Excel spreadsheets used by the finance department to control, validate and release the data for further processing.
A first step towards simplification was the introduction of Tableau. For this, an SQL statement was developed that imported the data into Google BigQuery and aggregated it there. Tableau was then used to visualise and analyse the data.
Although this approach worked, it created additional manual effort. Every update required changes to the scripts and a manual load of the results. This increased the risk of errors, which ultimately cost time and resources.
The turning point: introducing Databricks
The decision to introduce Databricks as the central data management platform marked a fundamental change of direction. The key challenge was that the relevant data was spread across different systems and formats. Databricks consolidated and organised all data in a single, unified store using Delta Lake. The lakehouse architecture was chosen to build a modern platform that optimally supports big data file formats, object storage and lean data pipelines. This went hand in hand with the principle of step-by-step data refinement and data design in a medallion architecture, as proven in lakehouses built on Databricks.
In addition to the automated data pipelines, users can upload their own files and build individual ETL pipelines for their analyses from the entire data pool. Consolidating the data was very important to the customer, but so was enabling users to analyse their data themselves. Self-service was therefore considered from the outset and was a key principle behind the decision. A further goal was to automate existing processes, make them more scalable and migrate them to a modern cloud infrastructure – a plan that was successfully implemented with Databricks\u2019 built-in capabilities.
The most important steps:
1. Migration from SQL to PySpark
The core SQL statements were fully migrated to PySpark, the open-source engine natively supported by Databricks. This made the code much more flexible, easier to maintain and simpler to embed in pipelines.
2. Optimisation of the loading process
Initially, the jobs ran on an all-purpose cluster, which was sufficient for first tests. In later project phases, the jobs were moved to job clusters that start, run and stop automatically – a measure that noticeably reduced costs.
3. Expansion to a multi-pipeline structure
Each business unit had its own requirements, whether reporting, planning or controlling. As a result, a single initial SQL script evolved into a structure of five dedicated ETL pipelines that business units now use to load up-to-date data at the push of a button.
4. Use of Unity Catalog
Data from Excel or CSV files can also be used without difficulty: Unity Catalog integrates it into Databricks, where it is processed with the built-in functions and fed directly into existing pipelines.
Customer benefit: automation, efficiency and future-proofing
The transformation delivered not only technological but also economic benefits:
Automation instead of manual work: Manual data updates are no longer needed – automated pipelines make current, validated data available at any time. This frees up resources, above all working time, because fewer manual tasks are required and decisions can be made faster.
Cost efficiency through dynamic scaling: Job clusters ensure that compute resources are only requested when needed – a key advantage over permanently running systems. This reduced ongoing infrastructure costs.
Integrated processing of external data sources: Structured data such as Excel or CSV files that are not stored in classic databases can be organised in the unified metadata catalog and integrated and analysed directly in Databricks. This shortens analysis times, increases productivity and enables faster, well-founded business decisions.
The result: a future-proof, cloud-based data ecosystem that can be flexibly adapted and scaled – without any manual checks.
Conclusion: Databricks as a catalyst for data-driven organisations
What started as a manually maintained SQL script has evolved, with a data lakehouse and Databricks, into a modern, maintainable and automated data landscape. What previously took around two to three days of processing and countless rounds of coordination to obtain the data is now available in less than two hours – without any additional coordination.
The move from isolated tools to an integrated cloud platform not only enabled higher data quality and transparency, but also created a sustainable foundation for future requirements, such as AI use cases, machine learning, real-time analytics or self-service BI.
The project was supported by cimt, which contributed extensive experience in designing and implementing modern data platforms. Together with the customer, the team developed a future-proof solution that is not only technically convincing but also adapts flexibly to growing requirements. Particular emphasis was placed on ensuring that the platform creates long-term value and noticeably simplifies the customer\u2019s day-to-day work.
References
More business cases
Panasonic Europe – Scalable approval processes with Camunda and DMN
Panasonic Europe digitalises hundreds of approval processes with one generic BPMN model and DMN decision tables in Camunda; cimt is responsible for the methodological design and supports the implementation.
Sievert SE – A data foundation instead of data silos
Data silos and declining trust in KPIs shaped the situation at Sievert SE. cimt built a cloud data warehouse for controlling and sales with Talend, Snowflake and Power BI.