GEMA – Central process control with Camunda
GEMA orchestrates more than 580 cross-team processes with millions of records centrally using Camunda. cimt developed the process control and supported the migration to Google Cloud.
Xing SE Big data integration with Talend and Exasol
Xing SE
Customer
XING SE
Industry: Social Media
Headquarters: Hamburg
Your contact
Rouven HomannMember of the Executive Board, cimt Group
Consulting topics
Results
- Processing up to 50 million events per day for the marketing automation application “Braze”
- Data available within 10 minutes of its creation
- Easier GDPR compliance for more than 15 million users who entrust their personal data to XING
Better system connectivity for better business networking
With more than 15 million members and offices in Hamburg, Munich, Barcelona, Vienna, Zurich and Porto, XING is the leading online business network for professional contacts in the German-speaking region. 90 percent of XING page views come from the DACH region (Germany, Austria and Switzerland). Professionals from all industries use XING to connect with each other and to look for jobs, projects, cooperation partners, expert opinions and business ideas. XING members interact in around 90,000 groups or meet in person at one of more than 130,000 networking events per year. The company was founded in Hamburg in 2003, has been listed on the stock exchange since 2006 and has been part of the TecDAX since September 2011.
Processing large data volumes in a time-critical event streaming environment
XING offers a decisive advantage in providing local news and job information. As XING’s services are based on collecting, storing and processing high-quality, up-to-date data, the company depends on efficiently integrating data from a wide variety of sources into its own systems. “The systems we used before forced us to write scripts for data integration. It was very difficult to keep track of where data sets were generated,” says Mustafa Engin Sözer, Senior Business Intelligence & Big Data Architect at XING. “We knew we wanted to do this without scripting wherever possible, centralise our metadata and simplify data integration. We were also looking for ways to improve data quality.”
Sözer also describes further data challenges XING was facing: “Some data sources deliver a large number of small files that could not be processed individually for technical reasons. So we were looking for an efficient way to process this data and make it available in the central data warehouse within 10 minutes of its creation.” Sözer adds that the file formats were also a problem: the data was in Apache Avro format, which conventional data processing tools usually do not support. XING therefore needed a solution that works with a wide range of file formats and supports interoperability within the data processing toolchain. For example, Apache Drill is used to move big data workloads onto big data clusters, followed by ELT-style data processing. While Avro files can be processed in near real time with Apache Spark, maintaining Spark jobs requires programming skills and a basic understanding of distributed computing – something less technical business developers may not have. XING therefore wanted to reduce the complexity of its integration processes so that stakeholders and developers with a less technical background could manage the integrations.
“Data analysis is the key to the success of an online network. With Talend, we can find the signals in our data that support our decision-making in real time – and use them to take our service quality to a new level.”
MUSTAFA ENGIN SÖZER
Senior Business Intelligence and Big Data Architect, XING SE
Why Talend
XING evaluated several solutions for its data integration needs and chose Talend for a number of reasons. “The features that convinced us about Talend include its open source approach, the wide range of connectors, the fact that it is Java-based and its familiar, easy-to-use Eclipse-based interface,” says Sözer. “We also liked the metadata management and automated documentation features, the fast adoption of new technologies and the ability to implement new use cases.”
Today, XING uses Talend as a bridge between an on-premises 150 TB MapR-DB NoSQL database and a 60 TB Exasol database used for analytics. “We set up Apache Drill with Talend on MapR to build a data processing pipeline that converts a huge number of Avro files into CSV format,” explains Sözer. “This means we only have to write a single query to read thousands of files, which drastically reduces the processing effort.” Using Talend for this specific use case also allows XING to keep track of metadata and gain a global view of all data assets – something that would not have been easy with a regular Spark job.
Sözer adds that Talend also helps with dynamic data extraction. With Talend, XING can easily retrieve specific columns from the data in the MapR file system, apply business or technical transformations and then store them in Exasol, where MicroStrategy uses them to generate reports. This also makes it possible to adapt data extraction to changes and ensures that only relevant columns are moved to Exasol, while all raw data can remain in the MapR file system or database for further processing, with the focus on batch analytics. “Talend helps us simplify the entire integration. Data processing becomes easier to understand and trace – with consistently high performance,” says Sözer. “This makes it easier for us to involve people with a less technical background in the process.”
Why cimt
“Working with the Talend experts at cimt made the migration from the previous solutions to Talend, and the raising of standards and the framework required for a healthy Talend ETL environment, a smooth process,” says Sözer. Rouven Homann, Member of the Executive Board, cimt Group, adds: “Our combined experience and knowledge of both technologies, Talend and Exasol, enabled us to develop a custom Exasol SCD component for XING (tExaSCDELT) based on ELT for Talend. We are very pleased that XING was willing to make this component available to the Talend open source community.”
Connecting professionals, increasing productivity and success
The driving force behind integrating many different systems and centralising data processing was the goal of analysing marketing data in near real time and delivering meaningful insights into campaign effectiveness. The real-time marketing response data provided by the marketing automation vendor Braze consists of thousands of small Avro files that are temporarily stored on Amazon S3. “We use Talend to connect systems and deliver the data to the central data warehouse as quickly as possible, so that it can be used to create analytical reports, support operational marketing and targeted campaigns, and monitor campaign performance,” says Sözer.
The processing capabilities are impressive. According to Sözer, this approach enables XING to process 2 million events per hour – up to 50 million events per day at peak times. Key benefits of XING’s new integration architecture include better business transparency, as data is consolidated on a single platform, and the ability to run analyses and reports more efficiently and thus optimise decision-making across the company. “In addition,” says Sözer, “maintenance costs have fallen while productivity and efficiency have increased.”
“With Talend, we gain insights and can measure our performance against key figures,” explains Sözer. “For example, we can now analyse data faster and more accurately and extract metrics and KPIs that are used across XING to implement business strategies. We also have better statistics on the number of daily and weekly active users, new job offers, the number of users who clicked on specific offers, and more.”
More than 14 million users entrust their personal data to XING. The company therefore has a special responsibility towards its customers, who all expect the social network to keep their data secure and treat sensitive information confidentially. Talend also helps XING comply with strict standards for corporate governance, data protection and the GDPR. “Online business networking is based on trust,” says Sözer. “That is why storing and tracking metadata centrally is crucial for compliance. With Talend, we centralise all source and target systems. This allows us to analyse data and find out which data sets are relevant for which requirement. We can also see whether data is personal or not. And we take full control of our data and metadata.”
References
More business cases
Panasonic Europe – Scalable approval processes with Camunda and DMN
Panasonic Europe digitalises hundreds of approval processes with one generic BPMN model and DMN decision tables in Camunda; cimt is responsible for the methodological design and supports the implementation.
GEMA – New data strategy with Databricks
GEMA used SQL, Excel and manual reconciliation for its reporting. cimt supported the build-up of a lakehouse platform with Databricks, through which business units now receive their data automatically – in under two hours instead of two to three days.