Streamlining Master Data Management with the power of Generative AI

At DevSparks 2024, held in Bengaluru, Abirami Sukumaran, Developer Advocate at Google, held a session on the capabilities of Generative AI models in understanding natural language, the need for precise and deterministic outputs in critical Master Data Management tasks. She addressed the issue of traditional generative models providing inconsistent responses, and how tools like Gemini 1.0 Pro, BigQuery, and Gemini Function Calling can help overcome these challenges, illustrated through a simple data management use case demonstrating the reliability of Gemini's AI solutions.

Here are some highlights from the workshop:

Understanding Master Data Management

Master Data Management (MDM) ensures a single, consistent view of entities within an organization, such as customers, products, and locations. This process involves aggregating data from various sources to create a golden record, representing the most accurate and complete version of the entity.

Imagine the scenario where multiple records of the same person exist under different names and locations. MDM's role is to recognise these disparate records as the same individual, consolidating them into a single, accurate profile.

The importance of accurate data

MDM is crucial for businesses that value personalisation. Entrepreneurs and startups need to understand their customers’ preferences and behaviors. For instance, when a frequent traveler books a hotel, they expect consistent service based on their past preferences—whether that is requesting fluffy pillows or a high-floor room with a scenic view. Without proper MDM, the hotel might not recognise the right individual, in case there are duplicates in the customer information, their preferences, resulting in a less personalised experience.

Challenges with traditional MDM

Traditional MDM methods involve laborious processes, including data replication, schema mapping, data cleaning, record matching, and data enrichment. These steps often require significant time and resources, leading to delays and inaccuracies. Despite extensive efforts, the accuracy of the resulting golden record might only reach 60-80%.

The role of Generative AI

Abirami Sukumaran introduced the transformative potential of Generative AI models, specifically Google's Gemini 1.0 Pro, in streamlining MDM processes. These AI models can simplify data management tasks, such as deduplication and enrichment, reducing the time and effort required to create accurate golden records.

How Generative AI improves MDM

Enhanced natural language understanding: Generative AI models excel at interpreting natural language, enabling more accurate data matching and merging.

Consistent outputs: AI-driven MDM solutions can provide precise and predictable results, crucial for critical business operations.

Addressing inconsistencies: Traditional generative models often produce inconsistent responses. Gemini 1.0 Pro, along with tools like BigQuery and Cloud Functions, can mitigate these issues by leveraging Function Calling, Embeddings and Vector search.

A practical example

At the session, Abirami Sukumaran gave the example of a city bike data management app for New York City. She explained how the app needed to provide users with real-time information about bike availability at various stations, ensuring no duplicate data exists. The data, sourced from BigQuery, required enrichment and deduplication to provide accurate, usable information. Using Generative AI, the enrichment step involved making API calls to enhance the raw data with complete address details and standardised formats. The deduplication process then used vector search to identify and flag duplicate records, ensuring a single accurate record for each bike station.

Conclusion

The integration of Generative AI into MDM processes revolutionises master data management, making it faster, more accurate, and less labour-intensive. By positioning AI models like Gemini 1.0 Pro closer to the data and leveraging advanced tools, businesses can achieve superior data quality with minimal coding effort. This evolution in MDM not only enhances operational efficiency but also empowers businesses to deliver personalised and consistent experiences to their customers.

You can catch all the details of this masterclass in the below detailed video,