Workshop: Streamlining Agentic AI with Confluent and Databricks

Duration: ~1 hour

Difficulty: Intermediate

Technical Requirements: Working knowledge of cloud platforms (AWS or Azure), SQL, and basic command-line operations

Workshop Type: This workshop is designed to work in three modes: instructor-led, self-service, and demo.

📖 Overview

This hands-on workshop demonstrates how to build a complete real-time AI-powered marketing pipeline for the hospitality industry. You will play the role of a data engineer at River Hotels, a fictitious hospitality company, to create an end-to-end data architecture proof-of-concept that transforms raw customer interactions into personalized marketing campaigns using cutting-edge streaming technologies.

Watch this ~13 minute demo video to see the solution to a similar use case.

If you have any issues with or feedback for this workshop, Please let us know in this quick 2-minute survey!

🏨 Use Case

River Hotels is at a critical juncture. Despite being a successful hospitality company with properties across multiple continents, they're slowly losing ground to more agile competitors who can respond to market opportunities in real-time. The executive leadership team has identified a fundamental problem: their data infrastructure is holding back their ability to compete effectively in today's fast-paced hospitality market.

🗄 Datasets

There are five normalized interrelated datasets that you will be streaming to Confluent Cloud:

1. Customers: Master customer profiles containing contact information and demographics. These records serve as the foundation for customer behavior analysis across all other data streams.

2. Hotels: Comprehensive hotel property data including amenities, descriptions, locations, and capacity details.

3. Clickstream: Real-time website interaction events capturing customer browsing behavior, page views, and hotel searches.

4. Hotel Reviews: Customer feedback with ratings (1-5 stars) and detailed review text, linked to specific bookings.

5. Bookings: Reservation transaction data linking customers to hotels with check-in/check-out dates, pricing, and guest counts.

Expand the accordion below for more background details about this use case. Otherwise, continue on to the next section of this workshop.

Use Case Details

⚠️ The Challenge

River Hotels' current batch-processing approach means that by the time sales and marketing teams get insights about customer behavior, booking patterns, and market opportunities, those insights are already stale. Competitors could be capturing customers who browse River Hotels' website but don't book immediately, while River Hotels' marketing team is still waiting for last week's data to become available.

As the lead data engineer, you've been called into an urgent cross-departmental meeting where each team has laid out their critical business needs:

---

💰 The Sales Dilemma

"We're flying blind," says the Sales Director. "When a potential corporate client asks about our occupancy rates or customer satisfaction trends, I have to tell them I'll get back to them next week. By then, they've already signed with our competitor who had those numbers instantly available."

---

🚀 The Marketing Predicament

The Marketing Manager shares a similarly frustrating story: "Every week, we try to run a 'discount deal' campaign for an underperforming but highly-rated property. But our current process takes 2-3 days just to identify which hotel needs promotion and another 2 days to analyze, create, review, and publish messaging that resonates with potential guests. By the time we launch the campaign, it's been over a week and the opportunity is lost."

---

⚙️ The Engineering Conundrum

The Engineering Director is supportive but realistic: "Whatever solution we build needs to integrate with our existing PostgreSQL database infrastructure and can't require a massive operational overhead. We're already stretched thin, and we need something built on proven, enterprise-grade technology that our small team can actually maintain."

---

💼 The Business Imperative

The CEO has made it clear: "We're losing deals because we can't act on opportunities fast enough. Our competitors are using AI and real-time data to personalize customer experiences and optimize their operations. We need to catch up, and we need to do it quickly."

---

🎖️ Your Mission

Your task is to design and implement a proof-of-concept that transforms River Hotels from a data-lagging organization into a real-time, AI-powered competitor.

Solution Requirements

1. 📡 Capture customer behavior as it happens
2. ✨ Enrich data with meaningful insights
3. 🤖 Process intelligently with AI
4. 🔓 Unlock teams to act on insights immediately

🎯 Success Metrics

1. 💫 Data Freshness - Moving from week-old batch data to insights that are less than one hour old
2. ⏱️ Operational Efficiency - Reducing manual data analysis from days to minutes through enriched datasets and automation
3. 🏆 Competitive Advantage - Responding to market opportunities in real-time rather than after the fact

---

🛠️ What You'll Build

By the end of this workshop, you will have constructed a sophisticated data pipeline that:

1. Captures Real-Time Customer Behavior: Set up PostgreSQL CDC to capture customer and hotel data changes, plus generate realistic clickstream, booking, and review data using Java Datagen
2. Processes Streaming Data with AI: Use Confluent Cloud for Apache Flink SQL to denormalize bookings with temporal joins and enrich hotel reviews with aspect-based sentiment analysis (AI_SENTIMENT for cleanliness, amenities, and service)
3. Streams to Delta Lake: Leverage Confluent Tableflow to automatically sync processed data streams as Delta tables in AWS S3 or Azure ADLS2
4. Generates AI-Driven Insights: Use Databricks Genie to analyze booking patterns, customer preferences, and hotel performance metrics
5. Creates Personalized Campaigns: Deploy AI agents in Databricks that identify underperforming hotels with good customer satisfaction, generate targeted social media content based on customer review analysis, and create lists of potential customers for marketing outreach

🎓 Key Learning Outcomes

• Infrastructure as Code: Deploy complex multi-cloud resources (AWS or Azure, Confluent Cloud, Databricks) using Terraform
• Change Data Capture: Implement PostgreSQL CDC Connector for real-time database change streaming
• Stream Processing: Build sophisticated Flink SQL queries for real-time data enrichment and AI model integration
• Data Lake Integration: Use Tableflow to seamlessly bridge streaming data and analytics platforms
• AI-Powered Analytics: Apply generative AI for both data summarization and marketing content creation
• Event-Driven Architecture: Design systems that react to customer behavior in real-time

🔗 Data Entity Relationship

This diagram depicts how these datasets relate to each other:

erDiagram
    CUSTOMER {
        string CUSTOMER_ID PK
        string EMAIL UK
        string FIRST_NAME
        string LAST_NAME
        string BIRTH_DATE
        int REWARDS_POINTS
        long CREATED_AT
    }

    HOTEL {
        string HOTEL_ID PK
        string NAME
        string AMENITIES
        string DESCRIPTION
        string CITY
        string COUNTRY
        int ROOM_CAPACITY
        long CREATED_AT
    }

    BOOKING {
        string BOOKING_ID PK
        string CUSTOMER_EMAIL FK
        string HOTEL_ID FK
        long CHECK_IN
        long CHECK_OUT
        int OCCUPANTS
        int PRICE
        long CREATED_AT
    }

    CLICKSTREAM {
        string ACTIVITY_ID PK
        string CUSTOMER_EMAIL FK
        string HOTEL_ID FK
        string ACTION
        int EVENT_DURATION
        string URL
        long CREATED_AT
    }

    REVIEW {
        string REVIEW_ID PK
        string BOOKING_ID FK
        int REVIEW_RATING
        string REVIEW_TEXT
        long CREATED_AT
    }

    %% Relationships
    CUSTOMER ||--o{ BOOKING : "makes"
    CUSTOMER ||--o{ CLICKSTREAM : "generates"
    HOTEL ||--o{ BOOKING : "receives"
    HOTEL ||--o{ CLICKSTREAM : "viewed_in"
    BOOKING ||--o{ REVIEW : "has"

Loading

🧩 Key Components

1. Data Sources

   • Java Datagen: Realistic synthetic data generation of:
     • Customer and Hotel data, which is sent to an PostgreSQL database
     • Bookings, Reviews, and Clickstream events, which are all produced to Kafka topics

2. Ingestion Layer

   • PostgreSQL CDC Connector: Real-time change data capture from PostgreSQL
   • Kafka Producers: Stream synthetic data directly from Java Datagen to Confluent Cloud topics

3. Processing Layer

   • Apache Flink SQL: Real-time stream processing and data enrichment
   • (Optional) AWS Bedrock Integration: AI-powered review summarization using Claude models
   • Stream Analytics: Identification of high-value prospects and customer behavior analysis

4. Integration Layer

   • Confluent Tableflow: Automated streaming data to Delta Lake format
   • AWS S3: Delta Lake storage for processed data streams

5. Analytics Layer

   • Databricks SQL: Advanced analytics and querying capabilities
   • Databricks Genie: Natural language interface for business intelligence
   • AI Agents: Intelligent hotel selection, review analysis, and customer targeting for automated marketing campaigns

🛠️ Technical Stack

Core Technologies

• Terraform: Infrastructure as Code for multi-cloud deployment
• Apache Kafka: Distributed streaming platform via Confluent Cloud
• Apache Flink: Stream processing and real-time analytics
• Delta Lake: Open-source storage framework for data lakes

Cloud Platforms

• Confluent Cloud: Fully managed Apache Kafka service
• AWS / Azure: Cloud providers for compute, networking, and Delta Lake storage (S3 or ADLS Gen2)
• Databricks: Unified analytics platform for big data and ML

AI/ML Services

• Databricks Genie: Natural language interface for analytics
• Databricks Mosaic AI Models: Large, powerful LLMs that can be utilized in custom agents

Tools

• Docker: Containerization for PostgreSQL database and Java Datagen
• Git: Version control
• AWS CLI: AWS command-line interface
• Java Datagen: Realistic synthetic data generation via the workshop's custom Java application (this workshop previously referenced a third-party hosted data generator; that component has been replaced in-repo).

🔬 Workshop Labs

This workshop supports three modes. Choose the path that matches your situation:

🎓 Instructor-Led

This mode is hands-on with the Confluent Cloud and Databricks products to build out an AI marketing agent, but you will use cloud accounts and infrastructure that will have already been pre-provisioned for you.

Use it only when instructed to by your workshop instructor/leader.

Lab	Duration	Details
LAB 1: Claim Your Account	~5 min	Claim your workshop account: complete the Google Form, receive credentials, verify access to Confluent Cloud and Databricks.
LAB 2: Explore Your Environment	~10 min	Tour your environment: explore your Kafka cluster, CDC topics, connectors, Flink compute pool, and Databricks workspace.
LAB 3: Stream Processing	~15 min	Transform streams: build Flink SQL queries with temporal joins, denormalize booking data with CDC dimensions, enrich reviews with AI aspect-based sentiment analysis.
LAB 4: Tableflow	~10 min	Configure catalog and enable Tableflow: connect Confluent Cloud Tableflow with Databricks Unity Catalog, stream clickstream, denormalized bookings, and sentiment-enriched reviews as Delta Lake tables.
LAB 5: Stream Lineage	~10 min	Visualize data flow: explore Stream Lineage to trace data through CDC, Flink, and Tableflow.
LAB 6: Analytics & AI	~25 min	Generate insights: explore hotel performance and sentiment analytics, use Databricks Genie, deploy AI agent for personalized marketing automation.
LAB 7: Wrap Up	~5 min	Clean up and recap: Review accomplishments, provide feedback.
Optional: Data Governance	~10 min	Explore data quality rules: observe pre-deployed CEL rules, live DQR demo with /test-dqr, governance tags and business metadata.

🛠️ Self-Service

This mode is very hands-on, where you will set up your own cloud accounts, deploy infrastructure with Terraform, and execute some steps manually to build out the AI marketing pipeline.

Use it to become more familiar and learn how you could do something similar within Confluent Cloud and Databricks.

Lab	Duration	Details
LAB 0: Prerequisites	~10 min	Set up prerequisites: create cloud accounts, install Git and Docker, clone the repo, build Docker images.
LAB 1: Account Setup	~15 min	Configure cloud platform accounts: set up Confluent Cloud API keys, configure Databricks service principal, establish AWS credentials.
LAB 2: Cloud Infrastructure	~15 min	Deploy infrastructure with Terraform: provision AWS, Confluent Cloud, and Databricks resources. Verify data generation and CDC connector.
LAB 3: Tableflow & Unity Catalog	~15 min	Configure Tableflow: connect Tableflow with Unity Catalog, enable Tableflow on clickstream topic.
LAB 4: Stream Processing	~15 min	Transform streams: build Flink SQL queries with temporal joins on pre-configured CDC topics, configure Tableflow sync.
LAB 5: Stream Lineage	~10 min	Visualize data flow: explore Stream Lineage to trace data through CDC, Flink, and Tableflow.
LAB 6: Analytics & AI	~25 min	Generate insights: explore hotel performance and sentiment analytics, use Databricks Genie, deploy AI agent for personalized marketing automation.
LAB 7: Cleanup	~5 min	Clean up resources: remove UI-created resources and terraform destroy the remainder.
Optional: Data Governance	~15 min	Explore data quality rules: observe pre-deployed CEL rules and DLQ routing, governance tags, business metadata, create your own rule.

🚀 Demo

This builds on the self-service mode by automating almost all of the manual steps that enable the real-time AI marketing pipeline. This demo mode can be used for short-term, long-term, and even "always-on" demos.

Use it to show quick and immediate value with minimal in-product (Confluent Cloud, Databricks) set up.

Demo Terraform roots: terraform/aws-demo (AWS) and terraform/azure-demo (Azure). For automated multi-account provisioning, use wsa-spec-aws-demo.yaml or wsa-spec-azure-demo.yaml with WSA.

Azure note: AI_SENTIMENT / reviews_with_sentiment and the hotel_performance view are AWS-only today. Azure demo still provisions denormalized bookings, clickstream Tableflow, Unity Catalog integration, and the marketing notebook.

Lab	Duration	Details
LAB 0: Prerequisites	~10 min	Set up prerequisites: create cloud accounts, install Git and Docker, clone the repo, build Docker images (AWS or Azure).
LAB 1: Account Setup	~15 min	Configure cloud platform accounts: set up Confluent Cloud API keys, configure Databricks service principal, establish AWS or Azure credentials.
LAB 2: Deploy and Observe	~25 min	Deploy everything with Terraform: one terraform apply provisions cloud infra, Confluent Cloud, Flink Materialized Tables, Tableflow, Unity Catalog integration, and Databricks notebook. Guided tour of the pipeline.
LAB 3: Analytics & AI	~30 min	Generate insights: explore pre-created analytics (sentiment / hotel_performance on AWS), use Databricks Genie, run the pre-imported marketing agent notebook.
LAB 4: Cleanup	~5 min	Clean up resources: terraform destroy handles everything including Tableflow.
Optional: Data Governance	~10 min	Demonstrate data quality rules: observe pre-deployed CEL rules, live DQR demo with /test-dqr on AWS EC2, DLQ observation.

Additional Resources

• Recap: Summary of accomplishments and business value delivered
• Troubleshooting: Common issues and solutions
• Stream Processing Insights: Detailed guide for streaming join patterns and schema management

🏁 Conclusion

Congratulations, you have completed this hands-on workshop on creating a streaming AI agent on AWS or Azure with Confluent and Databricks!

Important

Your Feedback Helps!

Please help us improve this workshop by leaving your feedback in this quick 2-minute survey!

Thanks!