DZone DevOps and CI/CD Zone

Scaling Teams, Scaling Systems: Unlocking Developer Productivity With Platform Engineering

Ammar Husain — Tue, 14 Jul 2026 15:00:00 GMT

Modern software delivery is complex. Developers are responsible not only for writing code that meets business requirements — both functional and non-functional — but also for navigating a long chain of supporting steps. From containerization, testing, configuration, security, deployment, and monitoring, each stage often relies on specialized tools and teams.

When these processes aren’t standardized, every project risks reinventing the wheel. The result is inconsistency, delays, and frustration. For example, requesting a new test environment might require submitting detailed tickets to a DevOps team, slowing timelines and draining energy. As organizations scale, so does the complexity — and the pain of delivery.

Building Reliable Async Processing Pipelines Using Temporal

Akhil Madineni — Mon, 13 Jul 2026 13:00:06 GMT

Asynchronous processing pipelines are a cornerstone of modern distributed systems, but wiring them together reliably can be complex. A typical pipeline built with queues or message brokers requires custom retry logic, dead-letter queues, cron recovery jobs, and database status flags to ensure every step eventually succeeds.

Temporal replaces this heavy plumbing with durable workflows. In a Temporal workflow, the business logic of the pipeline is written as ordinary sequential code, yet it executes reliably across failures. The platform persists every state transition and step so that if a worker crashes or a network blip occurs, execution resumes exactly where it left off.

Candidate Generation Decides Your Pipeline's Cost, Not the LLM

Deepak Gupta — Thu, 09 Jul 2026 18:00:00 GMT

When the Most Capable Model Is the Wrong Starting Point

The fastest way to exceed a document pipeline budget is to let an LLM inspect every document before you have performed lightweight filtering. This sounds obvious, but the bottleneck is invisible at the prototype stage. A single model call is cheap, and it works well on the 20 documents in your test set. Then you hit production traffic.

The failure mode is usually pretty similar across teams: tens of thousands of LLM calls per day, tens of millions of tokens, and a monthly bill that drifts past the assigned budget. No candidate generation. No triage. Raw corpus straight to the model. The cost compounds because the corpus does not shrink without an upstream triage. A more capable model just gives you a more expensive way to process noise.

From Bash Script to Operational Triage: What Eight Months of Kubernetes Debugging Taught Me

Shamsher Khan — Thu, 09 Jul 2026 15:00:06 GMT

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed.

Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet.

Getting Started With RabbitMQ in Spring Boot

Gunter Rotsaert — Wed, 08 Jul 2026 17:00:03 GMT

RabbitMQ is an enterprise-grade open-source messaging and streaming broker. In this blog, you will learn some basic concepts of RabbitMQ and how to use it in a Spring Boot application. Enjoy!

Introduction

Before diving into the programmatic details, first some concepts need to be explained. Do realize that in this blog, only the surface is scratched from what is possible with RabbitMQ. A detailed overview can be found in the official RabbitMQ documentation.

Building Production-Grade Delta Lake Pipelines With Apache Spark on Databricks

Jubin Abhishek Soni — Wed, 08 Jul 2026 14:00:00 GMT

Why Delta Lake?

Apache Parquet on cloud storage was a great first step for data lakes — but it left engineers dealing with a painful set of problems in production:

No ACID transactions — concurrent reads/writes could corrupt data silently
Schema drift — nothing stopped upstream systems from changing column types
No deletes or updates — GDPR compliance meant rewriting entire partitions
Painful failure recovery — half-written data after a job crash became your problem

Delta Lake solves all of this by sitting on top of Parquet and adding a transaction log (_delta_log/) that records every operation atomically. On Databricks, Delta is the default table format, deeply integrated with Apache Spark, Auto Optimize, and the Photon execution engine.

AI Won't Keep You from Hitting the Scalability Wall

Bru Woodring — Wed, 08 Jul 2026 12:00:00 GMT

Using AI to build integrations? You might just be hitting the scalability wall faster. Discover why faster builds don't solve the long-term cost of ownership.

There's an idea making the rounds in B2B SaaS product and engineering meetings right now. It sounds reasonable. It feels optimistic. And it's leading companies straight into the same trap they've always fallen into, just at an accelerated rate.

How to Build a Production-Ready RAG Pipeline With Vector DBs

Mark Saxon — Mon, 06 Jul 2026 19:15:01 GMT

The adoption of retrieval-augmented generation (RAG) from research papers to production systems has been rapid. Those who tried it in 2023 are now deploying it at scale for enterprise search, internal knowledge bases, and customer-facing assistants. However, a lot is still between a working prototype RAG and one that can withstand traffic on the road, using real data, and real modes of failure.

This article explains what this gap is, how to plug it, and where most production pipelines fail.

Building an AI Agent That Responds to Real-Time Events With AWS Bedrock, Kinesis, DynamoDB, and S3

Jubin Abhishek Soni — Fri, 03 Jul 2026 13:00:04 GMT

Most recommendation systems are batch jobs. They crunch last night's data, write a recommendations table, and serve it all day. That works fine until your user watches three thriller movies in a row at 9 pm and your system is still recommending rom-coms because the batch hasn't run yet.

In this post, I'll walk through building an agent system that reacts to streaming user behavior in real time using:

One Stolen Key, One Stolen Token: Why Machine Identity Is Cloud-Native's Quietest Crisis — and the Only Fix That Actually Holds

Igboanugo David Ugochukwu — Wed, 01 Jul 2026 17:00:00 GMT

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software.

With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025.

Building Production-Safe Agentic Remediation With Docker MCP Gateway: Lessons From 43% to 100% Accuracy

Mohammad-Ali Arabi — Mon, 29 Jun 2026 17:00:00 GMT

Our first version was wrong 57% of the time.

Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken.

Data Pipeline Observability: Why Your AI Model Fails in Production

Abhilash Rao Mesala — Fri, 26 Jun 2026 16:00:05 GMT

The 3:00 AM Incident That Changed Everything

It was a Tuesday morning when the alerts started firing. Our recommendation engine, the one that drives 30% of our revenue, had tanked. Accuracy dropped from 94% to 58% overnight. The data science team immediately blamed the model. They started tweaking hyperparameters, re-training on new data, and running diagnostics. Nothing worked.

I got pulled into the war room at 3:00 AM. The first thing I asked wasn't "What's wrong with the model?" It was "What changed in the data pipeline?"

Turns out, everything.

A vendor had pushed a schema change upstream. A field that used to be required became optional. Null values started flowing through our pipeline. Our feature engineering code didn't handle nulls gracefully; it just propagated them downstream. By the time the data reached the model, 40% of our feature vectors were corrupted.

The model wasn't broken. The data was.

We spent six hours manually rolling back the schema change, re-running the pipeline, and restoring service. The incident report was brutal: "Lack of data validation caught a breaking change too late."

That's when I realized we needed observability in our data pipeline, not just in our models.

The Problem: Data Quality is Invisible Until It Breaks

Here's the uncomfortable truth about data pipelines: they fail silently.

Your ETL job completes successfully. Your Spark cluster finishes transformations. Your data warehouse loads without errors. Everything looks green in the monitoring dashboard. But the data itself? Garbage in, garbage out.

There are three categories of failures that break AI models in production:

Missing Values: A source system stops populating a field. Your pipeline doesn't validate it. The model gets NaN values it never saw during training. Predictions become random noise.
Schema Changes: An upstream team adds a new column, renames an existing one, or changes data types. Your pipeline doesn't expect these changes. Either it crashes, or worse, it silently maps data to the wrong columns.
Distribution Shifts: The statistical properties of your data change. A field that was always between 0 and 100 suddenly has values of 50,000. Your model's scaling assumptions break. Predictions become nonsensical.

None of these show up in traditional infrastructure monitoring. Your CPU is fine. Memory is fine. Network is fine. But your data is on fire.

The Solution: Observability at Every Layer

I started building a three-layer observability framework using dbt, Great Expectations, and custom validation logic. The goal was simple: catch data quality issues before they reach the model.

Layer 1: dbt Tests (The First Line of Defense)

dbt tests are your cheapest, fastest way to catch obvious data quality issues. They run after every transformation and fail the entire pipeline if something's wrong.

Here's what we implemented:

       SQL
      
 
 0"" data-lang="text/x-sql">
      -- models/staging/stg_user_events.yml
version: 2

models:
  - name: stg_user_events
    columns:
      - name: user_id
        tests:
          - not_null
          - unique
      - name: event_timestamp
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "event_timestamp <= current_timestamp()"
      - name: event_value
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "event_value > 0"
     

These tests are simple but powerful. They catch:

Missing required fields (not_null)
Duplicate records (unique)
Impossible values (event_timestamp in the future)
Out-of-range values (negative prices)

We run these tests on every dbt run. If any test fails, the pipeline stops. No data reaches the model. No silent corruption.

The beauty of dbt tests is that they're version-controlled, documented, and part of your transformation code. When a schema change happens, you update the test, commit it, and everyone knows what changed.

Layer 2: Great Expectations (The Statistical Validator)

dbt tests catch structural issues. Great Expectations catches statistical anomalies, the subtle shifts that break models.

Here's a real scenario: our user_age column had a distribution of 18-65 for two years. Then one day, we started getting ages of 200, 500, 1000. A data entry bug upstream. dbt tests wouldn't catch this because the values are technically valid integers. But Great Expectations would.

         Python
        
 

        # great_expectations/expectations/user_events_expectations.py
from great_expectations.core.batch import RuntimeBatchRequest
from great_expectations.data_context import DataContext

context = DataContext()
suite = context.create_expectation_suite(
    expectation_suite_name="user_events_suite",
    overwrite_existing=True
)

validator = context.get_validator(
    batch_request=RuntimeBatchRequest(
        datasource_name="my_spark_datasource",
        data_connector_name="default_runtime_data_connector",
        data_asset_name="user_events"
    ),
    expectation_suite_name="user_events_suite"
)

# Expect user_age to be between 18 and 120
validator.expect_column_values_to_be_between(
    column="user_age",
    min_value=18,
    max_value=120
)

# Expect event_value to have a mean between 50 and 200
validator.expect_column_mean_to_be_between(
    column="event_value",
    min_value=50,
    max_value=200
)

# Expect less than 5% missing values in critical columns
validator.expect_column_values_to_not_be_null(
    column="user_id",
    mostly=0.95
)

# Expect the distribution to match historical patterns
validator.expect_column_kl_divergence_from_list(
    column="event_type",
    partition_object={"event_type": ["click", "view", "purchase"]},
    threshold=0.1
)

validator.save_expectation_suite(discard_failed_expectations=False)
       

Great Expectations runs after dbt tests. It validates:

Value ranges (age between 18 and 120)
Statistical properties (mean event value between 50 and 200)
Null rates (less than 5% missing in critical columns)
Distribution shifts (event_type distribution matches historical patterns)

If Great Expectations detects an anomaly, it alerts us. We investigate before the data reaches the model.

Layer 3: Custom Validation (The Domain Expert)

dbt and Great Expectations are generic. Your domain is specific. We added custom validation logic that understands our business.

          Python
         
 
 -10,
            'max': max_val,
            'min': min_val
        }
    
    # Validate 3: Temporal consistency
    # Events should be recent (within last 30 days)
    if 'event_date' in df.columns:
        df['event_date'] = pd.to_datetime(df['event_date'])
        days_old = (datetime.now() - df['event_date'].max()).days
        results['temporal_freshness'] = {
            'passed': days_old < 30,
            'days_old': days_old,
            'threshold_days': 30
        }
    
    # Validate 4: Business logic
    # Revenue should always be positive
    if 'revenue' in df.columns:
        negative_revenue = (df['revenue'] < 0).sum()
        results['business_logic_revenue'] = {
            'passed': negative_revenue == 0,
            'negative_count': negative_revenue
        }
    
    return results

def validate_and_alert(df: pd.DataFrame, validation_results: dict) -> bool:
    """
    Check all validations and alert if any fail.
    Returns True if all pass, False otherwise.
    """
    all_passed = True
    
    for check_name, check_result in validation_results.items():
        if not check_result['passed']:
            all_passed = False
            print(f"ALERT: {check_name} failed")
            print(f"Details: {check_result}")
            # Send to monitoring system (Datadog, New Relic, etc.)
            # send_alert(check_name, check_result)
    
    return all_passed" data-lang="text/x-python">
         # pipelines/validation/custom_validators.py
import pandas as pd
from datetime import datetime, timedelta

def validate_feature_engineering(df: pd.DataFrame) -> dict:
    """
    Custom validation for features before they reach the model.
    Returns a dict of validation results.
    """
    results = {}
    
    # Validate 1: Feature completeness
    # We need at least 95% of features populated
    feature_cols = [col for col in df.columns if col.startswith('feature_')]
    null_rate = df[feature_cols].isnull().sum().sum() / (len(df) * len(feature_cols))
    results['feature_completeness'] = {
        'passed': null_rate < 0.05,
        'null_rate': null_rate,
        'threshold': 0.05
    }
    
    # Validate 2: Feature scaling
    # After normalization, features should be roughly between -3 and 3 (3 sigma)
    for col in feature_cols:
        max_val = df[col].max()
        min_val = df[col].min()
        results[f'{col}_scaling'] = {
            'passed': max_val < 10 and min_val > -10,
            'max': max_val,
            'min': min_val
        }
    
    # Validate 3: Temporal consistency
    # Events should be recent (within last 30 days)
    if 'event_date' in df.columns:
        df['event_date'] = pd.to_datetime(df['event_date'])
        days_old = (datetime.now() - df['event_date'].max()).days
        results['temporal_freshness'] = {
            'passed': days_old < 30,
            'days_old': days_old,
            'threshold_days': 30
        }
    
    # Validate 4: Business logic
    # Revenue should always be positive
    if 'revenue' in df.columns:
        negative_revenue = (df['revenue'] < 0).sum()
        results['business_logic_revenue'] = {
            'passed': negative_revenue == 0,
            'negative_count': negative_revenue
        }
    
    return results

def validate_and_alert(df: pd.DataFrame, validation_results: dict) -> bool:
    """
    Check all validations and alert if any fail.
    Returns True if all pass, False otherwise.
    """
    all_passed = True
    
    for check_name, check_result in validation_results.items():
        if not check_result['passed']:
            all_passed = False
            print(f"ALERT: {check_name} failed")
            print(f"Details: {check_result}")
            # Send to monitoring system (Datadog, New Relic, etc.)
            # send_alert(check_name, check_result)
    
    return all_passed
        

This custom validation runs after Great Expectations. It checks:

Feature completeness (95% of features populated)
Feature scaling (normalized features in the expected range)
Temporal freshness (data is recent)
Business logic (revenue is positive)

If any check fails, we block the pipeline and alert the team.

The Real-World Gotchas We Discovered

Gotcha 1: Validation Overhead

Running dbt tests, Great Expectations, and custom validation on every pipeline run adds latency. We went from 15-minute runs to 25-minute runs. The trade-off was worth it (catching one data quality issue saved us more time than we lost), but you need to plan for it.

Gotcha 2: False Positives

Great Expectations' distribution shift detection is sensitive. Legitimate business changes (a marketing campaign causing a spike in user_age distribution) triggered false alerts. We had to tune thresholds carefully and add context to alerts.

Gotcha 3: Schema Changes Are Sneaky

A vendor added a new column to an upstream table. Our pipeline didn't break; it just ignored the new column. But the data science team expected it. We added schema validation to catch new columns and alert us.

Gotcha 4: Null Handling Varies

Python treats null as None. SQL treats it as NULL. Spark treats it as null. When data flows between systems, nulls get lost or misinterpreted. We had to standardize null handling across the entire pipeline.

The Framework: A Decision Matrix

Here's how we decide which validation layer to use:

Issue Type	Caught By	Example	Action
Missing required field	dbt tests	user_id is null	Fail pipeline immediately
Duplicate records	dbt tests	Same user_id appears twice	Fail pipeline immediately
Impossible values	dbt tests	event_timestamp in future	Fail pipeline immediately
Out-of-range values	Great Expectations	age > 150	Alert, investigate, fail if severe
Distribution shift	Great Expectations	event_value mean changes 50%	Alert, investigate, continue if acceptable
Business logic violation	Custom validation	revenue is negative	Alert, investigate, fail
Schema change	Custom validation	New column added upstream	Alert, investigate, update tests

The Results: From Chaos to Confidence

After implementing this three-layer framework:

Incident reduction: We went from 2-3 data quality incidents per month to 0 in six months.
Time to resolution: When issues do occur, we catch them within minutes instead of hours.
Model stability: Model accuracy stopped fluctuating. It's now consistently 93-95%.
Team confidence: Data scientists trust the data. Engineers trust the pipeline.

The best part? We caught the schema change incident before it happened. Great Expectations detected the distribution shift, we investigated, found the upstream change, and coordinated with the vendor team before any data reached production.

Getting Started: The Minimal Viable Observability

You don't need to implement everything at once. Start here:

Week 1: Add dbt tests for not_null and unique on critical columns.
Week 1: Add dbt tests for not_null and unique on critical columns.
Week 1: Add dbt tests for not_null and unique on critical columns.
Week 4: Set up alerting so you're notified when validations fail.

That's it. You now have observability in your data pipeline.

Conclusion: Observability Saves Models

Your AI model isn't failing because it's bad. It's failing because the data feeding it is bad. And you won't know the data is bad until you look.

The best models in the world can't save you from garbage data. But good observability can. dbt tests, Great Expectations, and custom validation aren't fun. They don't make it into conference talks. But they'll save your production system at 3:00 AM.

Start small. Test early. Validate often.

A Tool Is Not a Platform (And Your Team Knows the Difference)

Jeleel Muibi — Thu, 25 Jun 2026 19:00:05 GMT

Most infrastructure teams have a moment where someone says “we should build a platform.” The motivation is real: teams are duplicating work, the current setup is hard to use consistently, and a more structured approach would help. A few months later, the platform is a Terraform module collection, a GitLab CI template, a shared repository of scripts, and a README that several people have tried to keep current.

That is a useful thing. It is not a platform.

Implementing Asynchronous Communication Between Microservices Using Kafka and Spring Boot

Mallikharjuna Manepalli — Wed, 24 Jun 2026 13:00:05 GMT

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another.

Code Patterns in Spring Boot With Kafka

Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing.

Architectural Collapse: How Extension Poisoning, Node Vulnerabilities, and Infrastructure Fog Enabled the GitHub Repository Breach

Akash Lomas — Tue, 23 Jun 2026 19:00:00 GMT

Enterprise perimeter defenses are fundamentally built on an obsolete assumption that the developer’s workstation is a secure, trusted anchor point. The massive security breach executed by the threat group TeamPCP, resulting in the exfiltration of 3,800 internal GitHub source code repositories, completely shattered this illusion.

This was not a standalone exploit. It was a multi-vector convergence where vulnerabilities in the Node/NPM ecosystem, the systemic ungoverned architecture of the Visual Studio Code Marketplace, and the tactical “fog of war” caused by a period of historic GitHub infrastructure instability came together to create the perfect attack.

Automating Power Automate: How to Ensure Cloud Flows Are Active After Every Pipeline Deployment

karthik nallani chakravartula — Fri, 19 Jun 2026 18:00:00 GMT

You've spent hours — maybe days — building and testing a Dynamics 365 Power Platform solution. Your Azure DevOps pipeline runs clean. The managed solution imports successfully into the target environment. All green.

Then the business calls. Nothing is working. The automations aren't firing.

Your AI Coding Agent Can't Steal What It Never Had: The Docker Sandbox Isolation Story

Shamsher Khan — Fri, 19 Jun 2026 13:00:00 GMT

I ran an AI coding agent against a broken Kubernetes deployment for five minutes. The agent called Anthropic's API dozens of times — reasoning about manifests, running kubectl commands, redeploying workloads. It made fully authenticated requests throughout the entire session.

The API key was never in its environment.

Cutting Data Pipeline Costs and Data Freshness Issues With Netflix Maestro and Apache Iceberg: A Practical Tutorial

Intiaz Shaik — Tue, 16 Jun 2026 16:00:01 GMT

Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue.

In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July 2024, along with Apache Iceberg, a standard table format for analytics on object storage. The former helps by shifting from time-based scheduling to event-driven, whereas the latter removes the overhead of listing files that slows down queries on large datasets and increases their costs.

Getting Started With GitHub Copilot CLI for Coding Tasks

Gunter Rotsaert — Tue, 16 Jun 2026 14:30:00 GMT

Nowadays, there are quite a lot of AI coding assistants. In this blog, you will take a closer look at GitHub Code CLI, a terminal-based AI coding assistant. GitHub Copilot CLI integrates smoothly with GitHub Copilot, so if you have a GitHub Copilot subscription, it is definitely worth looking at. Enjoy!

Introduction

There are many AI models and also many AI coding assistants. Which one to choose is a hard question. It also depends on whether you run the models locally or in the cloud. When running locally, Qwen3-Coder is a very good AI model to be used for programming tasks. In previous posts, DevoxxGenie, a JetBrains IDE plugin, was often used as an AI coding assistant. DevoxxGenie is nicely integrated within the JetBrains IDE's. But it is also a good thing to take a look at other AI coding assistants. In previous blogs, Qwen Code and Claude Code were used in combination with local models.

From ETL to Lakeflow: Shifting to a Declarative Data Paradigm

Seshendranath Balla Venkata — Mon, 15 Jun 2026 15:00:00 GMT

If you've worked on a data platform for more than a few years, you've almost certainly built the same pipeline twice. First, the way the team wrote pipelines in 2019: a notebook here, a Python script there, an Airflow DAG to glue it all together, and a long document explaining the order things had to run in. Then the rewrite, two years later, when somebody quit, and nobody could remember why a particular task had a sleep(180) in it.

Lakeflow is Databricks' answer to that pattern, and the shift it's pushing for is bigger than the marketing makes it sound. It isn't a new orchestrator. It's a move from imperative pipelines, where you write the steps, to declarative pipelines, where you write the destination and let the engine figure out the steps. What follows is the practical version of that shift — what's actually different, where the gains are real, and how to migrate without ending up with a half-converted lakehouse.