DZone Data Zone

Engineering Audit-Ready Data Pipelines for Regulatory Reporting

SUNILKUMAR REDDY Eraganeni — Mon, 19 Oct 2026 19:00:00 GMT

There's a particular kind of silence that happens on a call when a regulator asks, "Can you walk me through exactly how this number was calculated?" and the person on the other end has to say, "Let me get back to you." I've been on that call. It is not a fun place to be, and it usually isn't because anyone did anything wrong. It's because the pipeline that produced the number was never built to answer that question in the first place.

Regulatory reporting, whether it's a liquidity return, a capital calculation, or something more niche, has a strange property compared to most data engineering work. Nobody outside the team really cares how the dashboard behind it looks. What they care about is whether you can prove, months or years later, exactly where every figure came from, who touched it, and why it changed between two reporting periods. That's a fundamentally different design goal than "make the report run fast," and honestly, a lot of pipelines get built for speed and convenience first, with auditability bolted on afterward as an afterthought. It shows.

Moving 900+ GB of Prometheus Without a Cutover Weekend

Daniil Romashov — Fri, 16 Oct 2026 14:00:00 GMT

The Prometheus I was responsible for had grown past 900 GB of time series data, and at that size it had every disease that comes with it.

Queries timed out unless you narrowed them down to an hour. Cardinality kept growing, because teams added labels "for debugging" and then sometimes forgot to remove them. On top of that, we were shipping a copy of fractions of our metric data to an external SaaS vendor, just in case, and that bill grew every quarter. And the fleet under all of this ran latency-sensitive trading services, so monitoring itself was not allowed to eat an extra CPU core at a wrong moment.

Data Governance Is a Security Control, Not a Compliance Checkbox

Rahul Singh — Wed, 14 Oct 2026 19:00:01 GMT

I've spent years fixing broken Glue jobs and Lake Formation permissions as an AWS support engineer, then moved to pre-sales architecture for banking data platforms. One pattern shows up in almost every migration: teams build IAM policies and a data catalog as two separate workstreams, run by two separate teams, that never talk to each other. Then six months post-launch, someone asks, "Who can see the SSN column in the gold layer?" and nobody can answer without a week of manual auditing.

That gap isn't a process problem. It's an architecture problem. This article walks through why classification and access control need to be the same system, with the actual policy constructs to make that work on AWS, and where teams get this wrong.

Rotate AWS Secrets Manager Secrets Without Restarting Your Go Service, With mamori

Xavier Portilla Edo — Wed, 14 Oct 2026 14:00:06 GMT

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved.

There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened.

Introducing mamori: Typed, Watchable Config and Secrets for Go

Xavier Portilla Edo — Wed, 14 Oct 2026 12:00:04 GMT

Every Go service I have ever shipped ends up growing the same organ. It starts small: a couple of os.Getenv calls. Then a secret needs to come from AWS Secrets Manager, so you add the SDK. Then someone wants that secret to rotate without a deploy, so you add a time.Ticker. Then the ticker needs a mutex because two goroutines read the config while it is being replaced. Then a bad value slips in and takes down production, so you add validation. Then a secret shows up in a log line during an incident, and now you are adding a redaction wrapper too.

By the end you have a bespoke ConfigManager in every repo: a loader, a fetcher, a refresh loop, a mutex, and a prayer. It is never quite the same twice, it is rarely tested well, and it is exactly the kind of code nobody wants to own.

From Legacy ETL to Modern ELT: Engineering a Governed dbt and Databricks Architecture

SUNILKUMAR REDDY Eraganeni — Mon, 12 Oct 2026 12:00:00 GMT

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had touched in four years. The original author had left the company. Nobody knew why a particular join filtered out cancelled orders, only that removing it broke three downstream reports. That afternoon summed up, for me, everything wrong with how a lot of organizations still build their data pipelines.

Legacy ETL was never really about "extract, transform, load" as much as it was about hiding logic somewhere convenient at the time and hoping the convenience lasted. It usually didn't.

How Cache Invalidation Caused Memory Pressure and API Failures at Scale

Semyon Slepov — Fri, 09 Oct 2026 15:00:00 GMT

Most production systems do not fail loudly. They degrade quietly.

This case started with an API that was technically “healthy.” It was serving traffic, success rates were high, and no critical alerts were firing. Yet recurring degradation periods caused the service to miss its service-level agreement (SLA) target for request success. During the worst degradation periods, request success could fall to approximately 98%, against a target of 99.9%. Some requests were simply failing. Because the API sat directly in the purchase path, the gap between 98% and 99.9% meant real checkout failures, abandoned transactions, and lost product revenue.

Threat Modeling Context-Aware AI: A STRIDE Walkthrough

Sandeep Gadde — Fri, 09 Oct 2026 14:00:10 GMT

Context-aware AI has become the default architecture for grounding Large Language Models (LLMs) in private, enterprise data. It is also quietly one of the most attack-rich architectures to reach production in years: it combines untrusted user input, untrusted document content, a probabilistic model that follows instructions from both, and privileged access to internal data stores.

Most teams threat model their APIs and their infrastructure. Very few threat model the AI context pipeline itself. This walkthrough does exactly that: it takes a realistic reference architecture, walks it through STRIDE, and ends with a prioritized mitigation list you can apply to your own system.

Building IoT Time-Series Applications With Java and Apache IoTDB

Otavio Santana — Tue, 06 Oct 2026 16:00:12 GMT

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and requires efficient querying. Typical use cases include monitoring temperature, pressure, vibration, energy usage, machine status, and device telemetry. This approach also applies to manufacturing, smart infrastructure, fleet monitoring, utilities, and edge computing.

IoTDB stands out for its focus on large-scale time-series workloads from devices and industrial systems. It is purpose-built for high-frequency data ingestion, historical analysis, and time-based queries. This makes it ideal for applications that require insight into both the current state and historical trends of devices or processes.

OpenSearch Heap Sizing: Swap, Page Cache, and the 50% Rule

Maxim Muzafarov — Tue, 06 Oct 2026 14:00:06 GMT

OpenSearch is an open-source, distributed search and analytics suite derived as a fork of Elasticsearch and maintained under the Apache 2.0 license. When it comes to memory configuration, the guidance is often reduced to a few rules of thumb: swapoff -a, vm.swappiness=1, or bootstrap.memory_lock, and allocating 50% of available memory to the JVM heap while leaving the rest for Lucene and the filesystem page cache, OpenSearch off-heap caches, network buffers, and other system needs.

These recommendations are repeated throughout documentation, blog posts, and operational guides, yet their origins and the mechanisms that justify these specific values are rarely examined. Undoubtedly, they provide a reasonable and safe starting point or a safe upper bound in most of the cases, but a safe default is not necessarily an optimal configuration.

Building an AI-Ready Data Layer Without Rebuilding the Enterprise

Rajaganapathi Rangdale Srinivasa Rao — Tue, 06 Oct 2026 13:00:01 GMT

Most enterprise AI programs do not fail because the model is too weak. They stall because the data underneath the model is fragmented, delayed, poorly documented, or too expensive to access repeatedly.

The common response is to propose a complete platform replacement. That sounds clean on a diagram and becomes dangerous in production. Existing warehouses often support financial reporting, operational dashboards, manufacturing analytics, and regulatory processes that cannot pause while a new AI platform is assembled.

Building a Migration Readiness Engine for Analytics Workflows

Ravi Krishna Palivela — Mon, 05 Oct 2026 15:00:00 GMT

Platform migrations are not only technology problems.

They're visibility problems as well.

How Go Maps Work: From Buckets to Swiss Tables

Ilia Ivankin — Thu, 01 Oct 2026 15:00:00 GMT

Hey Mates!

“How does a map work in Go?” is one of my favorite interview questions. It sounds simple, but it opens up a conversation about hashing, collisions, memory layout, and why two implementations with the same average O(1) lookup complexity can behave quite differently. With Go 1.24, that conversation got more interesting: the map implementation switched to Swiss Tables. Let’s walk through how the old implementation worked, what changed, and how the new design finds your keys.

TL;DR

Go 1.24, released in February 2025, completely replaced the internal implementation of the built-in `map` with a design based on Google's Swiss Tables. The syntax and the behavior guaranteed by the Go specification did not change, so existing programs required no migration. In microbenchmarks, some map operations became up to 60% faster. Across the Go team's full-application benchmark suite, geometric mean CPU time improved by about 1.5%. Datadog reported roughly 70% less memory for one exceptionally large map—an impressive case study, not a universal promise.

Jakarta Batch in Practice: Reliable Chunk-Oriented Processing for Enterprise Workloads

Otavio Santana — Tue, 29 Sep 2026 14:00:00 GMT

Batch processing remains vital because many business operations aren't suited to interactive requests. Tasks such as recalculating prices, reconciling transactions, migrating records, generating reports, processing invoices, reclassifying customers, or applying rules across millions of records may require considerable time. Handling these as standard requests leads to fragile systems, increased user wait times, frequent timeouts, challenging retries, and possible data inconsistencies.

A batch model handles large workloads predictably, incrementally, and with control over progress and recovery. Rather than processing a massive operation as a single loop, batch processing uses jobs, steps, chunks, checkpoints, filtering, and restartability. This approach separates long-running data tasks from the user experience while delivering a structured execution model. In this article, we will focus on Jakarta Batch and examine its sustained relevance for modern enterprise applications.

The Warning That Never Stops the Agent

Ninaad Rao — Tue, 29 Sep 2026 13:00:00 GMT

I went looking for a flag that could cap the amount spent on a session on DeepAgents. The kind of hard spending cap that stops an agent loop before it burns through a budget. What exists instead is a warning, and the gap between "warning" and "stopping" turned out to be the more interesting story.

What Already Exists: A Number With No Teeth

deepagents-code has a CostTrackingMiddleware that owns a thread's cumulative spend. This is a real, checkpointed dollar figure priced from actual per-request token usage, not an estimate. A separate feature built on top of that number, merged earlier, adds a one-time warning once the total crosses a configured threshold (default $50):

Beyond Screenshots: Building Replayable Production Diagnostics for Hard-to-Reproduce Bugs

Uthej Mopathi — Mon, 28 Sep 2026 17:00:00 GMT

Production bugs frequently arrive with too little evidence. A screenshot captures the final visual state, a crash report identifies a failing stack, and a support ticket describes what appeared to happen. None reliably explains the sequence that produced the failure. Modern applications are asynchronous systems driven by navigation, network responses, feature flags, background work, local persistence, and changing UI state. A useful production bug report therefore needs more than the final frame. It needs a bounded, privacy-safe execution history that reconstructs the path leading to failure.

The foundation of a replayable bug report is a semantic event stream. Continuous video recording is expensive, difficult to search, and likely to capture information unrelated to diagnosis. Structured events are smaller and describe meaningful transitions directly. Navigation changes, button actions, state mutations, network outcomes, feature flag evaluations, lifecycle transitions, and persistence failures can use a common event model.

Prompt Caching Doesn't Save Money on Turn One

Ninaad Rao — Fri, 25 Sep 2026 18:00:00 GMT

I went looking for a clean way to show what prompt caching actually saves an agent, and the first thing I found was a fact that's easy to miss if you only read the "up to 90% savings" headline. Caching a fresh conversation's first turn costs more than not caching it. There's no cache to read from yet, so you pay the input price on the content plus a 25% premium to write to the cache, and get nothing back. The savings arrive starting on turn two, once there's something to read.

Where This Lives in deepagents

Deep Agents ships AnthropicPromptCachingMiddleware from langchain-anthropic in its default middleware stack. It doesn't decide what to cache by guessing. It tags exactly two things on every model call:

Locking Down the Enterprise: Data Security Patterns for AI Integrations

Balaji Venkatasubramaniyar — Fri, 25 Sep 2026 12:00:06 GMT

Every enterprise conversation about artificial intelligence eventually arrives at the same uncomfortable question: what happens to our data once it leaves our perimeter? Whether you are wiring a large language model into a claims processing pipeline, standing up a retrieval-augmented generation (RAG) system for internal knowledge search, or letting an agentic workflow take autonomous actions against production systems, the answer determines whether your AI initiative becomes a competitive advantage or a compliance incident waiting to happen.

This article lays out a layered, defense-in-depth approach to securing enterprise data across the AI lifecycle — from data classification and access control, through transit and storage, vendor contracts, prompt hygiene, architecture patterns, and compliance mapping. It is written for architects, technical leads, and engineering managers who are past the "should we use AI" conversation and are now living in the "how do we do this safely at scale" reality.

How to Prevent Retry Storms With Retry Budgets in Distributed Systems

Uthej Mopathi — Wed, 23 Sep 2026 13:00:05 GMT

Retries are one of the simplest ways to make a distributed system appear more reliable. A transient connection failure, overloaded replica, or short-lived network interruption can disappear after another attempt, which is why retry support exists in major RPC frameworks and cloud SDKs.

The danger begins when every layer makes the same decision independently. A mobile client retries an API gateway, the gateway retries a service, that service retries another service, and the final dependency retries a database call. The original request has not become more important, but the system has multiplied the work required to fail. AWS describes a five-deep service stack in which three attempts at each layer can drive 243 calls against the database when the deepest dependency is failing. Google’s SRE guidance similarly warns that retries can amplify overload and contribute to cascading failure.

Exploring the DBMS_DEVELOPER Package: JSON Metadata for Oracle Objects

arvind toorpu — Fri, 18 Sep 2026 16:00:02 GMT

Oracle Database 23ai introduced the powerful DBMS_DEVELOPER package, giving developers and database administrators a streamlined way to access database object metadata in JSON format. This feature represents a significant advancement in how we interact with database schemas, offering a more structured and programmatic way to extract and analyze metadata compared to traditional dictionary views or the older DBMS_METADATA package.

In this article, we'll explore the capabilities of DBMS_DEVELOPER, focusing on its GET_METADATA function through detailed examples and practical implementation scenarios.