DZone Databases Zone

Your Spark Job Isn't Slow Because of Bad Code. It's Slow Because of the Wrong Join

Syed Siraj Mehmood — Thu, 03 Sep 2026 15:00:04 GMT

I learned this lesson the hard way.

We had a critical data pipeline running for over 3 hours every single day. The logic was perfectly clean. The overarching schema was explicitly right. There were absolutely no obvious memory leaks, and absolutely nothing looked fundamentally broken in the raw PySpark transformations.

Building a Python API Client That Doesn’t Fall Apart When the API Misbehaves

Ally Garcia — Thu, 03 Sep 2026 14:00:10 GMT

The first version of almost every API client I write looks embarrassingly simple.

Send a request. Parse the JSON. Return the result.

Extracting Entities and Relationships From Engineering Documents With spaCy

Sriharsha Makineni — Wed, 02 Sep 2026 18:00:09 GMT

Engineering teams generate a lot of useful knowledge, but most of it is locked inside text.

A service ownership note may tell you who owns an application programming interface (API), while a runbook may tell you which database a service relies on. An incident review may detail how one fault impacted the other systems. Each of these is individually useful. It’s when we are able to link together all of these facts that we get our greatest value.

3D Air Quality Maps With Neo4j, Python, and R

Akmal Chaudhri — Wed, 02 Sep 2026 13:00:11 GMT

Air quality across the Pyrenees corridor is a pressing concern, particularly during summer heatwaves and wildfire seasons. In this region, cities on both sides of the French-Spanish border share air masses, making cross-border analysis a natural fit for a graph database.

In this article, we'll build a pipeline that fetches real-time air quality data from the IQAir API, stores it in Neo4j as a graph, and produces complementary views of the data:

Stop Hardcoding Database Checks: Building a Metadata-Driven Data Quality Framework

Kshitish Nath — Tue, 01 Sep 2026 15:00:05 GMT

In high-volume data platforms, hardcoding validation logic into individual processing pipelines creates significant operational drag. As an enterprise data asset footprint grows, maintaining manual checks for hundreds of tables inevitably leads to mounting technical debt, silent schema drift, and a fragmented audit trail.

To achieve data governance at scale, data architects must decouple validation rules from the execution engine. By utilizing a centralized metadata repository to dynamically generate validation suites, organizations can transform data quality from a reactive, script-based bottleneck into a configuration-driven infrastructure asset.

How to Detect AI-Generated Images in C# Using an API

Brian O'Neill — Tue, 01 Sep 2026 14:00:06 GMT

When someone uploads an image to your application, it might look perfectly fine at first glance. It might open correctly, pass file validation, and avoid triggering any obvious red flags. But that still doesn’t necessarily mean the file is trustworthy.

In modern systems, especially marketplaces, identity verification flows, insurance submissions, academic portals, and editorial pipelines, how an image was created can matter just as much as what that image shows. A synthetic, AI-generated image can be technically valid and "safe" while still being completely inappropriate for the context in which it’s used.

Natural IDs in Your Database. I Am Telling You for the Last Time!

Mikhail Polivakha — Mon, 31 Aug 2026 18:00:01 GMT

Hi everyone! This is Mikhail Polivakha, tech lead of the Axelix project (btw, give us a star!). In my experience consulting teams that build enterprise applications, I keep getting asked:

What about natural keys in the database? Say I have a column that lets me explicitly identify a record, should I use it as the Primary Key?

Securing Database Connections With Mutual TLS

Rahul Roy — Mon, 31 Aug 2026 17:00:14 GMT

Stolen credentials served as the entry point in 22% of breaches last year, and in attacks on basic web applications that figure climbs to 88%. Those numbers describe a password problem, and databases sit at the end of nearly every attack path.

A username and password prove nothing about the machine presenting them. Mutual TLS closes that gap by requiring both sides of a connection to present certificates and prove who they are before a single query runs.

How I Built a SQL Diagnostic Tool That Works Without Touching Your Database

Sudhakararao Sajja — Mon, 31 Aug 2026 15:00:12 GMT

Most developers I've worked with write SQL every day. Very few of them are DBAs. According to the 2024 Stack Overflow Developer Survey — 65,000 developers across 185 countries — database administrators make up just 0.3% of the developer population. The tools built for SQL performance were designed for that 0.3%. I built QueryTuner for everyone else.

I've spent 13 years as an application architect. In that time, I've watched the same situation repeat itself across teams: a query is slow, the developer who wrote it has to fix it, and the tools available to them are either way too expensive or way too generic. Enterprise monitoring agents like pganalyze or Datadog Database Monitoring cost hundreds of dollars a month and require installing an agent with full database credentials. Generic AI LLMs don't know whether you're on Oracle or MySQL. There's nothing useful in between.

Deliberate Decoupling: 6 Architectural Patterns From a Regulated WAS-to-AWS Migration

Alka Nimje — Fri, 28 Aug 2026 16:00:01 GMT

Key Takeaways

In regulated industries, cloud migration success is determined less by technology selection and more by how deliberately you decouple risk vectors — compliance risk, organizational hesitation, user adoption gaps, and integration changes — so no single failure can derail the whole program.
You can successfully migrate an application to AWS while keeping data on-premises by routing through a REST API abstraction (e.g., IBM’s DB2 REST API layer) paired with dedicated AWS security groups controlling cloud-to-on-prem traffic, allowing the data migration to proceed on its own compliance and trust-building timeline.
The most dangerous compliance gap in regulated applications isn’t declared sensitive fields — it’s free-form text fields where users may inadvertently type SSNs, credit cards, or other regulated identifiers; proactive tokenization in the application’s write path closes this gap before any audit finds it.
Long-tenured business users carry a decade of UX muscle memory that QA testing cannot replicate; allocating real production validation time (such as a 15-day dark deployment cohort) is essential when migrating systems users have relied on daily for 10+ years.
Before starting a regulated cloud migration, ask which risk vector each architectural decision is decoupling and whether your team is aligned on why — this single question reframes "cloud migration" from a technology project into a coordinated risk-management exercise.

Introduction

Most published writing on legacy-to-cloud migration treats it as a technical exercise: pick the stack, plan the cutover, flip the switch. In regulated industries, that framing fails — and the failure mode isn’t a missed deployment window. It’s a stalled program, a failed compliance audit, or a client who pulls back from the cloud strategy entirely.

A cloud migration in healthcare insurance is as much about regulatory risk management, organizational trust-building, and user adoption as it is about microservices and Fargate. Get the technology right and miss the risk choreography, and the project doesn’t ship.

When Guest Access Becomes an Attack Surface: A Technical Analysis of the City-Forum Campaign

Igboanugo David Ugochukwu — Thu, 27 Aug 2026 16:00:01 GMT

Learn how attackers enumerated Salesforce Experience Cloud and ServiceNow portals — and how defenders can detect and prevent the same abuse.

When Guest Access Becomes an Attack Surface

Modern enterprise portals increasingly expose APIs to unauthenticated users. The problem is not necessarily that those APIs are vulnerable. The problem is that the anonymous identity behind them may have been granted more access than the organization realizes.

Running Sentiment Analysis Inside Neo4j With a Java Plugin

Akmal Chaudhri — Thu, 27 Aug 2026 14:00:00 GMT

In a chapter of The SingleStore Cookbook, there is a complete sentiment analysis pipeline using Rust compiled to WebAssembly and loaded directly into SingleStore via its Code Engine. The result was clean: one CLI command to deploy, sentiment scoring running inside the database engine alongside the data and a full stock-price-plus-headlines analytical pipeline built on top of it.

Can we do the same thing in Neo4j?

Designing Rayfall: One Expression Language for a Columnar Database

Anton Kundenko — Tue, 25 Aug 2026 17:00:00 GMT

Columnar engines naturally organize computation around vectors to make effective use of single instruction, multiple data (SIMD) instructions. This makes vectors first-class citizens in such engines. The difficult design question appears when an engine's internal application programming interface (API) must be exposed to users: where should programming happen?

A native C API is sufficient for embedding, and many engines stop there. Building a complete analytical database, however, requires a full-featured language for programming on top of the engine.

Tail-Based Sampling in the OpenTelemetry Collector: Keeping the Traces That Matter

Mateen Ali Anjum — Tue, 25 Aug 2026 16:00:01 GMT

Head-based sampling makes a decision the instant a trace starts, before anyone knows whether that trace is boring or the one you will spend Friday night chasing. That is the wrong time to decide. At that point the request has not failed yet, and the slow dependency call that will define it is still milliseconds away. Head sampling commits before any of that is visible, so it discards a random slice of exactly the traces you will later wish you had kept.

Tail-based sampling flips the order. It buffers the spans of a trace until the trace is complete, then decides once the errors and timing are actually on the record. The OpenTelemetry Collector ships a tail_sampling processor that does this well. It also has one operational trap that most tutorials skip, and getting it wrong quietly corrupts every decision the processor makes. This walks through a policy set that keeps the traces worth keeping, and then through the trap.

Multi-Account AWS Architecture: Isolating PHI Workloads Without Slowing Down Engineering Teams

Garik H — Mon, 24 Aug 2026 19:00:01 GMT

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data?

The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it.

From Chat Completions to Responses: Why Is OpenAI Upgrading Its Core API?

Jake Tao — Mon, 24 Aug 2026 13:00:01 GMT

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/why-openai-upgrading-api.

If you’ve ever built a large-language-model application, you’ve most likely started with this endpoint:

How to Secure Fintech REST APIs Against BOLA Vulnerabilities

Nanne Parmar — Mon, 24 Aug 2026 12:00:01 GMT

Broken Object Level Authorization (BOLA) occurs when a REST API exposes an object identifier—such as an account, transaction, or loan ID — without verifying whether the authenticated user is authorized to access that specific resource.

To protect fintech REST APIs, implement server-side authorization checks for every object request, validate permissions using the user's authenticated context and resource ownership, and avoid relying on client-supplied IDs alone. Using unpredictable identifiers such as UUID v4 or ULIDs can reduce object enumeration, but they should be treated as an additional security layer—not a replacement for authorization.

Building Meeting Audio RAG on Microsoft Foundry With Fast Transcription and Foundry IQ

Jubin Soni, FBCS — Fri, 21 Aug 2026 19:00:00 GMT

Every recorded meeting your organization has ever held is already a knowledge base. It just happens to be stored in the least queryable format imaginable, which is a wall of MP4 files sitting in a storage account that nobody opens twice. The good news is that the gap between that wall of files and a working question-answering agent is now much shorter than it used to be, because Microsoft Foundry ships the two halves you need in one place. Fast transcription turns the audio into diarized text in seconds rather than in real time, and Foundry IQ turns that text into a permission-aware knowledge base that any agent can query through a single endpoint.

This walkthrough builds the whole thing end to end. By the end you will have a pipeline that watches a blob container for new recordings, transcribes them with speaker labels, chunks them into speaker turns with enough metadata to make citations useful, indexes them as a Foundry IQ knowledge source, and exposes a Foundry agent that answers questions like "what did we decide about the pricing migration in Q2 and who pushed back" with real references back to the moment in the recording.

From Bottlenecks to Reliability: A Practical Guide to Scaling Temporal in Production

Akhil Madineni — Fri, 21 Aug 2026 18:00:01 GMT

Temporal is designed to preserve Workflow state through process crashes and infrastructure failures, but durable state does not remove ordinary capacity limits. In production, the control plane can remain healthy while throughput collapses because Worker slots are saturated, Task Queues mix incompatible workloads, or a failover activates a region without enough Worker capacity. Temporal Workers run outside the Temporal Service and execute Workflow and Activity code, so production scalability depends as much on Worker and routing design as on the service itself.

The Worker Fleet Is Usually the First Capacity Boundary

Schedule-to-Start latency is best treated as queueing delay rather than application execution time. It measures the interval between a Task being enqueued and a Worker starting it. Rising Schedule-to-Start latency, growing approximate backlog, and exhausted Worker task slots indicate that Tasks are arriving faster than the fleet can consume them. Temporal Cloud exposes temporal_cloud_v1_approximate_backlog_count, while SDK metrics expose Workflow and Activity Schedule-to-Start latency and available task slots. Temporal guidance recommends watching these signals together because backlog depth alone does not identify whether the limit is Worker count, Worker configuration, or polling behavior.

Reliability Without Control: Operating SRE Practices in Platform–SaaS and API-Dependent Systems

Oreoluwa Omoike — Thu, 20 Aug 2026 19:00:00 GMT

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority.

Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control.