There was no bug. The code did exactly what I'd designed it to do months earlier, which was the problem. I had stored raw third-party storage URLs in our database and passed them straight to the frontend. Image tags, video tags, email templates — all of them pointed at someone else's domain. It worked in browsers. Then it hit a corporate email client with a strict image-domain allowlist. Our third-party hosts looked like tracking domains, so the client blocked every image.

So we did the work. We evaluated the resources data engineers actually use, judged on five things that matter: relevance to the DE interview loop, depth of practice, realism of the questions, feedback quality, and price. Below is the ranked list.

If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both integrate with ADLS Gen2, both have Spark, and both promise to be your unified data platform. The overlap is real, and the marketing doesn't help.

This post is an honest breakdown of where each genuinely excels, where they overlap, and how to decide without getting lost in feature comparison tables.

Columnar storage stores data in a column-wise format rather than the traditional row-wise storage, offering significant performance benefits for read-heavy operations such as reporting and analytics.

Despite this, API design has long faced a notable gap.

That model still fits a large part of analytics. But it starts to break down when queries read small subsets of data, especially when those reads are repeated. At that point, the cost is no longer dominated by scanning. It depends on how much data the reader must process before it can return the result. That is where comparing Parquet with Lance becomes useful; the difference is not just in file format, but in the read path itself. The Lance paper frames this problem well by focusing on how structural encoding affects random access and scan performance.

In this tutorial, I'll walk through building a production-grade feature engineering pipeline on Azure Databricks using:

LangChain's SQL integration fixes this, translating plain English questions like "Which product category had the highest revenue last year' into valid SQL, executing it, and returning a human-readable answer.

Here, we want to explore one specific problem that makes this high availability subjective. It is an issue where individual layers hold this promise, while as we go to higher-level abstractions, the intelligence silently dies. The article focuses on how topology awareness needs to be preserved mindfully as we move up the stack, and that, when using smart clients and drivers, we should inherit the responsibility not to silence their intelligence.

Five Key Takeaways

1. When your OpenAPI specification becomes the single source of truth, the gap between your API contract and your gateway configuration simply stops existing.
2. Generating the gateway from the spec scales far better than hand-maintaining per-endpoint configuration as your API surface grows into the hundreds.
3. Generated, human-readable service code keeps day-to-day operations manageable — you can read it, reason about it, and trace failures like ordinary software.
4. The genuinely hard part is not the generation; it's the regeneration workflow and the discipline around where custom logic is allowed to live.
5. Adopt the model on new APIs first, prove it's boring and trustworthy, and only then migrate existing ones.

The Quiet Way Gateways Rot

Every public API gateway I've worked with started its life clean and, over a few years, quietly accumulated a second universe of hand-written configuration sitting alongside the services it fronts. None of it looked dangerous at the time. A path rewrite here. A parameter rename there. A response transform to make an internal field look the way customers expect it to. A content-type translation to bridge two teams that made different choices years apart. Each individual edit was sensible, small, and well-intentioned. The danger was never any single change — it was the accumulation, and more importantly, the separation.

That configuration described how the gateway should behave, but it lived in a different place from the thing it was describing: the API's actual contract. Two artifacts, two repositories, two owners, two review processes, two release cadences — all trying to stay in agreement about the same set of endpoints. Anyone who has run a system like this knows how that story ends. The two drift apart. A backend team renames a field and ships their service. The matching gateway mapping doesn't get updated because it's someone else's pull request in someone else's repo. Nothing fails loudly. A customer-facing response is simply, silently wrong. And the place you now have to go and debug is the gateway — the one component that every single request flows through, and therefore the one component nobody wants to touch under pressure.

In these scenarios, simply using the application's identity is often insufficient. The downstream service may need to know which user initiated the request and enforce authorization based on that user's permissions. This is where the OAuth 2.0 On-Behalf-Of (OBO) flow becomes invaluable.

In this post, I'll walk through building an agent system that reacts to streaming user behavior in real time using: