Every business needs to gain insights from its own data, which is where data science and data analytics operations come into play. Business operations have different levels of data maturity, which Joe Reis & Matt Housley describe as “the progression toward greater data utilization and leveraging data as a competitive advantage”. They simplified data maturity into three levels: starting with data, scaling with data, and leading with data.

Starting with data is relatively simple. You create your end-to-end pipelines with a 20-minute YouTube tutorial (Ok, no, but you get the idea). At this stage, you want to move from generation to storage and downstream use cases as quickly as possible. As the business needs robust infrastructure, they begin scaling using formal practices and incorporate some undercurrents of the data lifecycle to finally leverage data as a competitive advantage.

The data lifecycle captures a picture of the data at different points in time and, as it evolves, adapts to support requirements and business operations. Often, you need a panoramic view of your business to answer important questions, such as what type of transformation the data undergoes from the user to our database, and whether there is any corrupted data. How are we protecting our users’ privacy to meet our data privacy policies? What is the company’s specific definition for “Active User”? These are common questions that can take some businesses weeks to answer.

A rule of thumb: the combination of how you answer these questions and how long it takes you to answer them predicts your company’s data maturity level.

For some businesses, this is a dynamic concept that evolves over time. You don’t have to get everything figured out, but you can have a good development process that lets you answer these questions more quickly and easily. Dynamic data maturity doesn’t apply to every business, and that’s the main reason not all businesses scale in the same way: some need to start right from the kick-off at the 3rd maturity level, which could take them longer to consider data as part of the business.

This makes more sense with an example:

A common use case for a healthcare system is patient registration and labeling, a critical component for follow-ups, treatment plans, and insurance policies. Dealing with sensitive data is problematic from a data engineering perspective, but even more so if there is no standardized process or internal guidelines for everyone to follow.

In medicine, there are guidelines with thresholds that change behavior and treatment plans; usually, those thresholds are standardized for the population (ChatGPT won’t do the medical heavy lifting if you don’t specify this context).

AI and data systems in healthcare are only as safe as the context they are given. Knowing what to ask is what makes the difference.

• Bad prompt: “I have 13 g/dL hemoglobin in my blood results. Is that okay?”

• Better prompt: “I am a 35-year-old male, living at sea level, with no chronic conditions. My hemoglobin is 13 g/dL. Is that okay?”

Without the metadata (age, sex, altitude), the raw value (13) is functionally useless for a correct diagnosis.

Defining a patient using a different guideline for the same condition can lead to incorrect assumptions about prognosis and inefficient treatment plans or options. Anemia, for example, means different things for different people. It varies by age group and even by where you live. A software engineer shouldn’t have to be a hematologist to build a dashboard, and a doctor shouldn’t have to be a data engineer to trust the results. The Metadata Catalog is the contract that allows both to be right.

Data catalogs standardize operations and ensure consistent definitions across teams, regardless of background. They enable data innovation while incorporating regulatory and IT requirements through data governance. As explored in Fundamentals of Metadata Management, the catalog acts as the “search engine” for your organization’s truth.

This conceptual “contract” must be physically enforced within your infrastructure. Just as an insurance company needs your diagnosis, but a doctor needs the exact lab values, not all teams require data at the same stage of the lifecycle. The operational side of that metadata contract is where storage meets management. By using a metadata system as a knowledge base, teams can trace transformations, understand fields, and make better business decisions.

Conceptually, you can divide access to your data at different lifecycle stages, such as Dataplex in GCP, which separates data into Raw zones (the triage) and Curated zones (the lab). Ensuring teams have access only to the specific tools and data they need reduces errors, prevents security breaches, and simplifies management.

The Metadata Map: A single dataset exists across multiple dimensions—from technical processes to privacy constraints to end-user visualizations. The catalog is what links these angles into a single source of truth.

Metadata management makes everyone’s job easier. By capturing data in a metadata system and understanding its transformations, you can manage your team effectively. Whether you are starting with the basics or operating in high-stakes fields like finance or healthcare, evaluating your position in the data lifecycle helps you provide informed recommendations. This leads to data-driven decisions that actually impact the business's core operations. Ultimately, a metadata catalog is more than a technical requirement; it is the infrastructure of trust. It ensures that when your business scales, your shared understanding of the truth scales with it.

Resources:

Fundamentals of Metadata Management

Fundamentals of Data Engineering

The main driver here is running pipelines when something happens.

Predefined schedules or cron jobs work better for predictive tasks, like getting the weather every day or preparing the morning news, but this can quickly get noisy. You probably want to know specific things about the weather that change your default behavior.

Let’s see it in perspective: you want to get the weather every day, but that doesn’t drive your behavior. After several iterations, you see that you got tons of weather messages, and that 90% of them you didn’t even look at. This is a clear indication that you need a change of architecture.

Instead, the system might send you a notification when it’s rainy, so you grab your jacket, then you change the system to be more direct and specific. When the weather is predicted to be rainy, it should send you a message to get your jacket. This is actionable: something happens, and the final portion of the pipeline is triggered only because of an important event. Which of these posts are you more likely to stick with, the weather report or the actionable message?

The Benefits of an Event-Driven Approach

There is an event-driven opportunity hidden in almost every scheduled pipeline, but it requires a shift in mindset to be specific and actionable.

While many platforms support these patterns, let’s explore a practical implementation using Google Cloud to illustrate the core concepts.

Imagine you work at a company where employees upload documents every day, such as contracts, invoices, and reports. Every time a file lands, several things need to happen: it needs to be scanned for viruses, indexed for search, archived, and the uploader notified.

You could build this as one giant sequential process:

But what if the scanner is slow? Everything waits. What if you need to add a new step six months later? You modify existing code, risking breaking everything.

Event-driven architecture, combined with parallel solutions, solves this. Instead of one sequential chain, each step happens independently, reacting to the same event:

Adding a new step? Just add a new listener. Nothing else changes.

This sounds like a great system; it captures core Data engineering principles.

Decoupling: Services don’t know about each other
Event-Driven Architecture: React to things that happen, don’t poll
Scalability: Each service scales independently
Observability: Centralized logs across all services
Idempotency: Safe to process the same event twice
Security: Each service has only the permissions it needs

It might sound complicated, but it is the most intuitive way to build this architecture when you look at it in perspective.

Let’s start by setting up the API service in Google Cloud.

We need permission to run Google Cloud by enabling APIs.

The pipeline is simple. We need to ingest data, log that information, validate the schema, and provide a way for computers to communicate with each other. This is where FastAPI and Pydantic came in.

In this code, we specify the schema for the logger and provide a communication channel between the router and our code. Your code also needs libraries specified in the requirements, and everything is packaged in a Docker container.

The Python app has three functions:

• Listens for incoming HTTP POST requests from Eventarc

• Validates the CloudEvent payload using Pydantic

• Logs the file details to Cloud Logging

We have the recipe, now we need the kitchen where everything will run. In this case, Cloud Run.

Here, I want to explain how this works. You have a file, and you deploy this architecture. As you might expect, it needs to run somewhere, and it is definitely not on your machine. Cloud Run is a GCP solution to run anything you want. In our case, we will run a Docker container.

This code does a few things. It defines the project, builds the container image from the files, including the Dockerfile, and pushes it to the Google Container Registry. We now have a working image.

Assuming we already have the cloud storage bucket, we go to the chef of everything!

Orchestrating the Workflow with Eventarc

Eventarc works as a router based on attributes, so rather than sending everything to all systems, it sends only what you need. Eventarc does a few things. First, we enable the trigger that runs on every file upload. It sends the file to the Cloud Run we just created, and it will only trigger when the file is 100% complete and stored in our specified bucket. What if you want to add a new listener six months later? You just need to run the same command again, specifying the destination service.

The pipeline works and scales easily, saving your business costs and hidden fees. In 2026, GCP offers generous free tiers; essentially, the more you grow, the more you pay (which is fair).

Transitioning to an event-driven architecture is one of the most effective ways to optimize data pipelines. By moving away from rigid schedules and focusing on actionable events, you can build systems that are not only more responsive and scalable but also significantly more cost-effective. With modern cloud tools and AI assistance, the barrier to entry has never been lower, allowing you to process thousands of files with high precision for less than the cost of a cup of coffee.

That’s the feeling we all get when learning new things.

The fact is when you onboard to a new project you get handed over API keys for Google Cloud and AWS, gets you the name of the bucket where the bunch of pdf are sitting and you are getting asked to analyze the data as “ML Engineer”, but to analyze the data you need to move the data from those buckets, clean the data, organize it and set a system that updates itself.

This is an everyday thing for everyone that worked in the data industry. In fact, most Data Engineers I know started as ML Engineers or Data Scientists. The reality is that the curated data you have for ML projects is not how real work is done, and it is only a small percentage of the work you will do, especially at the early stages of your career.

If you got into a company that separates responsibilities, the intersection between ML Engineer and Data Engineer is still important.

This 2015 diagram outlines the requirements and needs for a machine learning system to function. Many of the requirements require Data Engineering skills, as they involve managing data and joining pieces together.

Yeah, the code was never the problem.

As you grow in skills and seniority, you may step down from being a full-stack developer, but your communication across teams will be demanded even more, and you need to effectively communicate with other teams, identify blockers, gather requirements, and different aspects of the data lifecycle, which means your data engineering skills become even more important.

Understanding the entire data cycle helps you grow. Data Engineering is one of those skills you shouldn’t miss out on learning if you want to build a long and successful career.

In this post, I will share five books to help you start your AI business in 2026 that address the pain points companies are facing as they integrate AI into their systems.

1. Fundamentals of Metadata Management by

   

The biggest hurdle for AI isn’t the code; it’s the mess. Many organizations don’t even know which apps they can turn off, leading to wasted money and slow progress. This creates a bottleneck that kills AI projects before they start. By mastering Metadata Management, you help them discover their IT landscape, making AI integration smoother.

2. Building Microservices by

Many companies have already started moving away from one big program to many small ones. They’ve made progress, but they're now “juggling” too many moving parts. This makes it hard to focus on AI. Your job with this book is to help them stop making mistakes. By mastering microservices, you help businesses reduce their workload and make room for AI growth.

3. AI Engineering by

There is a massive gap between a “cool AI chat” and a “reliable AI product.” Most businesses get stuck in that gap because they don’t understand the AI lifecycle. This book provides the bridge. It explains how to engineer the systems that surround AI models. By mastering these skills, you can help companies turn their AI ideas into a profitable, working reality.

4. AI Systems Performance Engineering By Chris Fregly

As companies move away from generic tools and start building their own “proprietary” models, they hit a wall: the cost of resources. This book shows you how to break through that wall. By learning to optimize hardware and algorithms, you become a specialist who can help companies scale their AI without blowing their budget.

5. LLMOps by

GenAI models play with different rules; many businesses have abandoned their AI initiatives because the model underdelivers in production. Dealing with systems that serve real customers' needs requires robust monitoring to address errors as soon as they appear. This book helps to manage the LLMs once they are in production.

Bonus: Ok, I said five, but this is an amazing resource that you cannot skip. While the book is in preview on O’Reilly, the core material is available now via their Maven course.

6. Evals for AI Engineers by and

AI Applications need to be evaluated in the context in which they are used. Systematic evaluation enables addressing customers’ needs rather than personal preferences. With this book, you will help companies improve their AI applications to make them reliable and secure for the purposes for which they were built.

AI is taking over the world. Teams are transitioning to AI with the data they have built over the years.

Businesses are starting to ask the hard questions.

• What applications do we have?

• Who owns them?

• What data do we have, and how do we use it in our processes?

• Which technologies support what capabilities?

• Where are our servers located?

• How do we get access?

The above are foundational questions share in the: Fundamentals of Metadata Management book by Ole Olesen-Bagneux).

These questions reflect the organization’s knowledge, speed, and maturity in solving everyday problems.

Managing your organization’s data helps you discover and respond in critical situations.

Metadata provides the answers for business organizations and critical insights. It has different levels that address different issues. Let’s go through each level one by one to understand how each level works, along with a practical example.

Endpoint Management System (EMS)

The EMS lists the servers, desktops, and mobile devices in a company. Yes, every single one—all the servers, all the desktops, all the mobile devices (Fundamentals of Metadata Management; ).

The Endpoint Management System is our first level in metadata management. Let’s explore with an AI Research organization.

You work for a company that relies on GPUs for AI research and development.

Recently, new researchers have joined the team. They need to start working with the right team to develop their skills.

Your manager comes to you with the following problem:

Researchers keep complaining they can’t find available GPUs. We have GPUs scattered across on-premise servers and AWS instances, but nobody knows what we actually have or where it is.

Our latest hire needs the highest-performing GPU server to get started.

Metadata not only can quickly answer questions and provide solutions by querying the data, but it can also direct the researcher to the highest-performing GPU and the software needed to get started.

Integration Repository (IR)

IR describes the nature of the integration between two applications, in regard to what data is being exchanged and how it is being exchanged (Fundamentals of Metadata Management; ).

There are three ways the data travels between applications:

• Stream

• API

• Batch

Each of the three has trade-offs you need to test in your applications.

Let’s continue with our example. How would you solve this problem?

The AI research opens a ticket that says: I have access to the GPU, but I need the integration with the data and the steps to use it. Please, help me with that?

Without metadata, you will have to ask the team how they accessed the data, often in the hope that they can remember the details. But metadata is also a source of cumulative expertise within your organization, leveraging the knowledge gained over years of human effort.

With metadata, the process involves simple steps for you to get the data:

1. Get the Integration Repository.

2. Filter by the technology in use, in our example, the server endpoint and the GPU.

These steps provide the information needed to get started.

Asset Management System

An AMS is used to control the cost of an IT landscape. Typically, the AMS is also used for cost-reduction purposes. AMSs are applied reactively to understand exactly how many instances of a given software application are installed on what devices and servers and if they are actually used (Fundamentals of Metadata Management; ).

The AMS is a reactive system, meaning it scans IT usage across the organization to reduce cost. The picture below is an example of how the AMS data could look:

Stakeholders need to justify the funds allocated to the research team.

They need to test usage, idle time, and the project’s efficiency.

AMS is the most direct link from your organization to AI. You can get reports with action items to work on. Hopefully, in the future, the reports will focus on the solutions that AI agents implemented rather than the problems AI found, as Joe Reis points out.

Today’s analytics tools are still stuck solving problems of the past: visualize what happened, slice and dice dimensions, let a human interpret, make a decision, and (maybe) act. That’s still important, but we can start thinking about how to take this a step further, with teams of hundreds, thousands, or even millions of autonomous agents (The Postmodern Data Stack & Action-Oriented Architecture, ).

Configuration Management Database (CMBD)

The primary purpose of CMBD is to maintain the organization’s existing IT landscape; it does so by documenting hardware and software at the instance level.

On a regular Monday at 9 am, you get an urgent ticket from the research team: the training jobs scheduled to run every two weeks failed, and we don’t have an explanation for why. Please, find a solution?

How much time would it take you or your team to identify the problem and provide a solution? Early morning issues shows why metadata is crucial to managing IT in an organization.

Metadata helps identify the details of the failure, enabling a quick response and solution.

Let’s compare the solution with and without CMDB to see the benefit of having a metadata management system.

Without CMDB:

• Hours/days of investigation

• Checking logs manually

• Asking around, “Did anyone change anything?”

• Trial and error fixes

• Risk of breaking other systems with rollback

With CMDB:

• Check the configuration history for the GPU server

• See: PyTorch updated from v2.0.1 → v2.3.0 on Friday

• See WHY: “Added support for FlashAttention-3 library for researcher_003’s project”

• See WHO else depends on this: 3 other projects using this server

• Make an informed decision: rollback vs. fix vs. isolate

We can see that the issue came from a change in the PyTorch version. We know why we made the change and how a rollback would affect other people’s work.

In this case, the PyTorch update supports the new attention mechanism library. With this information, we can develop solutions that do not disrupt other systems, saving the organization time and money.

Enterprise Architecture Management Tool

The EAM tool is the most high-level and strategic tool for performing IT management. It is intended for strategic decision making about the future state of an IT landscape, not the current state of the IT landscape (Fundamentals of Metadata Management, ).

The core capabilities of the EAM Tool are to manage the organization’s capabilities to look into the future.

In our example, before the problem of training jobs failing during a PyTorch version update, someone could have asked: What will be the consequences of replacing the current PyTorch version? The team could then identify that a training job is running on a specific PyTorch version and avoid the disaster.

Overall, EAM Tool allow to answer these questions about the organization IT landscape:

• What will be the consequences of replacing our current CRM system with this CRM system?

• What are our on-premises dependencies, and how would they affect a cloud migration?

• Why do we have three applications delivering on the same capability, which ones could we most easily descope technically, and what would be the economic benefit? (Fundamentals of Metadata Management,

  Ole Olesen-Bagneux).

Metadata management offers a tool to reduce complexity in everyday problems, from answering the next ticket to planning the organization’s future. Adopting Metadata Management is therefore the most valuable resource for organizations seeking to master the art of answering the hardest business and everyday questions.

Resource:

PD: If you know Python and want to see or play with concrete examples, I wrote a lab in this GitHub repository or Kaggle Notebook that mimics an AI Lab that manages a GPU as its IT.

See my previous post about Metadata Management.

Book Fundamentals of Metadata Management

I like to think of metadata as a “mess solver”; companies collect data every day, teams build systems and projects, and everyone struggles to know where things are and make use of past projects within their organization effectively. This is where the metadata journey begins.

Let’s start by understanding how organizations get to a place where they need metadata:

A team starts with a small project that slowly grows in complexity. As it develops, team members move on, leaving work behind, with no one knowing where things are or what they do. You hope the next hire understands the job and what the other developer did before. Eventually, the organization concludes that a system is needed to see all the companies’ cumulative knowledge.

The realization that storing data that aggregates everything the company has done across different teams and projects is the intuition behind the need for metadata. This is why metadata often comes in many shapes and versions, driven by operational need rather than theoretical knowledge.

To bridge this gap, let’s start by understanding what metadata is at its core level.

What’s metadata?

, in his Fundamentals of Metadata book provides a good definition:

Metadata is a description that is both attached to what is described and placed somewhere else in order to make what is described discoverable and manageable.

Metadata signals information that is located in a different place, making it easier to access and communicate what projects you have, where the data is stored, and how you can utilize it for your new projects.

Let’s see an example to have a concrete use case:

Use case: Education content manager

Take DeepLearningAI as an education company that focuses on data courses. They have several different types of courses, each one of which is for different targets and content:

• Short courses (Less than 1 hour)

• Courses (3-4 modules)

• Professional certificates (3-4 courses)

Imagine you are hired as a curriculum engineer at DeepLearningAI, whose responsibilities include creating labs, quizzes, and the content that students will be evaluated on. Let’s consider how it looks on the first day in the job without metadata and with metadata.

Curriculum engineer at DLAI without metadata

Welcome aboard, you will be working on the following projects:

• TensorFlow specialization.

• Data Engineering specialization.

• Short courses for Fine-tune LLMs.

Here is the link to a GitHub repository of each of those specializations and the short courses. As we add you to more projects, we will provide the links.

These projects don’t have much in common; this is where the challenge begins, as it is likely to get more context than you can find in the repository or your own experience.

DeepLearningAI is recognized for the high quality of its courses, which are curated by a team of experts with extensive technical expertise and domain knowledge in the subjects they teach. Technical teams work closely with subject matter experts to incorporate high-quality labs that are relevant, engaging, and fair.

You were assigned to implement the code that learners will see in the platform to test their knowledge. After working for several weeks, doing the work you were assigned to do with the resources you have, you get this feedback:

Hey, we actually tried something similar in a different course. Please review the lab we did and redo your lab, following this approach instead.

Do you see the idea here? You have a flawed onboarding process, in which the material wasn’t extensive enough to help you succeed in your first attempt.

Let’s make a few assumptions, according to Gemini (for demo purposes only), an hour of a curriculum engineer's time can be around $30, which translates to approximately $1,200 per week.

If a curriculum engineer starts working without direction in their first week, the company loses $ 1,200, which compounds for every new hire, every new lab, and every new project assigned to them.

Twenty-four new enrollments’ revenue ($1200/$49 = 24.49 enrollments) is wasted just because the company didn’t have a system in place to help the operations and onboarding process run appropriately.

You need a system that shows everything the developer needs to maximize the resource with the lowest effort for them. This is precisely where metadata became a business need.

Curriculum engineer at DLAI with metadata

Welcome aboard! You’ll start by working with TensorFlow, Data Engineering, and short courses for Fine-Tuning LLMs.

Here is the metadata about the TensorFlow specialization:

The results: You have a good understanding of each project, you have the prerequisites that can set the boundaries for the course content, and the tags that provide intuition on what the learner will be exposed to. Additionally, you have access to details that help you build learning content, such as library versions, hardware requirements, and connections to other resources that can aid in creating a better product and a smoother initial attempt.

If you have JSON-format metadata for each course and specialization, you can have more control over your workflow and start building more in less time.

The above examples demonstrate a basic understanding of what you can do with metadata and how straightforward it can be to get started.

The metadata concept, as easy as it sounds, doesn’t end here; in fact, it has more nuances than it initially seems. If you embark on a metadata journey, our role is to create metadata that delivers value beyond being a source of static information. Answering questions is just part of the solution; being a framework to grow the company is where the true value is hidden.

The Fantastic Four

Metadata has four main reasons to exist:

The driver: The motivation for building the metadata in an organization.

The purpose: The capabilities needed for your metadata.

The place: The type of metadata store it.

The structure: The information the metadata will have and how it connects to other resources.

Metatada starts with a pain point. What motivates us to add metadata in our organization? The answer to this is usually between three domains:

• Regulation

• Innovation

• Operation

The example of DeepLearningAI is at the operational level, speeding up processes, implementing systematic workflows, understanding what everything is doing within the organization, and taking action based on that.

The driver influences the purpose; in our example, we want to have a system that facilitates communication between different teams and developers simultaneously. The core capabilities we want to include in the example are information and delivering everything that the developer needs to do their job.

Depending on the team, it can take the form of metadata, but it can usually be stored in a spreadsheet or a JSON file in a repository or on Google Drive.

The structure is what information it stores and at what level.

A key thing to keep in mind is that growth usually responds to different needs than the initial drivers, which introduces the need to think of metadata as a strategy that helps coordinate data between teams and projects.

Metadata strategic coordination

The actual value of adding metadata in an organization came from designing a strategic coordination between them, which involves deciding what to extract from a data source, exposing it at a metadata layer, and possessing the technical capability to execute these actions.

Strategic coordination means going beyond delivering information; it involves aligning with the organization's goals and vision, as well as the ability to execute actions based on the data. In our example, it means maximizing the information collected and deciding what to include, but more importantly, what not to include.

The “unfair” advantage that metadata brings to your company enables you to compete in a crowded market with numerous solutions to the same problem, allowing companies to run smoothly in terms of cost and compliance.

In our example, coordinated efforts in metadata help visualize how the company expects to grow, how the individual repository contributes to the goal, and prevent redundant work.

Suppose the goal of DLAI is to reuse labs that have already been prepared by technical teams, reviewed by Subject Matter experts, and extensively tested by community testers to ensure relevance, engagement, and fairness.

From a business perspective, we can assume that the cost of each lab is approximately one week of a curriculum engineer's time, two days of review by a subject matter expert, and one week of community tester coordination by community managers. The cost of producing a lab can range anywhere from $3,000 to $5,000. So, the need for re-usability is not only to avoid duplicates but also a matter of business viability. Having a system that allows for the quick identification of wins and reuse in similar content while maintaining quality is where the value lies.

As we saw, metadata is both a tool and a strategy that helps your team align with the organization's goal and vision, enabling them to manage and provide value. The need for metadata is evident for most companies. However, as a concept is still a blurry idea that managers have in mind based on pain points they see in the organization, but still lacking materialization, when you provide context and see the opportunity cost, it becomes obvious the benefits of using it, and how it compounds in a data system.

Test your organization's metadata maturity:

• What applications do we have?

• Who owns them?

• What data do we possess, and how is it utilized in our processes?

• Which technologies support what capabilities?

• Where are our servers located?

• How do we get access?

Resources:

Fundamentals of metadata management: Chapter 2

My previous post on metadata:

• What applications do we have?

• Who owns them?

• What data do we possess, and how is it utilized in our processes?

• Which technologies support what capabilities?

• Where are our servers located?

• How do we get access?

The core purpose of metadata is to provide quick answers about a company's data, making it easy to retrieve, identify, and understand the context of information. Metadata functions as a super index, streamlining data access and retrieval.

This 'super index' isn't limited to just technical information. A company's knowledge, gathered through years of expertise, is also a rich source of metadata. This 'business metadata', the context, definitions, and rules that employees know, is often separate from the technical metadata found in the IT landscape. A robust metadata management strategy bridges this gap, connecting the technical data with the business knowledge to help a company work more effectively.

Ok, this was a lot, but let’s look at an example.

Let’s consider the healthcare industry. A patient's information might be stored in different tables within a single hospital's database. Metadata helps understand what information is available before they even query the data. But the challenge escalates when that information is fragmented across different medical institutions and specialties; we are still talking about the same person, but their data is scattered. Moreover, each hospital develops its own internal protocols based on experience, local health data, and available supplies. The diversity and dispersion of the data create the need for metadata management.

Just as a doctor needs a complete view of a patient's history and the hospital's expertise, every business needs a comprehensive system to make sense of its data. Having a comprehensive system that helps to answer questions across different data sources is what makes metadata a helpful product. A person can use the data to make informed decisions about what’s missing, how to improve, and where the data is lacking. This provides your team speed, reliability, and accuracy.

The biggest bottleneck in many organizations is the management of their systems as independent units. This creates a siloed environment where each team handles its own part of the system without connection to the rest. Building systems this way leads to unnecessary data duplication, systemic bottlenecks, and a nightmare for the onboarding process.

Returning to the healthcare analogy, medical specialties focus on specific organs, functions, diseases, and the unique needs of individuals. The hyperspecialization of medicine fragments the human as a “monolith service”; the cardiologist will only look at your heart, and the endocrinologist will only look at your hormones. However, we are not a monolithic service; most physicians understand this, but the healthcare system in many places does not. If the cardiologist examines your heart function, and you have an endocrinology problem, you would want those to be connected and evaluated as part of an integrated system.

The same applies to enterprise, the vision of independent data sources that are not connected and built on top of a metadata layer, as another monolith service takes the silo’s implementation of data, which creates redundant and uncoordinated repositories.

Building systems in silos creates unnecessary data duplication, leading to bottlenecks in your systems and a nightmare for the onboarding process.

People and data are the most valuable assets for a company, but without context, data are just noise. Metadata management provides the context to create a unified view of the organization’s knowledge. This provides speed, reliability, and an optimized view of the resources available. Organizations need metadata to transform questions into answers and problems into solutions, driving business decisions.

Resource:

This article was built based on the first chapter of the Fundamentals of Metadata Management by

There are four main takeaways from the course:

1. Learn until you’re dangerous

In today’s world is easy to get lost in the resources available, books, papers, and courses are released every day.

Taking all those resources is overwhelming, but changing the focus to learn enough to build your portfolio or enhance your learning is the key to continuous improvement.

Recruiters are looking for people who identify problems and actively find solutions, so don’t just take courses, build something around it.

The abundance of resources can make anyone feel overwhelmed, not lazy. Achieving clarity on what to learn and why—tying it to practical application, like building projects—is the most effective antidote to feeling lost.

2. Process not output

Learning by building projects is your presentation card, but simply completing them isn't enough. What truly sets candidates apart is demonstrating the process you took to develop. Often, interviewers ask: Why did you choose tool x instead of tool y? This question means “Do you understand what you did?”

It will be disappointing if your answer is: “Oh, no reason, I just followed a tutorial or copied the answer from ChatGPT.”

Following tutorials is ok, but you need to be able to explain the parts that you built and make decisions on them.

To achieve this depth of understanding, even when following tutorials, use your projects as learning tools, understand as many parts as possible, and ask yourself, Why am I using this approach and not another? This will build intuition to answer interview questions.

3. Technical interviews

Once you pass the initial screening process, you get into the technical interviews. We all know Leetcode is not the perfect way to assess coding abilities, but it certainly is the best way we know to assess someone’s ability in 30 minutes.

Companies want to see if you are willing to go through the process of learning complex topics, since you’ll need to in your daily job. Accepting technical interviews as the standard in the interview process can help you a lot in your job applications.

4. Don’t mess up

Preparation – through building projects, mastering fundamentals, and practicing for technical interviews – is crucial, as it directly decreases your likelihood of making mistakes once you're working.

In roles supporting millions of users, errors matter. Companies understand this and therefore look for candidates who demonstrate a deep, practical understanding of systems and how they function. Practicing a high level of competence requires truly grasping concepts and applying them rigorously in your learning and projects, which is the most effective way to prove you have the skills necessary to succeed in a high-impact environment.

Roadmap

With a good understanding of how the interview process works, you need a path to clarify your actions, while balancing between projects, fundamentals, and covering all materials in your career path. It might sound like too much, but the bar is high for anyone who wants to have a position in tech.

After reviewing Susan's interview path on the job-seeking process for ML, we can define a four-step approach to build a curriculum.

Step 1: Defining your role

This matrix from the Machine Learning Interviews book helps you to test your current skills against your desired job title.

Note: Currently, all work related to Large Language Models falls under the role of Machine Learning Engineer.

The scale uses 1 (not very important) to 3 (very important) in expertise. You can set it based on your confidence level in each section. Working with tools without practicing the fundamentals might require going back and studying.

Step 2: Identify Challenges in Your Interview Process

Understanding how your interview process works can help you determine where to focus your practice sessions. Consider how many interviews you’ve had in the past year and what percentage of your applications resulted in initial interviews to guide your next steps.

For instance, if you’ve had five interviews in the past quarter but consistently struggle with LeetCode questions, this is an area where you need to improve. On the other hand, if you master LeetCode but have problems getting the initial interview, your resume, portfolio, and online presence this is where you should focus your efforts.

If you have a solid understanding of the fundamentals and have received numerous interview invitations but still fail when asked to code something complex, such as a YOLO model from scratch or implement multi-head attention, this indicates a need for targeted learning.

In summary, assess your position in the machine learning interview process and target your learning approach accordingly.

Step 3: Fill in the gaps in your knowledge

The template below suggests an identification skill gap based on the things that matter the most to target your learning approach.

Let’s see an example of a Machine Learning Engineer

I chose four main areas to improve:

• Data wrangling

• Machine Learning (Theory and algorithms)

• Software engineering

• Multivariable calculus and linear algebra

If you want a position as an AI engineer, it includes everything above as a must-have skill, plus working with LLMs.

By understanding yourself and the skills needed for your desired role, you can target your learning approach on the areas with the biggest Return on investment.

Conclusion

Drawing from Susan Shu Chang's insights, we've evaluated the core principles for mastering the ML Interview process: learning with a clear, project-driven purpose, understanding why you make technical choices, recognizing the importance of the industry's technical assessments, and building the competence necessary to avoid costly errors on the job.

Applying these principles requires more than just effort; it demands a targeted strategy. With this roadmap – defining your ideal role, honestly assessing your current skills and interview performance, and pinpointing the specific areas where your knowledge or practice falls short – you transform overwhelming preparation into a clear, actionable plan.

This focused, self-aware approach provides the clarity and direction you need for maximizing your learning efforts. To illustrate how to put this targeted strategy into practice, my next post will share a detailed curriculum, resources, and a schedule specifically designed to help you fill the identified gaps for AI Engineer roles.