{"id":111089,"date":"2024-04-11T09:36:57","date_gmt":"2024-04-11T16:36:57","guid":{"rendered":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?p=111089"},"modified":"2024-08-14T10:41:25","modified_gmt":"2024-08-14T17:41:25","slug":"ai-101-what-is-model-serving","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/ai-101-what-is-model-serving\/","title":{"rendered":"AI 101: What Is Model Serving?"},"content":{"rendered":"
\n
\"A<\/figure>\n<\/div>\n\n\n
<\/div>\n\n\n\n

If you read a blog article that starts with \u201cIn today’s fast-paced business landscape\u2026\u201d you can be 99% sure that content is AI generated. While large language models (LLMs) like ChatGPT, Gemini, and Claude may be the shiniest of AI applications from a consumer standpoint, they still have a ways to go from a creativity standpoint<\/a>.\u00a0<\/p>\n\n\n\n

That said, there are exciting possibilities for artificial intelligence and machine learning (AI\/ML) algorithms to improve and create products now and in the future, many of which focus on replicated operations, split second database predictions, natural language processing<\/a>, threat analysis<\/a>, and more. As you might imagine, deployment of those algorithms comes with its own set of complexities.\u00a0<\/p>\n\n\n\n

To solve for those complexities, specialized operations platforms have sprung up\u2014specifically, AI\/ML model serving platforms. Let\u2019s talk about AI\/ML model serving and how it fits into \u201ctoday\u2019s fast-paced business landscape.\u201d (Don\u2019t worry\u2014we wrote that one.)<\/p>\n\n\n\n

What Is AI\/ML Model Serving?<\/h2>\n\n\n\n

AI\/ML model serving refers to the process of deploying machine learning models into production environments where they can be used to make predictions or perform tasks based on real-time or batch input data. <\/p>\n\n\n\n

Trained machine learning models are made accessible via APIs or other interfaces, allowing external applications or systems to send real-world data to the models for inference<\/a>. The served models process the incoming data and return predictions, classifications, or other outputs based on the learned patterns encoded in the model parameters.\u00a0<\/p>\n\n\n\n

Practically, you can compare building an application that uses an AI\/ML algorithm to a car engine. The whole application (the engine) is built to solve a problem; in this case \u201ctransport me faster than walking.\u201d There are various subtasks to help you solve that problem well. Let\u2019s take the exhaust system as an example. The exhaust fundamentally does the same thing from car to car\u2014it moves hot air off the engine\u2014but once you upgrade your exhaust system (i.e. add an AI algorithm to your application), you can tell how your engine works differently by comparing your car\u2019s performance to a base-level model of the same one.\u00a0<\/p>\n\n\n\n

Now let\u2019s plug in our \u201csmart\u201d element<\/a>, and it\u2019s more like your exhaust has the ability to see that your car has terrible fuel efficiency, identifies that it\u2019s because you\u2019re not removing hot air off the engine well enough, and re-route the pathway it\u2019s using through your pipes, mufflers, and catalytic converters to improve itself. (Saving you money on gas\u2014wins all around.) <\/p>\n\n\n\n

Model serving, in this example, would be a shop that specializes in installing and maintaining exhausts. They\u2019re experts at plugging in your new exhaust and having it work well with the rest of the engine even if it\u2019s a newer type of tech (so, interoperability via API), and they have thought through and created frameworks for how to make sure the exhaust is functioning once you\u2019re driving around (i.e. metrics). They\u2019ve got a ton of ready-made parts and exhaust systems to recommend (that\u2019s your model registry). When they install your new system in your engine, they might have some tweaks that work specifically in your system, too (versioning over time to serve your specific product).  <\/p>\n\n\n\n

Ok, back to the technical details. From an architecture standpoint, model serving also lets you separate your production model from the base AI\/ML model in addition to creating an accessible endpoint (read: an API or HTTPS access point, etc.). This separation has benefits\u2014making tracking model drift and versioning simpler, for instance.\u00a0<\/p>\n\n\n

\n
\"A\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Source.<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n

Like traditional software engineering, most AI\/ML model serving platforms also have code libraries of fully or partially trained models\u2014the model registry in the image above. For example, if you\u2019re running a photo management application<\/a>, you might grab an image recognition model and plug it into your larger application.\u00a0<\/p>\n\n\n\n

This is a tad more complex than other types of code deployment because you can\u2019t really <\/em>tell if an AI\/ML model is functioning correctly until it\u2019s working on real-world data. Certainly, that\u2019s somewhat true of all code deployments\u2014you always find more bugs when you\u2019re live\u2014but because AI\/ML models are performing complex tasks<\/a> like making predictions, natural language processing, etc., even a trained model has more room for \u201cerror\u201d that becomes evident when it\u2019s in a live environment. And, in many use cases\u2014like fraud detection or network intrusion detection\u2014models need to have very low latency<\/a> to perform properly.\u00a0<\/p>\n\n\n\n

Because of that, deciding what kind of code deployment<\/a> to use can have a high impact on your end users. For example, lots of experts recommend leveraging shadow deployment techniques, where your AI\/ML model is ingesting live data, but running on a parallel environment invisible to end users, for phase one of your deployment.\u00a0<\/p>\n\n\n\n

Machine Learning Operations (MLOps) vs. AI\/ML Model Serving<\/h3>\n\n\n\n

In reading about model serving, you\u2019ll inevitably also come across folks talking about MLOps as well. (An Ops for every occasion, as they say. \u201cThey\u201d being me.) You can think of MLOps as being responsible for the entire, end-to-end process, whereas AI\/ML model serving focuses on one part of the process. Here\u2019s a handy diagram that outlines the whole MLOps lifecycle:<\/p>\n\n\n\n

\"A\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Source.<\/a><\/figcaption><\/figure>\n\n\n\n

And, of course, you\u2019ll see one box on there that\u2019s called \u201cmodel serving\u201d.<\/p>\n\n\n\n

How to Choose a Model Serving Platform<\/h2>\n\n\n\n

AI model serving platforms typically provide features such as scalability to handle varying workloads, low latency for real-time predictions, monitoring capabilities to track model performance and health, versioning to manage multiple model versions, and integration with other software systems or frameworks.\u00a0<\/p>\n\n\n\n

Choosing the right one<\/a> is not a one-size-fits-all approach. Model serving platforms give you a whole host of benefits, operationally speaking\u2014they deliver better performance, scale easily with your business, integrate well with other applications, and give you valuable monitoring tools from both a performance and security perspective. But, there are a ton of other factors that can come into play that aren\u2019t immediately apparent, such as preferred code languages (Python is right up there), the processing\/hardware platform you\u2019re using, budget, what level of control and fine-tuning you want over APIs, how much management you want to do in-house vs. outsourcing, how much support\/engagement there is in the developer community, and so on.<\/p>\n\n\n\n

Popular Model Serving Platforms<\/h3>\n\n\n\n

Now that you know what model serving is, you might be wondering how you can use it yourself. We rounded up some of the more popular platforms so you can get a sense of the diversity in the marketplace: <\/p>\n\n\n\n