Part 3: Understanding AI Model Configurations

As you explore open models, you will see many AI model configurations. These architectural choices determine how a system performs in production.

Modern platforms like LaunchDarkly now allow enterprises to create and manage custom variations of these configurations, enabling teams to restrict specific models at the project level to ensure compliance and architectural consistency across the organization.

Some of the most common differences involve:

• model size

• model architecture, including mixture-of-experts (MoE) designs

• specialization

• numerical precision, like AI model quantization fp8

• context window size

You don't need to be a machine-learning researcher to understand what these mean.

Model Size

When reviewing AI model configurations, you might see:

7B
14B
32B
70B
235B

The B means billions of LLM parameters.

Very generally:

Smaller model
     ↓
less memory
lower cost
faster inference

Larger model
     ↓
greater capacity
more memory
higher cost

But a higher count of LLM parameters doesn't automatically mean better performance on every task.

Dense Models

A dense model uses the entire model essentially when processing information.

Conceptually:

Request
   ↓
Full model
   ↓
Response

For example:

32 billion parameter model

means a very large portion of those parameters participate in inference.

Dense models are relatively straightforward to understand and deploy.

Mixture-of-Experts Models

You may also see names such as:

30B-A3B
235B-A22B

These often refer to mixture-of-experts (MoE) models.

Imagine a company with many specialists:

Database expert
Python expert
Math expert
Language expert
Security expert
Architecture expert

Instead of asking every specialist to participate in every question, a router selects a smaller group. That is the basic idea behind the mixture-of-experts (MoE) architecture.

Question
   ↓
Router
   ↓
Relevant experts
   ↓
Response

A model might contain:

30B total parameters

but activate only around:

3B parameters

for a particular token.

This mixture-of-experts (MoE) approach can provide access to a larger overall model while reducing some of the computational work required for each request.

What Does FP8 Mean?

You may encounter models such as:

Qwen-27B-FP8

FP8 is primarily about how the model numbers are represented in memory and processed by hardware. With FP8 quantization, the system enables more efficient computation.

Models contain billions of numerical values.

You can store those values at different levels of precision.

For example:

FP32
FP16
FP8
INT8
INT4

Reducing precision through AI model quantization to FP8 can reduce:

• GPU memory requirements

• compute requirements

• inference cost

For example:

FP16 model
      ↓
more memory

FP8 model
      ↓
less memory
potentially higher throughput

This does not necessarily mean that the FP8 model was trained from scratch as a different model. It may be an optimized version of the same underlying model.

What About INT4?

INT4 compresses the model even further than FP8. This is another form of quantization that impacts AI model configurations by lowering hardware barriers.

This can make it possible to run fairly capable models on much smaller hardware.

Large server model
      ↓
quantization
      ↓
smaller memory footprint
      ↓
possible local deployment

This can be useful for:

• developer laptops

• edge devices

• private environments

• experimentation

• offline applications

Quality may decrease, so organizations should test the model against their actual workload.

Specialized Models

Some AI model configurations are designed for particular workloads.

For example:

General model
Coding model
Vision model
Speech model
Embedding model
Image generation model

A coding model may be particularly strong at:

• reading source code

• generating code

• debugging

• software engineering

A vision-language model may understand:

Text
+
Images

An embedding model has a very different job.

It converts information into numerical representations that can be used for:

• semantic search

• RAG

• similarity matching

• recommendations

The lesson is simple:

Different AI workloads may require different types of models.

Real-world deployments often use these configurations to solve specific operational challenges. For instance, Priceline uses a configuration involving speech-to-text APIs and WebSockets to power Penny, an AI travel assistant capable of natural, interruptible voice interactions. Similarly, Uber utilizes specialized models to optimize dynamic pricing by analyzing real-time supply and demand data to balance vehicle distribution during high-demand periods.

Context Window

Another term you will encounter is context window size.

The context window size determines how much information the model can work with during a request.

Context might include:

Your question
Previous conversation
Documents
Source code
Database results
Agent tool output
Instructions

A larger context window lets the system analyze much larger documents or codebases.

But bigger is not always better.

Large contexts:

• cost more

• require more computation

• may contain irrelevant information

A good AI architecture gives the model the right information, not simply as much information as possible.

A Practical Model-Selection Framework

Choosing a model is less about finding the biggest name and more about matching the AI model configurations to the job.

Start with six questions:

1. What task must the model perform?
   Classification, summarization, coding, image understanding, embedding, and agent reasoning each favor different architectures.

2. What quality level is acceptable?
   A customer-facing answer may need higher accuracy than an internal draft.

3. What response time is required?
   Real-time chat and batch analytics have very different latency budgets.

4. What data must remain private?
   Private or regulated data may require on-premises or VPC deployment, which affects which precision levels and sizes are practical.

5. What hardware or API budget is available?
   A 70B dense model at FP16 is expensive; a quantized 7B model on existing hardware may be enough.

6. Has the model been tested on representative business examples?
   Benchmarks are useful, but the final decision should rest on your own documents, code, and questions.

Use the answers as filters, not scores. The right model meets the business requirement at the lowest sustainable cost.

Cost efficiency varies wildly even among models with similar benchmarks. For example, as of mid-2026, DeepSeek V4 Flash achieves an 88.8% SWE-Bench score at just $0.44 per 1M input tokens, which is roughly one-tenth the cost of competitors with similar performance. In contrast, flagship models like Claude Fable 5.1 command much higher prices, reaching up to $50.00 per 1M output tokens for top-tier intelligence scores.

Configuration Comparison at a Glance

Common Mistakes

• Selecting by benchmark score alone. Leaderboards measure average performance, not performance on your documents, customers, or code.

• Assuming the largest context window size is always best. Long contexts cost more and can dilute attention; the goal is the right context, not the most context.

• Confusing total LLM parameters with active parameters. A 235B-A22B mixture-of-experts (MoE) model does not use 235 billion parameters for every token.

• Treating AI model quantization fp8 as free. FP8 and INT4 reduce resource use, but they can also reduce quality; test before deploying.

• Using one general model for every task. A coding model, an embedding model, and a vision model will usually outperform a single generalist on those specific jobs.

Organizations often fall into the trap of circular debugging, where they attempt to fix performance issues by cycling through different model configurations without realizing the problem is systemic to the platform. According to Microsoft engineers, warning signs include the AI repeatedly saying “let me try” or proposing increasingly complex solutions for simple tasks. In these cases, reframing the problem or starting a fresh session is more effective than further configuration tweaks.

Key Takeaways

The Complete AI Foundations Series

• Part 1: What Is an AI Model?

• Part 2: What Is an AI Agent?

• Part 3: Understanding AI Model Configurations (you are here)

• Part 4: Choosing the Right Model and Agent

• Part 5: Running Open Models Locally for Specialized Tasks

• Part 6: Local AI in Practice: Email, Files and Document Automation

• Part 7: The Hybrid Strategy: When to Stay Local and When to Go Cloud

Further reading

• AI model configuration override reference — Official documentation on how to programmatically override parameters like temperature and frequency penalties.
• AI functions in Trino — A guide to configuring external LLM providers within data catalogs for textual analysis.
• Common AI Implementation Mistakes — An executive look at data quality and intellectual property risks in generative AI deployments.