Understanding the "Why" Behind AI's Black Box

Last week, I found myself examining the puzzling output of our newest AI feature—an autonomous decision-making framework for Building Information Modeling (BIM). Despite weeks of careful development, the system was making increasingly strange material recommendations that our construction clients would never implement. This wasn't my first encounter with unexpected AI behavior. Three months earlier, our medical records analysis system had inexplicably begun prioritizing certain diagnostic codes over others with no instruction to do so. And before that, our media content moderation tool struggled with detecting subtle policy violations that human moderators easily identified.

My journey to understand these AI mysteries draws on an unlikely background. In art school, I learned that mastering materials was fundamental—understanding why watercolor behaves differently on hot-pressed versus cold-pressed paper, or why gouache and oils demand entirely different approaches despite both being paint. During my teenage years in the late 80s and early 90s, I worked as a bicycle mechanic during that industry's peak experimental era. I discovered that understanding mechanical systems holistically—how components interact under different conditions—was far more valuable than memorizing repair procedures for specific models.

These experiences fundamentally shaped how I approach technology. Watching engineers troubleshoot AI systems today, I've noticed a concerning pattern: the "change parameters and pray" approach has become the default method across industries. Engineers tweak hyperparameters, adjust training data, and hope for better results without truly understanding the underlying mechanisms—much like someone adding more paint when a watercolor isn't working, rather than recognizing they're using the wrong paper for the medium.

Getting passable results from ChatGPT or implementing the latest open-source model isn't enough. To truly master these systems and maximize their impact, we need to understand why they work the way they do, not just accept "because AI" as an explanation for unexpected behaviors.

The Interdisciplinary Map

My approach to understanding AI systems isn't about becoming an expert in everything—that's a fool's errand. Instead, it's about developing enough understanding across key domains to make better decisions, identify what's going wrong, and know when to bring in specialists versus when to solve problems yourself.

This interdisciplinary perspective is more crucial than ever in navigating the endless sea of AI hype. It provides a framework for separating meaningful innovations from flashy demos that lack substance. Just as understanding bicycle frame geometry and material properties helps you see through marketing claims about the latest carbon fiber component, understanding AI fundamentals helps you recognize when an impressive demo is actually leveraging relatively simple principles.

Let me walk you through the domains that have proven most valuable in my journey to understand AI systems, with specific examples of how each has helped solve real problems across different industries.

Programming and Data Analysis: Following the Flow

Python proficiency and familiarity with libraries like NumPy, pandas, and TensorFlow form the technical foundation for AI work. But more important than syntax is understanding how data flows through your systems—a principle that parallels how an artist must understand how paint flows across different surfaces.

When our medical records analysis tool began making strange recommendations, our initial debugging focused on model architecture. But the real issue emerged when I examined how patient data was being preprocessed. The system was silently dropping certain diagnostic fields when they appeared in specific positions within the record structure—something we wouldn't have caught without understanding the entire data pipeline.

You don't need to be a Python virtuoso, but you should understand enough to trace data transformations and identify where information might be lost or misinterpreted. Just as a bicycle mechanic needs to understand how force transfers through a drivetrain to diagnose shifting problems, an AI practitioner needs to visualize how data transforms through each processing stage.

Mathematics and Statistics: The Language of Patterns

A basic grasp of linear algebra, calculus, and statistics provides the vocabulary to understand what's happening inside model operations. These mathematical concepts are to AI what color theory and perspective are to visual art—fundamental principles that explain why things work the way they do.

When our content moderation system struggled with certain policy violations, I suspected a dimensionality problem in how the model was representing content characteristics. By understanding enough about vector spaces and how embeddings work, we identified that our model was compressing too much information during feature extraction, causing similar but distinct content types to become indistinguishable.

The solution wasn't more data or a more complex model, but a mathematically sound adjustment to our embedding approach. Without this mathematical understanding, we might have spent months on solutions that couldn't possibly work—like trying to fix a bicycle's handling by adjusting the seat height when the real issue is in the frame geometry.

Cognitive Science: Mental Models and Biases

Cognitive science—the study of how minds work—offers powerful insights into AI systems. Many AI models attempt to replicate human cognitive processes, whether explicitly or implicitly.

This reminds me of learning color theory in art school. Understanding how the human eye and brain perceive colors—why complementary colors vibrate when placed side by side, or how atmospheric perspective creates depth—transformed how I approached visual composition. Similarly, understanding how human cognition works transforms how we approach AI systems.

Working with a cognitive scientist, I gained insight into how mental models—simplified representations we create to make sense of the world—influence AI system behavior. Our BIM framework had formed its own "mental model" of construction materials that didn't align with how professionals categorize them. By recognizing this mismatch, we were able to restructure our training data to better align with human conceptual categories.

In our medical records system, we discovered a similar pattern. The model had developed its own internal representation that overweighted recent diagnoses while underweighting chronic conditions—similar to recency bias in human thinking. By recognizing this parallel to human cognitive biases, we implemented a more balanced approach to temporal information.

When you encounter puzzling AI behavior, ask yourself: "If this system were human, what cognitive bias might explain this pattern?" Often, the answer will point you toward a solution.

Philosophy of Mind: The Intentional Stance

Philosophy of mind, particularly theories of intentionality, gives us frameworks for understanding AI behavior. Daniel Dennett's concept of the "intentional stance"—treating a system as if it had beliefs and goals—has proven remarkably practical in my work.

When debugging our media content moderation system, we made a breakthrough by asking: "What would this system have to 'believe' about content policies to make these decisions?" This reframing helped us identify that the model had developed an implicit hierarchy of rules that didn't match our intended policy structure.

The issue wasn't a bug in our code—it was a fundamental mismatch between how human content moderators understand policy priorities and how the model had interpreted them during training.

You don't need to become a philosopher, but understanding these conceptual frameworks gives you new ways to think about what's happening inside complex systems. Just as understanding the principles of mechanical advantage helped me diagnose bicycle brake problems that seemed mysterious to others, philosophical frameworks help me diagnose AI behaviors that seem inexplicable to others.

Complex Systems: Emergence and Self-Organization

Complex systems theory helps us see AI deployments as more than just algorithms. They're adaptive systems with emergent behaviors that can't be predicted by examining individual components in isolation.

This concept resonates deeply with my experience as a bicycle mechanic. A bicycle is more than the sum of its parts—the interaction between components creates a riding experience that can't be predicted by examining the derailleur or brakes separately. A seemingly minor adjustment to one component can transform the entire system behavior. Similarly, a small change in how an AI system processes inputs can cascade through the system, creating unexpected emergent behaviors.

Our BIM framework wasn't just making recommendations—it was participating in and reshaping design workflows. Similarly, our medical records system was subtly changing how clinicians documented patient encounters to match what they thought the AI expected. By recognizing these feedback loops, we implemented monitoring systems that could detect when human behavior was adapting to accommodate AI quirks rather than the system truly improving.

Information Theory and Representational Limitations

Information theory provides precise tools for understanding what our systems are doing at their core: processing and transforming information. Concepts like entropy and channel capacity help us understand fundamental limitations in our systems.

I recently stated that "LLMs are terrible at spatial relationships and things like up vectors"—an assertion that sounds definitive but requires more nuance. The truth isn't as simple as declaring these systems "good" or "bad" at spatial understanding. Just as different art media have different affordances—watercolors excel at atmospheric effects but struggle with opaque coverage—LLMs have specific representational limitations based on how they encode and process information.

The deeper insight lies in understanding why these limitations exist. LLMs process everything as token sequences in a high-dimensional space where spatial relationships aren't explicitly encoded. Unlike humans who have dedicated neural pathways for spatial processing developed through physical interaction with the world, LLMs learn spatial concepts indirectly through language descriptions. This isn't a flaw in design but a consequence of their fundamental architecture and learning process—much like how the properties of watercolor aren't a flaw but a consequence of its water-soluble nature.

When our team debated increasing model complexity for our medical analysis tool, I brought in information-theoretic constraints: "Before we add more parameters, we need to determine whether the patterns we're trying to capture exist in our data." This saved us from building overparameterized models that would memorize noise rather than discover true patterns.

Control Theory: Stability and Feedback

Control theory has proven crucial for maintaining stable behavior in our autonomous AI systems. Through discussions with robotics engineers, I learned about feedback loops, stability analysis, and robust control—concepts directly applicable to ensuring AI systems behave reliably over time.

"Our content moderation system isn't just making predictions; it's part of a feedback loop," I explained to my team. "Users adapt to the system's decisions, which changes the input distribution, which affects future decisions. Control theory gives us tools to ensure this loop remains stable."

This insight led to implementing guardrails and monitoring systems that detect when our AI frameworks begin to exhibit unstable behavior, allowing us to catch potential issues before they impact actual operations. Understanding these principles is similar to knowing how a bicycle's trail and rake affect stability at different speeds—it lets you anticipate problems before they manifest.

Domain-Specific Knowledge: NLP and Computer Vision

For language models and vision systems, understanding the basics of these specialized fields provides essential context. Concepts like tokenization, word embeddings, and semantic similarity helped us improve how our systems interpret text.

In our media analysis tool, understanding fundamentals of computer vision—from image processing to object recognition—allowed us to identify why certain visual content was being misclassified. The issue wasn't with the neural network architecture but with how images were being preprocessed and normalized before analysis.

Just as an artist needs to understand not just brushes but also pigment chemistry and light behavior, an AI practitioner needs to understand not just model architectures but the domain-specific principles that govern different types of data processing.

A Balanced Approach to Learning

My approach to these interdisciplinary fields is always guided by specific problems, not abstract theory. When our BIM system made puzzling material recommendations, I didn't dive into textbooks—I asked: "What cognitive or information processing principles might explain this behavior?" and let that question guide my exploration.

To develop this balanced understanding, focus on:

  1. Programming and data analysis skills: Enough to trace data flow through systems and identify potential transformation issues.
  1. Basic mathematics and statistics: Linear algebra, calculus, and statistics concepts that explain model operations.
  1. Machine learning fundamentals: Core concepts like supervised/unsupervised learning, feature extraction, and model evaluation.
  1. Deep learning basics: Enough to understand different network architectures and their appropriate applications.
  1. Domain-specific knowledge: Basic principles of NLP, computer vision, or other relevant fields.

You don't need expertise in everything, but each area provides a different lens for understanding AI behavior—much like how understanding both materials and techniques gives an artist different ways to diagnose problems in their work.

Beyond the Hype: Finding Your Path

My journey from art school to leading technology teams developing AI systems has been guided by an enduring belief in understanding the fundamentals of any medium I work with. In art school, I learned that mastering materials—understanding why watercolor behaves differently on hot-pressed versus cold-pressed paper—was often more valuable than perfecting specific techniques. As a bicycle mechanic, I discovered that understanding the principles of mechanical advantage and material properties mattered more than following repair manuals.

These experiences shape how I approach AI systems today. If you're working with these systems, here's my advice for developing this interdisciplinary understanding:

  1. Start with problems, not theories. Let specific challenges guide what you learn, just as an artist learns about pigment chemistry when trying to achieve a specific effect, not by memorizing chemical formulas.
  1. Look for patterns across domains. The most valuable insights often come from recognizing how similar principles manifest in different fields—just as understanding tension and balance transfers from bicycle mechanics to structural engineering.
  1. Beware of complexity theater. Many AI systems are presented as more sophisticated than they actually are. Understanding fundamentals helps you see through the smoke and mirrors, much like recognizing when an expensive bicycle component offers marginal benefits over its more affordable counterpart.
  1. Know when to go deep and when to bring in experts. You don't need expertise in everything, but recognize when a problem requires deeper knowledge in a specific area—like knowing when a bicycle's unusual handling requires deep knowledge of frame geometry rather than simple adjustments.
  1. Build a network of interdisciplinary thinkers. Some of my most valuable insights have come from conversations with people outside my immediate field—cognitive scientists, philosophers, and mechanical engineers have all contributed crucial perspectives to my understanding of AI systems.

Getting decent results from ChatGPT is just the beginning. Understanding why those results emerge, predicting when unexpected behaviors will appear, and seeing possibilities beyond the obvious—that's where the real value lies.

This interdisciplinary approach gives you the ability to move beyond surface-level implementation to truly harness the transformative potential of AI.

The future belongs to practitioners who bridge the gap between technical implementation and conceptual understanding—those who refuse to accept "because AI" as an answer and instead seek to understand the fundamental principles that govern these systems.

I hope you'll join me in applying this timeless approach to our newest technological frontier. The view is worth the climb.