Welcome to High Bit, a podcast hosted by Initialized Capital managing partner Brett Gibson about the art of technical problem-solving.

A high bit is the most significant part of the binary representation of a number. In programming language, it is commonly referred to as the most important thing you need to understand about a problem.

We spoke with our guests about just that. In each episode, we’ll break down a gnarly engineering problem and hear how the builder’s ingenuity and inventiveness led to a successful outcome.

&lt;p&gt;Much of AI interpretability work begins after a model has already been trained. Guide Labs is taking the opposite approach: building interpretability into the model from the start.&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;In this episode of High Bit, Brett Gibson talks with Julius Adebayo, cofounder and CEO of Guide Labs, about why post-doc tools struggle to explain models with billions of parameters, and how changing the training process can make AI systems easier to understand, audit, and control without the performance trade-off the field has long assumed.&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;Julius explains how his research challenged the long-held assumption that interpretability and model performance are fundamentally at odds. He also gets into Guide Labs’ move from next-token prediction to diffusion language models, why that architecture may offer greater control, and what breaks when you scale interpretability from hundreds of concepts to tens of thousands.&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;The conversation covers the unexpected engineering challenges behind training interpretable models, from preventing interpretability loss functions from destabilizing training to dealing with duplicated training data, and Guide Labs’ goal of making every output traceable to its context, the concepts influencing it, and the training data behind it.&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;Chapters:&lt;/p&gt;&lt;p&gt;(00:00) Change the way you train your model&lt;/p&gt;&lt;p&gt;(00:59) What Guide Labs builds&lt;/p&gt;&lt;p&gt;(01:56) Why interpretability has been hard&lt;/p&gt;&lt;p&gt;(05:33) Julius&amp;#39;s background: loans, X-rays, and a PhD&lt;/p&gt;&lt;p&gt;(07:04) The trade-off the field assumed was real&lt;/p&gt;&lt;p&gt;(08:01) Why he stopped trusting post-doc tools&lt;/p&gt;&lt;p&gt;(13:51) The three things that made them commit&lt;/p&gt;&lt;p&gt;(18:01) YC, the first models, and a team of nine&lt;/p&gt;&lt;p&gt;(20:19) Why language models were the hardest test&lt;/p&gt;&lt;p&gt;(23:10) Leaving next token prediction for diffusion&lt;/p&gt;&lt;p&gt;(26:41) Why GPT-3 wasn&amp;#39;t just scaling it up&lt;/p&gt;&lt;p&gt;(28:00) 200 concepts by hand, 50,000 you can&amp;#39;t&lt;/p&gt;&lt;p&gt;(31:25) Training lore and &amp;quot;divine benevolence&amp;quot;&lt;/p&gt;&lt;p&gt;(32:33) Setting a minimum bar to keep shipping&lt;/p&gt;&lt;p&gt;(34:33) How open to be about your own techniques&lt;/p&gt;&lt;p&gt;(35:51) The standard dataset full of duplicates&lt;/p&gt;&lt;p&gt;(39:31) Where AI coding tools stop helping&lt;/p&gt;&lt;p&gt;(41:43) What&amp;#39;s next: the model they&amp;#39;re training now&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;Follow Julius and Guide Labs:&lt;/p&gt;&lt;p&gt;X: @&lt;a href=&quot;https://x.com/juliusadml&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;juliusadml&lt;/a&gt; / @&lt;a href=&quot;https://x.com/guidelabsai&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;guidelabsai&lt;/a&gt;&lt;/p&gt;&lt;p&gt;LinkedIn:&lt;/p&gt;&lt;p&gt;Julius &lt;a href=&quot;https://www.linkedin.com/in/juliusadebayo/&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;https://www.linkedin.com/in/juliusadebayo/&lt;/a&gt;&lt;br&gt;Guide Labs &lt;a href=&quot;https://www.linkedin.com/company/guide-labs/&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;https://www.linkedin.com/company/guide-labs/&lt;/a&gt;&lt;/p&gt;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&lt;p&gt;Follow Brett and Initialized:&lt;/p&gt;&lt;p&gt;X: @&lt;a href=&quot;https://www.youtube.com/redirect?event=video_description&amp;redir_token=QUM4Zm9rVFJpeTNseDE3cmZOUG5pRjhuekdOWHxBR3JiS2FtOGR1SVlOT3Qzb1Z3RXRyZHBOaDllVkhteDRNSFhfU29XQ0FQRTVMTFVMR2pTNXl1eHBRZlFWSWxJZWdGX3dRNjV3LUU4d3psZUs5b245eVBrLS1UbGVFb3VPVV82&amp;q=https%3A%2F%2Fx.com%2Fbrettdg&amp;v=aaxjOK_FCHo&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;brettdg&lt;/a&gt; / @&lt;a href=&quot;https://www.youtube.com/redirect?event=video_description&amp;redir_token=QUM4Zm9rUjNOY3lOR09lVzZ0Tl96dm9GUFRWWHxBR3JiS2FuU0VpNGtoRm5OelJXck1saVdBSUc4cUkydHdFY2pIMlBHU0Fzdm44NmNKdFczTEM1UW1KM3JRcnZvMENEVWV3X0loR2xfQUh0aHlWVDhZMEFYUHoyRzE3TXhESy1y&amp;q=https%3A%2F%2Fx.com%2FInitialized&amp;v=aaxjOK_FCHo&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;Initialized&lt;/a&gt;&lt;/p&gt;&lt;p&gt;LinkedIn:&lt;br&gt;&lt;a href=&quot;https://www.linkedin.com/in/brettdgibson/&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;https://www.linkedin.com/in/brettdgibson/&lt;/a&gt;&lt;br&gt;&lt;a href=&quot;https://www.linkedin.com/company/initialized-capital/&quot; target=&quot;_blank&quot; rel=&quot;ugc noopener noreferrer&quot;&gt;https://www.linkedin.com/company/initialized-capital/&lt;/a&gt;&lt;/p&gt;<br />


High Bit

The Data