YouTube

41 slides extracted. Slide 1 — 0:26 (watch) Hello, everyone. I'm Audry, and I work at RunPod. Were any of you in my earlier session? Great! The introduction will be the same, but what I’m going to present is a bit different. Has anyone heard of RunPod or used our services before? If so, could you share how you’ve used us or how you heard about us? Slide 2 — 0:40 (watch) I found my university has some RunPod grids, which I use for LLM training. Slide 3 — 0:54 (watch) LLM training at your university? Where do you go to school? Oxford. I studied abroad there one summer; it's an amazing place. And your name is? Eunice. Slide 4 — 1:42 (watch) Eunice, I'll provide a brief introduction. We are an AI cloud infrastructure company with a mission to build the foundational platform for developers to scale their AI workloads. This means we supply the hardware, including GPUs and compute resources, making it easy for you to bring your code and models and deploy them quickly. We aim to eliminate the time spent on configuring infrastructure and scaling concerns. Many teams we’ve spoken to struggle with infrastructure, often spending more time on it than on their models. Issues like CUDA version alignment, compatibility of PyTorch versions, and testing new GPU SKUs can be burdensome. We strive to alleviate these configuration challenges so you can focus on training your models or building your applications. A bit of backstory: our founders, Zen and Pradeep, started RunPod in 2022 after a failed crypto mining venture left them with spare GPUs in their basement. Slide 5 — 2:38 (watch) They built a prototype that became the foundation of RunPod and posted on Reddit, asking if anyone wanted free GPUs in exchange for feedback. That is how our company started. Since then, we have been building in public with the community and generating revenue from the very beginning, which is quite rare. Slide 6 — 3:16 (watch) Today, we have around 500 developers on our platform and operate in over 30 data centers across 10 countries. In Europe, this includes France, Romania, and Iceland. Slide 7 — 3:38 (watch) In the Asia Pacific region, we recently achieved a significant milestone of 120 million in annual recurring revenue. Here is a quick overview of some of our customers. You may be surprised to see that some are AI-native companies alongside large enterprises. What they all have in common is a need for flexible and reliable GPU infrastructure. I would say we are definitely punching above our weight class. Slide 8 — 4:30 (watch) There are different ways to build on RunPod depending on your needs. If you require a persistent VM environment, using pods is a great option. You can rent a pod on demand, paying by the second, and tear it down when you're done. Pods provide reserved GPU access; as long as your pod is running, the GPU is exclusively yours. For serverless deployments, if you prioritize scaling due to variable workloads, we automatically scale your workers up and down based on demand, ensuring you don't pay for idle time. Clusters are ideal for multi-node training, and the hub allows you to deploy pre-vetted open-source AI repositories for popular models like Comfy UI, Stable Diffusion, and VLLM, making it easy to explore and get started quickly. Slide 9 — 5:30 (watch) Today, I will discuss serverless technology and the product we just launched. I will quickly switch my displays to mirror my screen. Slide 10 — 5:50 (watch) One significant challenge for developers during the iteration or development phase is the process of testing code related to their inference model. Typically, this involves making a commit, pushing it to GitHub, building a Docker image, pulling it from the container registry, and then loading it onto a server. After that, a GPU must be allocated, allowing for testing to see if everything works as expected. This cycle repeats until the developer is satisfied with the results. Slide 11 — 6:30 (watch) You then load it onto a server, allocate a GPU, and test to see if it works as expected. This process repeats until you are ready. Flash, our Python SDK, aims to eliminate this iteration cycle, allowing you to deploy your function on a GPU directly from your local development environment. I'll zoom in here quickly. Slide 12 — 7:30 (watch) Flash allows you to deploy your regular async Python function to a GPU cloud by simply adding our Flash endpoint decorator. This process packages everything within your function for the cloud, while your main function and any helper functions continue to run in your local development environment. If you require GPU compute, it will be handled in the cloud. Additionally, we support file reloading, so any changes made to your application are immediately repackaged and pushed up, enabling rapid testing and iteration. For example, I have a function called generate_image that loads PyTorch and a pre-trained stable diffusion model, specifically stable diffusion XL turbo, which is excellent for fast image generation. The function saves the image and returns it as base64 encoded. I can run this now since I've installed all my dependencies and set up a Flash project. I will execute the command `Flash run image_generation.py`. Slide 13 — 8:22 (watch) I will run the image generation script in Python. I have a small setup ready for this. Slide 14 — 8:38 (watch) Flash Run spins up a local development server, which is a FastAPI server. I can send my request to this endpoint. Slide 15 — 8:56 (watch) I'm going to do that quickly and get to my project. Slide 16 — 9:28 (watch) This is a helper script that sends a POST request and decodes the generated image so you can see what it looks like. The image generation is asynchronous. Let's pass a prompt to it. Can I get some help from the audience? What should we generate today? Anything random will do. How about cats flying in the sky? What does the sky look like? What time of day is it? Slide 17 — 10:28 (watch) Flying on a cloudy day in the sky somewhere in London. I passed the prompt correctly, and he's helping me debug live, which I appreciate. I passed the URL, and I must have had an issue with my HTTP. There we go. Slide 18 — 11:00 (watch) Let's return to the local development server. Slide 19 — 11:30 (watch) The request has been received, and the job has started and been queued. While we wait for it to finish, let’s focus on the endpoint decorator. This is where the key functionality occurs. I provide a name for my endpoint and specify a GPU family, such as the ADA or AD pros, which are variations of NVIDIA H100 cards. I can also set the maximum number of workers to five, allowing up to five instances to run simultaneously. Slide 20 — 12:28 (watch) I set one active worker, which is always running. It didn't take my prompt, likely because I didn't pass it as a flag. Now it's generating images of cats flying. Regarding the endpoint decorator, there are various configurations for timeout, which determines how long a worker remains idle. The output looks abstract, and while I'm not from London, perhaps someone can tell me if this resembles a London chimney. Slide 21 — 13:28 (watch) I don't like what just happened, so we're going to switch out our model. I'll comment out this code here and swap in DreamShaper, which is a fine-tuned model based on StableDiffusion 1.5. This model is optimized for quick generation, but I believe it will produce a better quality image, especially for art and illustrative styles. We've adjusted some parameters, increasing the inference steps to 25. The height and width will be set to 10 by 24, which is fine. Slide 22 — 14:12 (watch) Let's send the same request again and see what happens. The result is different. The speed of this process comes from not having to make a code change, commit it, rebuild my Docker image, upload it, and allocate GPU infrastructure. Instead, everything is happening right here in my IDE, allowing me to stay within the environment. Slide 23 — 14:34 (watch) This is beneficial, right? We all agree on that? Slide 24 — 14:50 (watch) One last aspect to highlight is the significant impact of using a developer tool like Flash, especially when you're not just making a single call to one model. It's about the orchestration code surrounding it. I have a pre-prepared pipeline to demonstrate this. Slide 25 — 15:06 (watch) Instead of generating and writing out every prompt manually, this pipeline will send a request to Gwen, which is already hosted on a public endpoint. Slide 26 — 15:20 (watch) Gwen3 will generate all the prompts for me. After that, it will send the requests to our DreamShaper running on our endpoint. There is one more pipeline that the requests will go through. Slide 27 — 15:36 (watch) It will send the request to NanoBanana2, a premium Google model known for its ability to compose images effectively. I hope to create some impressive pictures of our founders and share them with them after this demo. Slide 28 — 15:50 (watch) Now let's run the entire pipeline. Slide 29 — 16:26 (watch) Prompt: Two men walking in London on a cloudy day. Close-up of their faces. Any other requests for these two men? How do they look? Are they doing something? Yes, two men with glasses. Close-up of their faces. Now, let's generate three of those and compose them together. Slide 30 — 17:02 (watch) The pricing model is based on the duration of each request. You are charged only for the time that the request is actively running. Slide 31 — 17:12 (watch) I mentioned that this session would be entirely in the terminal, but I will switch back to the console briefly to show you what is currently running. Slide 32 — 17:38 (watch) This is the endpoint we created from the terminal. We provisioned five workers, and currently, three are running because I requested three photos. This indicates uptime, which reflects your charges. The cost of an H100 is currently $0.00116 per second. Slide 33 — 18:16 (watch) Pricing differs between serverless and pods. With pods, you don't benefit from scaling, which results in a premium for serverless options. We recommend starting with a low worker count or using pods if you're still experimenting. During experimentation, you may only need one or two GPUs at a time. Serverless is ideal when you require hundreds of workers running on numerous GPUs, distributed for improved availability across different data centers. Slide 34 — 18:38 (watch) Here’s our final presentation. Slide 35 — 18:46 (watch) This was our original prompt: two men with glasses walking in London on a cloudy day, with a close-up of their faces. Slide 36 — 18:58 (watch) On the left, you can see what DreamShaper generated based on the prompt engineering that Quen3 developed for us. This prompt is significantly improved compared to the one I initially provided, incorporating much better cues. I can read it out to you since it might be difficult to see. Slide 37 — 19:14 (watch) The image features thoughtful expressions and weathered faces, with a soft focus on the background clouds. The muted urban palette includes grays and deep blues under overcast lighting. On the right is the final composed photo. Slide 38 — 19:28 (watch) This is a striking picture of Pradeep, and it appears to be an old photo of Zen. Slide 39 — 19:40 (watch) I will scroll down to show you the reference photo that I sent. Slide 40 — 19:44 (watch) Overall, I wanted to demonstrate how you can get started quickly. Slide 41 — 19:56 (watch) You can start in your local development environment using open-sourced models or by bringing your own private model. This was a really enjoyable experience. Thank you all for joining me.

48 slides extracted. Slide 1 — 0:12 (watch) Welcome. I'm Matt, and this is Sunil. We work on agents at Cloudflare. Slide 2 — 0:26 (watch) I'm Matt Carey. I tend to hire my friends for our team. Please say something to introduce yourself, and make sure to speak into the microphone. Slide 3 — 0:32 (watch) Is my microphone on? Slide 4 — 0:38 (watch) I view Cloudflare as a friends and family company. We enjoy doing favors for our closest friends. No one pays for Cloudflare services, which are already very affordable. Slide 5 — 0:46 (watch) We began developing AI agents in earnest a little over a year ago, and the progress has been remarkable. How many people here are familiar with orange-billed, referring to durable objects? Slide 6 — 1:10 (watch) The technology has a poorly chosen name, but it is remarkable. Many people refer to it as a database, but it functions differently, incorporating database features. We began using it, and it proved to be very effective. It offers unique capabilities that some may view as vendor lock-in, while others may see them as innovative. Slide 7 — 1:34 (watch) Since then, the team has grown, with most of us based in London. However, one of our team members recently moved to Lisbon, which is fantastic. They enjoy their days drinking wine and eating fish, potatoes, and bread. I live in Newcastle, where the weather is much worse, but I do enjoy it. Slide 8 — 2:16 (watch) I tried watching Geordie Shore, and I regret it. Let's get back on topic. We've been building agents, and I can share our journey so far, the technology we're using, and some upcoming features. Thankfully, none of this is recorded, so I can discuss some secret features that are coming out. I’d love to open the floor for questions for a couple of minutes. Otherwise, we have five topics we thought would be interesting to discuss. What would you like to know about Cloudflare and AI agents? It starts with durable objects. Everyone believes that building servers today involves writing a function that takes a request and returns a response, which is the simplest model. For example, if you want to build a hit counter for your website, you might initialize a counter variable to zero. For every request, you would increment the counter. This works locally, but once deployed, it fails immediately. Why does it fail? There’s no state; the function spins up, performs its task, and then disappears. As a result, the counter will always be zero most of the time, although it might reuse the isolate under heavy traffic. ...

44 slides extracted. Slide 1 — 0:08 (watch) I'm Nupur, and I work with Qodo. Slide 2 — 0:18 (watch) At Qodo, we conduct agentic reviews. Slide 3 — 0:26 (watch) I have a background in DevSecOps, coming from an industry where everything is deterministic. In that environment, pipelines run and if they crash, we fix them. Now, I am working with agents in a space where nothing is deterministic. Slide 4 — 0:42 (watch) In the past few years, I have learned where and how agents fail and what insights can be drawn from these experiences. Today, I will share some of these learnings with you. Slide 5 — 0:58 (watch) The evolution of agents began with static prompts, which utilized a 4K context window. We aimed to include everything we considered important for the AI models to process and generate results. Slide 6 — 1:40 (watch) Initially, it was our responsibility to instruct LLMs on what to focus on. If we provided incorrect inputs, the results would be inadequate. We then considered that increasing the context window size could improve outcomes by allowing for more inputs, leading us to develop agentic workflows. We created agents equipped with tools, such as a search tool, to explore documents and execute commands. However, this created a loop where the tool continuously sought more inputs, unsure of when to stop. To address this, multi-agent systems have gained popularity, enabling multiple agents to collaborate on tasks. For instance, a security agent can identify security concerns, a review agent can evaluate tools, and a coding agent can address issues. Slide 7 — 2:26 (watch) The more tools you have, the more issues arise. Not every agent understands the context, leading to clashes in their interpretations, which prevents you from achieving the desired results. Slide 8 — 2:56 (watch) What can we learn from this? Context itself is not the issue. Models are increasingly capable of handling large amounts of context and data. However, this does not guarantee that the results will be intelligent or that they will prioritize important information. Current LLM models exhibit a pattern where they consider the initial and final inputs but often disregard the context in between. They focus on the starting and ending points to generate results, neglecting the intermediate context. Slide 9 — 3:48 (watch) This resembles a U-curve where elements from the beginning and end are relevant, but the in-between context is overlooked. We are actively working on this issue and benchmarking our findings. When we create agents, we assess whether the provided context influences the results. We are employing a multi-agent architecture for various tasks, such as code reviews. For each task, we assign it to an agent and request the outcome. During code reviews, we explore whether we can provide the entire context, such as the complete codebase, to improve results. However, we observe that the initial prompt or goal remains the primary focus throughout the process. Slide 10 — 4:24 (watch) When we provide input at the end, that input is in focus. However, the surrounding context, such as JIRA tasks or MCPs, can complicate matters. LLMs attempt to eliminate these extraneous details and make sense of the information independently. Slide 11 — 4:54 (watch) To address this issue, we create strategic solutions for context optimization instead of simply dumping all information into the models and expecting them to discern what is important. We begin by evaluating how to improve the context for the model. Currently, many solutions exist, and "context engine" has become a buzzword; everyone wants to develop one. However, a context engine functions like a bouncer. It guides the high-speed car, indicating what is more important. Slide 12 — 5:48 (watch) Creating a context engine is beneficial for managing a large, messy codebase. It establishes a search pattern and ranking logic, allowing the system to prioritize tasks based on importance. While indexing requires moderate effort, scaling becomes challenging with hundreds of repositories, as mapping and indexing can slow down and make it unpredictable to develop an effective context engine. Instead of heavily investing in a context engine, there are alternative methods for agents to gain context. One such method is hierarchical summarization, where summaries are generated for each file and folder. This allows agents to quickly assess the relevance of content without needing to sift through everything. Slide 13 — 6:48 (watch) Implementing this approach requires significant LLM processing. Each time a file is created or modified, some agents must generate a mapping for that change, resulting in a high upfront demand for LLM processing. An alternative is to use a knowledge graph. While knowledge graphs can be complex, they are highly effective for managing logical dependencies. For instance, if one file affects another, which in turn impacts additional files, you can create a graph to represent these relationships. However, the initial input required from developers is substantial, and it can be time-consuming to establish. Nevertheless, when dealing with complex logic or dependencies across multiple repositories, this method proves to be very effective. Slide 14 — 7:24 (watch) For most tasks, especially if you're not a product company but are building agents for your own processes, iterative retrieval is highly effective. Instead of creating a summary, it generates an index, similar to a library card, which you provide to your agents to indicate the topic. Slide 15 — 7:40 (watch) If that resonates with you, you can examine the code to find the results. While there are significant cost implications, you do not need to invest a lot of energy. Slide 16 — 7:56 (watch) The input required from developers for LLMs is minimal, yet it yields better results. Additionally, there is an option for self-correction, where the LLM performs a task and a critic node evaluates its relevance to the initial goal. Slide 17 — 8:16 (watch) If the context is lost, you can ask the agent to retry the task, as indicated by the critic node. This process introduces some latency due to the need to rerun the agents, but it does not require significant initial input from developers to create a solution. Slide 18 — 8:52 (watch) Another challenge that arises when creating these agents is the orchestration paradox. As LLMs become increasingly sophisticated, they tend to overthink their tasks. Instead of focusing on solving the problem, they spend time considering which tools to use and researching better methods. This leads to a cycle where they jump from one approach to another, wasting API tokens on exploration rather than execution. Consequently, they enter a research mode instead of actively addressing the problem at hand. Slide 19 — 9:26 (watch) For example, when using Opus, users often seek the best method repeatedly, challenging themselves to find alternatives. This results in a loop of trying different approaches rather than taking action. Slide 20 — 9:56 (watch) To resolve this, we employed an 80-20 hybrid approach, which I believe is one of the most effective solutions to the infinite loop problem. Our teams allow the agents to research for 80% of the time, giving them the freedom to explore and innovate. However, for the remaining 20% of the task, which requires final validation and summarization, we impose stricter guidelines. This portion is more deterministic and involves clear criteria that must be met. Slide 21 — 10:22 (watch) For example, if I receive X results, I want Y. This approach is more deterministic, allowing us to narrow down the research from the 80%. Slide 22 — 10:52 (watch) The 80% tool can still enter infinite loops, but we have mechanisms to address this. Some organizations implement counter mechanisms, where after four or five iterations, they must proceed with the last results. Others use timeout counters, where after five minutes, they work with the most recent decision and can revisit it if the results are unsatisfactory. While you can limit the 80%, it is important to note that for tasks like discovery or tool selection, the 80% research models are quite effective. However, when it comes to summarization or evaluating the research obtained, the focus shifts. Slide 23 — 11:46 (watch) To generate a result, the 20% models are highly effective. Typically, the 80% models require high reasoning capabilities, but this is not necessary for the 20%. The 20% tasks are deterministic and focus on delivering what is needed. For instance, the critic node we discussed does not require extensive research or analysis; it simply needs to understand the goal and the desired outcome to provide a summary. Additionally, when considering the next possible action based on the results, the 80% dynamic models can help determine what to explore next. However, it is essential to identify the proper results that the user is seeking from the 80% models. Slide 24 — 12:18 (watch) Those kinds of decisions can be made by the 20% model. Slide 25 — 12:46 (watch) As the context grows, teams often believe they can accomplish everything with a single agent due to the large context window. They assign various tasks to the agent, such as testing and reviewing, thinking that the shared context will yield comprehensive results. However, this can overwhelm the agent with inputs, causing it to lose sight of the original task. For example, if you assign four tasks, the agent may excel at two while neglecting the other two. This leads to great results for some tasks, but others may be overlooked entirely. Slide 26 — 13:46 (watch) We have a concept called mixture of agents, which is where you hear a lot of buzz about multi-agent architecture. Instead of creating one large agent, we develop specialized expert agents that excel at specific tasks. Each agent generates its own ideas or results, and we need to ensure that these results can be combined coherently. For example, if I ask one agent to find the best hotel, another to find the best location, and a third to find the best flights, I might receive conflicting results. One agent may suggest a hotel in Greece, while another finds a flight from Amsterdam to Portugal, leading to a situation where the results do not align logically. Slide 27 — 14:22 (watch) For this purpose, there is a concept called a judge agent. Its role is to gather all the results and determine if they can be integrated coherently. Slide 28 — 14:34 (watch) Now, you are leveraging the strengths of different agents to obtain the best results from each. The judge agent helps combine these results into a coherent whole, rather than presenting a collection of disparate outputs that do not make sense together. Slide 29 — 14:42 (watch) We have implemented a similar approach. Slide 30 — 14:54 (watch) This is our architecture, Qodo's architecture, where we use the same formula for code reviews. Slide 31 — 15:12 (watch) As part of a PR review, we have a context collector that gathers context from the PRs, the context engine, and various tools. However, it does not immediately begin the review process. Instead, it bifurcates all the collected context and passes it on to different agents. Slide 32 — 15:28 (watch) These agents specialize in their designated tasks. For example, one agent focuses on identifying security flaws, another on coding, and yet another on finding Jira issues. Slide 33 — 15:44 (watch) Once all these agents return their results, a judge agent evaluates them to determine relevance. It assesses which of the provided results are interesting and checks against the context engine, reviewing the pull requests to identify how many of the ten items actually make sense for your needs. Slide 34 — 15:56 (watch) Refining the results to ensure they are relevant to you. Slide 35 — 16:04 (watch) That concludes my presentation. Thank you. Slide 36 — 16:14 (watch) Any questions? Yes, how do you enable communication among the swarm? Are you referring to the agents? Slide 37 — 19:24 (watch) You have agent A, agent B, and the judge. They communicate through a tool we developed using LangChain, which facilitates interaction and infrastructure for different agents. LangChain collects responses and integrates them into the prompt for the next agent. We aim to gather results and create a refined prompt for the subsequent agent. If there are multiple inputs, we have an agent dedicated to collecting results and enhancing the prompt for the next agent. Regarding calibration, when conducting a code review with an agent, it is essential to establish what constitutes good and bad practices. We perform calibration by assessing the context we have. For instance, when receiving code reviews, the LLM does not inherently know what is significant for your specific workflow. Different industries, such as healthcare, retail, and finance, may utilize the same Java framework in varied ways, with different aspects being crucial for each. ...

75 slides extracted. Slide 1 — 0:04 (watch) Hello, I'm Tatiana, a Research Scientist on the MLX team. Slide 2 — 0:20 (watch) It's been a remarkable time for local LLMs. Models are continually increasing in size and gaining impressive capabilities, becoming smarter and tackling more challenging problems. As they improve, we leverage them for longer contexts, more complex tasks, and intricate workflows. Eventually, limitations arise in memory, compute, or bandwidth on a single machine. Slide 3 — 0:38 (watch) In our video, "Run Local Agentic AI on the Mac using MLX," we demonstrate how to run AI agents locally. Slide 4 — 0:52 (watch) With multiple devices, you can enhance local AI by running larger LLMs or accelerating them through distributed inference and training. Today, we will explore scaling across multiple Macs with MLX using the hardware available on your desk. Slide 5 — 1:08 (watch) We will begin with the command line interface to run models on your machines, then move to the Python API for experimentation, and conclude with Swift for embedding these workflows directly into your apps. Let's get started. Slide 6 — 1:20 (watch) First, we will examine the complete hardware and software stacks that enable distributed workloads on Apple Silicon. Slide 7 — 1:30 (watch) Next, we will combine everything by turning four M3 Ultras into a cluster. We will walk through each step, selecting the appropriate topology to connect the machines, enabling fast communication, and launching distributed jobs. Slide 8 — 1:46 (watch) Once the cluster is ready, we will focus on fast and local distributed LLM inference and fine-tuning. We will run it with MLX, compare it side by side against a single Mac, and examine how MLX distributes the model across the cluster. Slide 9 — 2:02 (watch) Most examples will be presented in the command line interface. In the end, we will demonstrate how distributed communication is also accessible through Python, Swift, and C++ APIs. Slide 10 — 2:12 (watch) Let's begin by examining distributed communication for Apple Silicon. Slide 11 — 2:16 (watch) To send and receive data quickly, machines must be connected through a physical link and interconnect. Slide 12 — 2:32 (watch) We also need a transport protocol, which is a mechanism that transfers bytes from one machine's memory to another's. Starting in macOS 26.2, Remote Direct Memory Access Protocol, or RDMA, is supported over Thunderbolt 5. RDMA enables data to move directly from one machine's memory to another's, minimizing CPU and operating system overhead. Slide 13 — 2:56 (watch) RDMA over Thunderbolt provides the high bandwidth and low latency communication necessary for distributed workloads. However, it only facilitates raw data movement between two machines. Slide 14 — 3:12 (watch) Distributed programs require a higher-level communication backend that provides communication primitives for sending data between individual machines and coordinating across the entire group. These operations are essential building blocks for distributed training and inference. This is where Jackal comes in. Slide 15 — 3:34 (watch) Jackal is an open-source collective communication library developed by Apple. It leverages RDMA over Thunderbolt and provides collective communication primitives for sending data between machines and combining results across the group, all without requiring you to manage the low-level transport. Additionally, it is not limited to machine learning; any distributed workload on Apple Silicon can utilize it. Slide 16 — 3:52 (watch) The final piece of the stack is a machine learning framework that utilizes the communication backend for distributed inference and training: MLX. Slide 17 — 4:10 (watch) MLX is an open-source machine learning library developed by Apple for Apple Silicon. It utilizes Jackal for low-latency distributed communication and offers tools for orchestrating distributed jobs across the cluster. If you are new to MLX, we recommend watching our video "Getting Started with MLX on Apple Silicon" from WWDC25. Slide 18 — 4:26 (watch) Now that we understand the full stack, let's put it all together and build a cluster, which is a group of machines working together on the same task. We will use four M3 Ultras. Slide 19 — 4:40 (watch) To set up the cluster, we need to connect the machines using Thunderbolt 5 cables. There are various ways to wire them together, and the topology directly affects communication time. First, let's examine the factors that define that time. Slide 20 — 4:52 (watch) Next, we will examine how to connect the machines, the topologies supported by Jackal, and the trade-offs associated with each topology. Slide 21 — 5:00 (watch) Next, we will demonstrate how to enable RDMA on the machines for fast communication. Finally, we will launch distributed jobs on the cluster using MLX. Slide 22 — 5:28 (watch) Communication time consists of two components: latency and transfer time. Latency is the fixed cost incurred for each communication operation, regardless of the data size. Transfer time refers to the cost of moving data through the link, which increases with message size and is influenced by the link's bandwidth. For small messages, the cost of data movement is minimal, making latency the dominant factor. Conversely, for large messages, the situation reverses. Depending on whether communication is latency-bound or bandwidth-bound, different topologies may be preferred. Slide 23 — 6:18 (watch) Jackal supports two topologies: a mesh and a ring. In a full mesh, every machine connects directly to every other machine, resulting in the lowest possible latency for group communication. In a ring topology, each node connects only to its two neighbors, requiring communication between non-adjacent nodes to travel through intermediate machines, which increases latency. However, the ring topology requires fewer cables and ports per machine, making it easier to scale to more nodes. Each node has only two connections, allowing the use of extra Thunderbolt ports to run two or three cables per neighbor, depending on the MAC, which increases the bandwidth per link and reduces transfer time. When machines are connected in a mesh, we can route each communication through either the mesh or the ring topology. Slide 24 — 6:56 (watch) Jackal automatically selects the optimal topology based on the message size and communication operation: it uses a mesh topology when latency is a priority and a ring topology when bandwidth is more critical. To take advantage of this flexibility, we will connect all M3 Ultras in a mesh configuration. Slide 25 — 7:08 (watch) Now that we have connected all M3 Ultras together, we need to enable RDMA on all machines. Slide 26 — 7:20 (watch) Open Settings on each machine, search for RDMA, click on "Enable RDMA over Thunderbolt," enable RDMA, and then reboot the machine. Slide 27 — 7:32 (watch) Great, Macs are connected with Thunderbolt 5 cables, and RDMA is enabled. Now we need a method to launch distributed programs. Slide 28 — 7:40 (watch) One way to launch distributed programs is over the local network, using Wi-Fi or Ethernet. Slide 29 — 7:48 (watch) From any machine with SSH access to the cluster, such as a MacBook in my case, we connect to each Mac, start the program, and from that point on, all machines communicate directly over the Thunderbolt links. Slide 30 — 8:02 (watch) MLX provides a launch helper that automates this process for you. Slide 31 — 8:08 (watch) You run MLX launch on your MacBook, and it orchestrates the cluster for you. Slide 32 — 8:12 (watch) You provide the executable you want to run and a JSON host file that describes your cluster. Slide 33 — 8:18 (watch) From there, it SSHs into each node using the host names from the provided host file and starts the executable on every machine. Slide 34 — 8:28 (watch) The host file that describes the cluster is structured as a JSON array, with one entry for each node. Slide 35 — 8:42 (watch) The host file is a JSON array with one entry per node. The "SSH" field specifies the hostname used by MLX launch to connect to the machine. The "IPs" field contains the machine's IP address on your local network, which Jackal uses for initial coordination between nodes. The "RDMA" field lists the RDMA device names for each Thunderbolt peer connection. Slide 36 — 8:58 (watch) You can write the configuration manually, but MLX also provides a helper script, MLX distributed config, that generates it for you. Slide 37 — 9:26 (watch) You pass a list of host names and an output path, and you can embed environment variables in the config, which will be set automatically on every node at launch time. Here, we set MLX Metal Fast Sync to 1, enabling faster GPU-to-CPU synchronization. This is critical for distributed tasks because computation runs on the GPU while communication runs on the CPU. You can also pass the autosetup flag to configure the Thunderbolt network automatically. The communication backend argument defines whether the setup is a mesh or a ring. For a mesh, the backend is set to Jackal, as shown in this example. For a ring, you would change it to Jackal ring. Slide 38 — 9:50 (watch) Let's run this command to generate the host file for our cluster. Slide 39 — 10:02 (watch) First, it checks that all hosts are reachable over SSH. Then, it processes the connections by examining each machine's Thunderbolt ports to determine which machines are physically connected to each other, thereby building a map of the topology. Slide 40 — 10:22 (watch) Since we passed autosetup, the script disables the Thunderbolt bridge on all machines and configures each Thunderbolt link for RDMA. Finally, it writes a JSON host file containing everything needed for the MLX launch. Note that without the autosetup flag, the script prints the configuration commands for your review, allowing you to run them manually. Slide 41 — 10:38 (watch) The cluster is now ready. We will move on to the exciting part: distributed language model inference and fine-tuning. The easiest way to begin this is through the command-line interface and MLX LLM. Slide 42 — 10:58 (watch) MLX LLM is an open-source Python package built on MLX that provides command-line tools and a Python API for running language models locally on Apple Silicon. To get started on a single device, check out our video, "Explore Large Language Models on Apple Silicon with MLX" from WWDC25. Slide 43 — 11:40 (watch) As we demonstrated last year, you can chat with a model on a single Mac using the command-line interface with MLX LLM chat. You run it in the terminal, specifying the model, such as QN3.6, and the maximum number of tokens for the response. MLX LLM loads and runs the model on a single machine. To chat with the same model on the cluster via the command-line interface, you wrap the command with MLX launch. On your MacBook, you run MLX launch in the terminal with the host file pointing to your cluster configuration. After the double dash, you pass the same MLX LLM chat command, but with the remote path to the executable on each node. The command is nearly identical. MLX LLM manages the model and coordinates the distributed inference for you. Remember that all necessary libraries, including MLX, must be installed on each Mac, and the executable must be accessible on all machines. Slide 44 — 12:22 (watch) With a single command in the command-line interface, we can run a model distributed across the entire cluster. We will compare the performance of QN3.6, a 27 billion parameter model, running on a single M3 Ultra versus four M3 Ultra machines. Slide 45 — 12:42 (watch) I have already started the MLX LLM chat on both sides. On the left, the model is loaded on a single M3 Ultra, while on the right, it is sharded across four machines. Let's prompt both to implement a transformer model in MLX. Slide 46 — 12:54 (watch) The cluster generates tokens at nearly three times the rate of a single machine for the QN3.6 model, which is quite an impressive speedup. Slide 47 — 13:22 (watch) Running a model across multiple Macs can significantly boost inference speed. The exact speedup depends on the model size and architecture. However, time improvement is not the only reason to adopt a distributed approach. Sometimes, a model is simply too large for one machine. For example, Kime 2.6 has one trillion total parameters. Even with 8-bit quantization, the weights alone require about one terabyte of memory. This amount does not fit on a single M3 Ultra but could fit across four machines. To split the weights and computation across machines, MLX and MLX LLM support two approaches: pipeline and tensor parallelism. Slide 48 — 13:58 (watch) Pipeline parallelism divides the model by depth, with each machine holding a group of layers. Data moves through the machines sequentially. While this approach does not accelerate inference—since each token must pass through the layer groups one after another—it simplifies communication. Machines only exchange activations at the boundaries between layer groups. Slide 49 — 14:24 (watch) Tensor parallelism splits the model by width, allowing each machine to hold part of every layer. This enables all machines to process the same token simultaneously, improving inference speed through parallelized per-layer computation. However, this approach requires much more frequent communication, occurring at every layer and for every token. Slide 50 — 14:42 (watch) Low latency is important, which is why the mesh topology is crucial in this scenario. Every machine can reach every other machine in a single hop. Slide 51 — 14:54 (watch) Tensor parallelism is the default sharding strategy in MLX for LLMs. To shard the model using pipeline parallelism, append the flag "pipeline" to the command. However, not all models support pipeline parallelism. Slide 52 — 15:06 (watch) Now, let's shard a one trillion parameter Kimi2.6 on our cluster. Slide 53 — 15:20 (watch) For this, we use MLX launch from our MacBook as before, pointing to the host file. I am not passing the pipeline flag, so we are using tensor parallelism. We need to wait a moment while MLX launch connects to every machine. MLX LLM loads and shards the model, then starts the job. Slide 54 — 15:36 (watch) The model is loaded. Let's prompt the model with "implement machine learning architecture for GPT in Python with MLX." With one command, a massive trillion-parameter model is running locally across your machines, answering your questions. Slide 55 — 16:02 (watch) With MLX and MLX LLM, you can run language model inference and fine-tune models on your hardware. This process is fast, efficient, and fully private, ensuring that your data never leaves your machines. We will start with a single machine and then scale to our cluster. Slide 56 — 16:22 (watch) When fine-tuning or training on a single machine, we split the training data into batches, which consist of multiple examples. For each batch, the mug computes gradients and updates the model weights. This process is repeated for one or more passes over the training dataset until the model achieves the desired quality. Slide 57 — 16:36 (watch) The faster we process the training data, the sooner fine-tuning is completed. To achieve this, we can utilize multiple machines to accelerate the process. Slide 58 — 16:54 (watch) The idea is straightforward. We replicate the model on every machine. Each machine receives a different batch of data and computes gradients locally. Then, we average the gradients so the model's update incorporates information from all batches. This approach is called data parallel training because the model is replicated while the data is processed in parallel across machines, resulting in a significant speedup. Slide 59 — 17:10 (watch) With n machines, we can process data up to n times faster. This is a significant advantage. Slide 60 — 17:14 (watch) We can utilize data parallelism with MLX LLM. Slide 61 — 17:30 (watch) The only difference from a single device is launching the job with MLX Launch from your MacBook and specifying a path to the MLX LLM LoRa on remote machines. Data sharding is managed by MLX LLM, and the command is nearly identical. We scale the batch size by the number of devices, ensuring that each machine processes the same number of samples per step as before. Slide 62 — 17:48 (watch) We will fine-tune QN3.5, which has 9 billion parameters, on both a single machine and a cluster. We will compare the number of tokens processed by the model per second in each case. Slide 63 — 18:00 (watch) We are launching fine-tuning on a single device on the left and on the cluster on the right using MLX. We utilize the launch and host file to specify the path to the MLX LLM LoRa on the remote machine. Slide 64 — 18:16 (watch) First, it loads the data and model, and then training begins. A single M3 Ultra processes approximately 180 tokens per second, while the cluster processes around 600 tokens per second, resulting in more than a threefold speedup for fine-tuning. Slide 65 — 18:30 (watch) With MLX, you can transform your devices into a local training cluster for efficient fine-tuning without relying on the cloud. Slide 66 — 18:46 (watch) So far, we have used the command-line interface for distributed inference and fine-tuning within the MLX LLM. However, MLX offers fine-grained control over sharding and distributed operations through flexible Python, Swift, and C++ APIs. This enables you to experiment with models in Python and C++ or embed models into your app using Swift. Let's look at the examples. Slide 67 — 19:14 (watch) To run distributed inference with the Python API and MLX LLM, we first initialize the distributed group for communication. Next, we define the type of parallelism we want, such as tensor parallelism. Then, we shard the model using the sharded load function. After that, we can use the model just as we would on a single device, as MLX LLM manages all distributed communications automatically. Slide 68 — 19:36 (watch) To gain more control over the model and its sharding, we can use low-level primitives from MLX. For instance, after defining a simple linear layer, we can shard it with tensor parallelism using the shard linear function. Slide 69 — 19:52 (watch) You can control basic distributed operations, such as all-reduce, in Python, Swift, or C++. After initializing the distributed group using Jackal, we perform a collective distributed sum across all marks for our tensor with the corresponding MLX prism. Slide 70 — 20:16 (watch) As mentioned at the beginning of the session, Jackal is available independently and can be utilized for any applications that require distributed communication, including non-ML applications. Jackal can be built without MLX and offers a C++ API with communication primitives. After initializing a Jackal group, we perform a collective distributed sum across all marks for our tensor directly through Jackal, rather than using MLX. Slide 71 — 20:38 (watch) You now understand both the high-level and low-level APIs for distributed inference and training with MLX and Jackal. You are prepared to build advanced distributed workloads using MLX. Slide 72 — 21:04 (watch) Throughout the session, we explored the full stack that enables distributed training and inference on Apple Silicon, from RDMA over Thunderbolt to MLX and MLXLM. We demonstrated how easy it is to scale from a single device to multiple devices, highlighting the benefits such as faster inference, the capability to run trillion-parameter models, and quicker fine-tuning, all with minimal changes to your single-device code. This supports command-line interface, Python, Swift, and C++ APIs. Slide 73 — 21:26 (watch) With Distributed Cluster, you can now run local AI agents powered entirely by MLX, quickly and privately, on your own hardware. Slide 74 — 21:42 (watch) To learn more, watch our WWDC 2026 video, "Run Local Agentic AI on the Mac Using MLX." For advanced distributed features, including custom parallelism strategies and training loops, refer to our documentation. You can also use MLXLM to serve models in a distributed manner with the built-in server. Slide 75 — 21:56 (watch) We look forward to seeing what you create with MLX on Apple Silicon.

64 slides extracted. Slide 1 — 0:04 (watch) Hello, my name is Xiao, and I am a GPU software engineer. Slide 2 — 0:16 (watch) Today, I will guide you through an exploration of Metal tensors and demonstrate how to write optimized custom machine learning kernels using tensor operations. Slide 3 — 0:34 (watch) Apple platforms offer robust support for running machine learning models across all layers of the software stack. High-level frameworks such as Core AI and MLX simplify model deployment with minimal code. In contrast, lower-level APIs like Metal Performance Shaders grant access to high-performance Metal kernels. All these layers leverage the low-level acceleration provided by Metal Performance Primitives and the Tensor Ops library. Slide 4 — 0:52 (watch) There are several reasons to work at the Metal level. Slide 5 — 1:00 (watch) Machine learning research evolves rapidly, so you may want to implement custom operations that integrate with high-level frameworks like Core AI. Additionally, writing Metal kernels may be necessary if you are contributing to an ML framework such as MLX or llama.cpp. Slide 6 — 1:14 (watch) If you are working on a Metal-based application, the easiest way to get started is by using the tensor ops library. Slide 7 — 1:22 (watch) Tensor Ops is a Metal shading language API that accelerates tensor operations on the GPU, including matrix multiplication and convolution. Slide 8 — 1:38 (watch) It automatically utilizes any available hardware acceleration across all Apple Silicon GPU generations, eliminating concerns about differences between hardware generations. Specifically, it fully leverages the Neural Accelerator in the M5 chip family. Slide 9 — 1:56 (watch) The Neural Accelerator is a new hardware block in the M5 chip, located directly within each shader core. It operates alongside the other GPU pipelines and is specifically designed to accelerate dense compute-bound tasks, such as the pre-fill stage of a large language model (LLM). Slide 10 — 2:06 (watch) You can refer to the related sessions to learn the fundamentals of getting started with tensor operations. Slide 11 — 2:12 (watch) In this session, I will build on those basics by discussing best practices for working with quantizer data. Slide 12 — 2:22 (watch) I will show you how to build advanced custom operations, such as flash attention. Slide 13 — 2:26 (watch) Let's dive into the first topic: working with quantizer data. Slide 14 — 2:32 (watch) State-of-the-art machine learning models are becoming larger. Slide 15 — 2:42 (watch) The inference stage is typically limited by memory bandwidth, making weight compression necessary to fit models into memory and to conserve memory bandwidth. Slide 16 — 2:52 (watch) The standard approach for compressing weights is quantization. This involves taking higher precision weights and reducing them to lower precision data types. Slide 17 — 3:08 (watch) For example, 16-bit half-precision weights can be compressed to just 4 bits. These quantized weights are paired with scale factors, allowing us to scale the quantized value back to the original range during computation. In addition to 16- and 32-bit floating-point types, tensor operations now natively support quantized data types. Slide 18 — 3:34 (watch) We added support for 4- and 8-bit integer types in the update to macOS and iOS 26, and we are extending support to additional data types in macOS and iOS 27. This includes 4- and 8-bit floating-point types, as well as 2-bit integer types. Slide 19 — 3:48 (watch) You can create and pass your IAFS quantizer tensors to tensor operations, which will automatically utilize any available hardware acceleration. Slide 20 — 4:04 (watch) Creating a tensor with a quantized data type is similar to creating a regular tensor. You fill in the descriptor's properties as you would for any other tensor, but specify a quantized data type. Then, create the tensor by calling newTensorWithDescriptor on your Metal device. Slide 21 — 4:18 (watch) You can store your quantized element data using the method described. Next, let's discuss scale factors. Slide 22 — 4:36 (watch) In macOS and iOS 27, a single MTL tensor object can represent scales alongside the tensor's quantized data as an additional scales plan. This plan supports the popular FP8 EAM0 block-wide scale factor format. Each element of the scales plan applies to a block for each element in the data plan. Declaring the scales plan is similar to declaring a tensor. Slide 23 — 5:04 (watch) First, create a descriptor object for the scales plan. Next, fill in the data type and block factors. Then, create an auxiliary plan map to specify that this plan is for scales. Finally, attach the auxiliary plan map to your original tensor descriptor. The quantized data, scales, and metadata will all be packed into a single tensor object. Now, let's put this into practice by extending a basic matrix multiplication kernel to support quantization. Slide 24 — 5:26 (watch) Matrix multiplication is a core operation in machine learning workloads. Slide 25 — 5:34 (watch) For instance, LMS performs millions of matrix multiplications during inference. Slide 26 — 5:48 (watch) We discussed how to write a high-performance matrix multiplication kernel using TensorOps in the M5 machine learning talk. The approach involves slicing the input matrices into smaller tiles and performing tile-wise matrix multiplications with TensorOps. This method maximizes parallelism and optimizes data caching. Slide 27 — 6:02 (watch) We can use quantization to further reduce memory traffic and accommodate larger models in memory. Slide 28 — 6:24 (watch) In the kernel, it is beneficial to define type aliases upfront before binding the tensors. Here, we declare a scale factor plan using the FP8 EAM0 data type and a block size of 32 by 1. This means every 32 elements in the data plan share a single element in the scales plan. Next, we declare a full tensor type, specifying the FP8 data type along with the scales plan. You can bind these tensors to buffer binding points, allowing the kernel to access the tensors allocated on the host side. Slide 29 — 6:46 (watch) Alternatively, if you don't want to create a full Metal tensor on the host, you can create a temporary tensor directly on the shader stack. Slide 30 — 6:54 (watch) The syntax is nearly identical; simply replace the tag tensor handle with tensor inline. Slide 31 — 7:02 (watch) Pass your buffer pointers and other metadata to the tensor constructor to create a tensor on the stack. Slide 32 — 7:12 (watch) We will divide the problem across multiple thread groups to enhance parallelism. First, we will slice out the tile for each thread group, and then we will perform the multiplication using tensor operations. Slide 33 — 7:26 (watch) To achieve this, call the slice function on your input and output tensors using the thread group ID. The data and the scales plan will be sliced simultaneously according to the block size. Slide 34 — 7:46 (watch) Setting up the matrix multiplication with a quantizer tensor is the same as with normal tensors. First, configure the matmul2d descriptor by specifying the tile sizes and other parameters. Next, create a matmul2d operation, indicating the number of command groups in the thread group. Finally, pass in your quantizer tensor, and the tensor operations will manage the dequantization for you. Slide 35 — 8:06 (watch) In most cases, you should input your quantizer data directly into tensor ops, allowing them to automatically leverage any available hardware acceleration. Slide 36 — 8:12 (watch) If you need to dequantize a custom format, tensor operations can still accommodate this requirement. Slide 37 — 8:28 (watch) The simplest approach involves each thread loading a chunk of quantizer data from device memory and dequantizing it to F16 values in the thread group memory. This data can then be passed as an inline thread group tensor to tensor operations. However, this method requires additional load and store operations through thread group memory. Ideally, we should keep all this data in thread registers instead. Slide 38 — 8:44 (watch) You can achieve this by dequantizing the data into a cooperative tensor, which can then be used as the input for the matmul2d operation. Slide 39 — 8:54 (watch) Cooperative tensors distribute their storage across the thread-private memory of the threads participating in the matmul operation. If you cannot use a quantizer tensor directly, you can still avoid the round trip through thread group memory. Slide 40 — 9:10 (watch) To recap, Metal tensors natively support a wide range of quantizer data types, including the new MX scaling formats and the EM0 scale factors coming in iOS and macOS 27. Slide 41 — 9:22 (watch) These new data types have additional alignment requirements compared to larger data types, so be sure to check the Metal documentation for details. Slide 42 — 9:34 (watch) Now let's advance to building a more complex custom operation using tensor operations. Slide 43 — 9:54 (watch) Attention is fundamental to every transformer network, including language models. To compute attention, you first multiply two matrices, known as Q and K. Then, you compute the softmax using reductions on the rows of the intermediate matrix. Finally, you multiply by a third matrix. The popular flash attention algorithm combines all these operations into a single kernel. Slide 44 — 10:18 (watch) To implement this with tensor operations, first set up a custom CMD group mapping, ensuring that each CMD group owns complete rows of the intermediate matrix. This configuration allows you to compute the softmax without exchanging data between CMD groups. Slide 45 — 10:34 (watch) You can achieve this with the execution CMD group operation scope. Each CMD group will perform independent matrix multiplications in parallel. Use the CMD group ID to slice your input tiles. Slide 46 — 10:50 (watch) We will use a cooperative tensor to store the intermediate matrix, allowing us to use it as input for the next step without writing it to memory. We will compute softmax on the result, which requires performing a couple of reductions on the cooperative tensor. Slide 47 — 11:02 (watch) Tensor operations include a reduce rows function to assist with this. Slide 48 — 11:12 (watch) Threads will exchange data to calculate the maximum for each row. The result is returned in another cooperative tensor. Let's set it up. Slide 49 — 11:26 (watch) First, create a cooperative tensor to store the reduction output. Next, pass the source and destination tensors to the reduce rows function. We will use the max reduction operation with an initial value of negative infinity. Slide 50 — 11:46 (watch) These two cooperative tensors have different shapes, so to facilitate the mapping between them, tensor operations include a map iterator function. This function takes an iterator pointing to an element in the 2D tensor and returns an iterator pointing to the corresponding element in the reduction destination. Slide 51 — 12:06 (watch) First, set up a loop over the 2D cooperative tensor using iterators. Then, call the map iterator function to map each element to its corresponding row maximum. Finally, dereference these iterators to compute the software maximum and store the result back into the cooperative tensor. Slide 52 — 12:20 (watch) Now we are ready to multiply this cooperative tensor by V. Slide 53 — 12:28 (watch) In macOS 26, you must first store the tensor in thread group memory. However, it is now possible to use cooperative tensors directly as inputs for manual operations. Slide 54 — 12:50 (watch) To do this, call the getLeftInputCooperativeTensor method, passing the source cooperative tensor as an argument. You can then use the result as an input to the second manual operation. However, not every cooperative tensor can be reused as an input; the layouts may differ based on data types and other factors. Before proceeding, call the isCompatibleAsLeft or isCompatibleAsRightInput method to check for compatibility. If it returns true, you can continue. Slide 55 — 13:14 (watch) If the compatibility check returns false, you will need to store and reload the data through thread group memory to convert it to the correct layout. Regardless of the situation, the call to op.run remains the same. Slide 56 — 13:22 (watch) These are the key TensorOps features required to build an advanced operation like FlashAttention using TensorOps. Slide 57 — 13:30 (watch) Now that we've discussed how to build this operation, let's examine its performance in a real model using Core AI. Slide 58 — 13:46 (watch) Core AI provides tools for Python developers to convert PyTorch models into Core AI models, including support for custom meta kernels. For details on integrating a meta kernel into a Core AI model, refer to the Deep Dive into Core AI Model Authoring and Organization session. Slide 59 — 14:24 (watch) I followed the steps outlined in that session to integrate our custom FlashAttention kernel into a SAM3 image segmentation model. We define the body of our custom attention kernel as a string in Python and register the TorchMetalKernel object. Then, we replace the default Hugging Face Attention implementation with one that calls our kernel. Finally, we load the model from Hugging Face and export it from PyTorch as an optimized Core AI asset. The export will take a moment to finish. Now we're ready for inference. SAM3 performs promptable concept segmentation, so we provide the model with an image and text, and it responds with a segmentation mask indicating where objects are located in the image. Slide 60 — 14:58 (watch) Here, I am prompting the model to label all pixels that contain a car in the image. Slide 61 — 15:06 (watch) Now, I will run the segmentation. Slide 62 — 15:18 (watch) The final result shows that the model correctly segmented the image, with the car highlighted in blue. This indicates that our attention kernel is fully integrated into the model as expected. Slide 63 — 15:32 (watch) Today, I covered the tools available for building optimized custom ML kernels on Apple Silicon. This includes quantized data types, advanced TensorOps features such as cooperative tensors and reductions, and integration with Core AI. Slide 64 — 15:54 (watch) To go further, explore the Metal Performance Primitives documentation for the full API reference and the programming guide for additional performance optimization guidelines. You can also download the TensorOps sample code to review details that I couldn't cover here. Additionally, check out the related sessions to learn more about Core AI and Metal. Thank you.

36 slides extracted. Slide 1 — 0:08 (watch) Hello, I'm Michal. Slide 2 — 0:18 (watch) Welcome to "Capturing Decisions for Humans and AI Alike." Slide 3 — 0:28 (watch) Yesterday, my team at Safe Intelligence released PACT27, a new product for testing agents. Prior to this, I worked at Microsoft and Red Hat, spending 10 years focused on a single product. Slide 4 — 0:40 (watch) The consistency problems we encounter with AI and the process of capturing decisions are evident in every product I have observed. These notes are a summary of that experience, and you can find me at the booth. Slide 5 — 0:54 (watch) BDD, PRD, and ADR are important acronyms in our field. Let's unpack their significance. Slide 6 — 1:02 (watch) You probably know this story. Slide 7 — 1:16 (watch) I hope this isn't an urban legend, but scientists placed five monkeys in a cage with bananas on a ladder. Whenever a monkey attempted to reach the bananas, they received a cold shower. The other monkeys would then beat up the one trying to climb. Eventually, they replaced the monkeys one by one until none of the originals remained. Despite this, every new monkey that tried to climb the ladder was beaten, unaware of the reason behind the behavior. Slide 8 — 1:44 (watch) Humans and LLMs share a common trait: limited context. People forget, and LLMs compact context. Humans may leave, and LLMs lack memory. After some time working on a product, the team begins to ask questions such as, "Why do we have this flow? What is the goal of this feature? Why is this code structured this way? Where does this belong?" Often, the founding engineer may not be available to provide answers. These issues can arise in any organization, and with AI, they may surface even sooner than before. Slide 9 — 2:10 (watch) An Architecture Decision Record (ADR) documents the rationale behind architectural choices and outlines how those decisions are enforced. It can include examples through reference documents and code snippets. Slide 10 — 2:32 (watch) We split code into layers to prevent n plus 1 queries. We enforce this separation by linting imports in modules. Additionally, we require reading from the database to reference plain shapes instead of ORM objects, which helps prevent these queries and duplication. There are also about 50 other ADRs that define the architecture of the product. Slide 11 — 3:02 (watch) There is no single format that must be used; it's merely a concept. Since it is a text-based approach, there is no specific way to enforce it. However, you will need a tool to enforce the rules. This tool will indicate the rules, explain why they are necessary, and guide you on how to address any issues. The agent will then attempt to locate the document that provides the rationale for the rule and additional information on how to resolve it. Slide 12 — 3:22 (watch) You can specify which files are relevant, such as Python files or specific folders, and detail how to enforce the rules. Slide 13 — 3:34 (watch) A Product Requirements Document (PRD) is a lighter document used when building a feature. It describes the purpose of the feature, the problems it addresses, and the user's journey through the application. Slide 14 — 3:56 (watch) It can be very concise; it doesn't need to be lengthy or exhaustive like a massive document. You can simply capture the why, the problem, the goal, and the journey that connects them. This approach benefits not only the agents but also you, six weeks from now, when you might forget the reasoning behind your decisions. Slide 15 — 4:12 (watch) Behavior-Driven Development, or BDD, focuses on the expected behavior of software. You may have encountered specification-driven development recently, but if you've practiced it, you might have experienced similar challenges as I have. Slide 16 — 4:36 (watch) To validate that the product adheres to the specifications, you need to ensure that the markdown document accurately describes how it is supposed to work. However, determining if it actually functions as intended can be challenging. Reading AI code is difficult, and reading AI tests is even harder. What if there were an intermediate layer that describes the product's behavior in human language? Behavior-driven development (BDD) is not a new concept, but it can be both executable and readable. Slide 17 — 5:06 (watch) Enter Cucumber. It may have been almost forgotten, but it's suddenly useful again. Cucumber is definitely easier to review than your average tests. You can connect scenarios directly to your PRDs and critical user journeys. It is readable and executable, effectively closing the loop that spec-driven development leaves open. Slide 18 — 5:38 (watch) These specifications are parsed into steps and executed as code. You can write, read, and review them, making them understandable. The language used is flexible and does not need to be enforced. There are various ways to write these features, but they all describe how to navigate the application, the purpose of each component, and its functionality. Additionally, they can reference all relevant documentation explaining the rationale behind these elements. Slide 19 — 6:08 (watch) Creating consistent UIs with agents presents an additional challenge. A design system and pattern library are essential for building these consistent UIs. This approach was effective before AI and remains effective today. Slide 20 — 6:48 (watch) You document your design language by specifying details such as the characteristics of a primary button. For example, you define it as blue, with a specific shape and size. You establish rules, such as having only one primary button visible on a site or page at any given time, and then you enforce these rules. You also define components and patterns. If you have multiple colors and states for buttons, you create components and previews to demonstrate their functionality, allowing both you and the agents to visualize them. You can then review whether these elements adhere to your established principles and visuals, and reuse them accordingly. Similar to coding, you build these components from small pieces into larger ones, composing and reusing them to avoid chaos. Slide 21 — 7:26 (watch) These are great ideas, but how do I enforce them so my team and agents adhere to them? How do I maintain consistency? The answer lies in the loop. Slide 22 — 7:38 (watch) You have probably heard about closing the loop, the reinforcement loop, and the harness, which help remind agents of their rules and how to follow them. Slide 23 — 7:46 (watch) Our loop is straightforward. Slide 24 — 7:56 (watch) We implement Git hooks to run predefined tasks, which are essential for delivering a pull request. These hooks include various checks, skills, and continuous integration (CI) processes. Slide 25 — 8:26 (watch) These tasks are executed on the CI and are the same tasks run as hooks. If agents skip or neglect to execute them, they will be caught. We include linting, formatting, type checking, code duplication checks, architecture checks, and document linting—essentially everything possible. In the past, code reviews focused on style issues like tabs and spaces, but that is no longer acceptable. These aspects are not open for discussion; they are rules that are enforced and automated. The focus has shifted to high-level concepts. What cannot be identified cannot be enforced. Slide 26 — 9:02 (watch) We enforce the architecture of the product and the code by separating modules and their imports to control what can be accessed from where. For instance, our end-to-end BDD test suite is prohibited from accessing the database. We restrict access to any module that could connect to the database, ensuring that the models iterate without using the database and rely solely on the browser features of the application. Slide 27 — 9:32 (watch) In the product, we enforce a rule that prevents database access from rendering templates. This ensures that there are no N plus one queries. Instead of continuously identifying these issues, we focus on defining methods to prevent them entirely. When an agent attempts to commit and push changes, they receive feedback on their commit. If the commit is rejected, they are linked back to the relevant document to read, fix, and iterate on their work. Slide 28 — 10:02 (watch) There are some considerations to address. This loop is generic; it involves doing work, pushing it, receiving feedback, and iterating. However, the focus of the loop can vary. Sometimes you might be working on a product feature, other times on a user interface, or even on back-end tasks. While the loop remains the same, the specific focus shifts depending on the context. Slide 29 — 10:40 (watch) We have different skills for various aspects of development. For Architectural Decision Records (ADRs), the agent looks them up to understand how to operate with them and identify the affected code. The same applies to Product Requirement Documents (PRDs). In the UI loop, we skip several checks to enable rapid iteration in a browser. The testing skill identifies which tests to run based on code coverage and file changes, allowing us to execute only the relevant part of the test suite rather than the entire suite. Additionally, we implement goal execution to retain the decisions made by the model for later review. While all these elements provide focus within the loop, the fundamental structure of the loop remains unchanged. Slide 30 — 11:06 (watch) There are drawbacks; it is very context-heavy. Slide 31 — 11:16 (watch) You can run out of half of the context when starting the research, but I have no fear of context compacts. This approach has been effective for the last six months. Slide 32 — 11:32 (watch) In my sessions, there are 20 to 50 context compacts, and that's acceptable. The important information survives, and the agent will always reference it again. The goal is to have multi-hour sessions with a clear objective, allowing the agent to operate autonomously within the defined rules. Slide 33 — 11:44 (watch) That is the goal. Slide 34 — 11:52 (watch) There are decisions that you can record. Slide 35 — 12:02 (watch) You can describe the reasons for the existence of certain parts of the product. Cucumber and BDD provide executable specifications that are readable, reviewable, and understandable. Slide 36 — 12:20 (watch) Design systems help create a consistent UI using components and enforce rules, such as prohibiting inline styles. You can use Harness to integrate everything. Thank you.

60 slides extracted. Slide 1 — 0:08 (watch) Hello, I'm Sydney, an open source engineer at LangChain. Slide 2 — 0:14 (watch) Hi, I'm Victor, a product manager, and today we'll discuss managed deep agents. Slide 3 — 0:30 (watch) Before we discuss managed deep agents, Sydney will explain deep agents as a harness. First, let's define an agent. An agent is a simple model and a tool-calling loop that iteratively calls tools until it completes a task and returns a final result. Slide 4 — 0:52 (watch) A harness can be defined as the combination of a model and its surrounding components. It connects the model to the real world and facilitates task completion. The harness includes skills, memory, the base system prompt, tools, sub-agents, and any additional context. Slide 5 — 1:16 (watch) The job of a harness is to provide the model with the right context at the right time for a given task. A model's power is directly tied to the context it receives, and the harness serves to bridge this gap. Slide 6 — 1:30 (watch) A harness is essential because agents have many responsibilities. They must operate in an environment where they can take actions. Slide 7 — 1:52 (watch) This action-taking provides agents with agency, making them useful. They must connect to your data to ensure their actions are relevant to your use case. Agents need to manage growing context over long runs to avoid context overflow. They should be able to parallelize tasks to complete complex operations efficiently. Additionally, they need to connect with humans in the loop for sensitive workflows. Slide 8 — 2:12 (watch) Finally, they should improve over time to remain relevant and useful. Slide 9 — 2:20 (watch) Deep agents are customizable agent harnesses specifically designed for complex real-world tasks. Slide 10 — 2:30 (watch) First, I will cover the four main capabilities of the deep agents harness, followed by an in-depth exploration of each one. Slide 11 — 2:46 (watch) First, we have the execution environment, which serves as the backbone of a deep agent. It begins with the file system, and you can optionally enhance this with a sandbox or a code interpreter. Next, we have context management, which I believe is the most important capability. Slide 12 — 3:14 (watch) Deep agents come equipped with various utilities to assist with context management. These include skill support, built-in short- and long-term memory, summarization capabilities, context offloading, and prompt caching. Another key capability is delegation. As agents operate over extended periods and handle complex workflows, they must plan and organize tasks effectively, utilizing sub-agents to delegate work. Slide 13 — 3:38 (watch) Finally, we incorporate steering into the deep agents harness, providing first-class human-in-the-loop support. Slide 14 — 3:44 (watch) Now, let's take a deep dive. Slide 15 — 4:10 (watch) Starting with the execution environment, which is the backbone of a deep agent, this capability powers all other functions. An agent utilizes a file system to read and write scratch files while tackling tasks, load and store persistent memories in the hot path, invoke skills relevant to specific tasks, and more. Agents excel at using file systems because they are trained in environments that emphasize file system usage and are exposed to extensive code. This is why providing an agent with a sandbox or its lighter-weight counterpart, the code interpreter, is highly effective. Slide 16 — 4:40 (watch) Providing an agent with code execution tools creates a secure environment for writing and running code. This capability enhances the agent's ability for creative problem-solving and enables dynamic runtime behavior. Slide 17 — 4:50 (watch) The second capability is context management. Slide 18 — 5:10 (watch) Deep agents come with built-in summarization and context offloading. As shown in the graph, deep agents periodically evict large messages—such as human messages, tool results, and tool calls—to the file system. This prevents context from building up too quickly in the window. Summarization is triggered less frequently, occurring when the history approaches the model's context limit. These features aim to avoid context overflow, a common issue for long-running or high-context agents. Slide 19 — 5:34 (watch) Deep agents also include built-in memory support. I would argue that memory is the most important type of context because it changes from run to run, allowing your agent to improve over time. Slide 20 — 6:16 (watch) It also includes provider-agnostic prompt caching, which is crucial for long-running, high-context agents that need to operate cost-effectively. Additionally, deep agents come with built-in skills support. Skills are integral to the context management system through a mechanism called progressive disclosure. Deep agents load minimal information about available skills into the system prompt initially, allowing the agent to dynamically access full skill resources and invoke those skills and their scripts as needed for specific tasks. Slide 21 — 6:36 (watch) This umbrella of context management addresses the need for a framework that provides the model with the appropriate context at the right time for each specific task. Slide 22 — 6:48 (watch) The third capability of the deep agents harness is delegation. Slide 23 — 7:04 (watch) The deep agents harness includes a planning tool that enables the model to organize work for complex tasks. It also supports sub-agents out of the box, which can be either general-purpose or specialized. For instance, when building a coding agent, you might want to attach specialized sub-agents for architecture design, code review, security review, and test writing and execution. Slide 24 — 7:20 (watch) Sub-agents are important for several reasons. Slide 25 — 7:30 (watch) First, sub-agents operate with isolated context, which aids in overall context management. Slide 26 — 7:42 (watch) When the main agent invokes a sub-agent, it begins with a fresh context that is relevant only to its specific task. The sub-agent then returns a streamlined final result to the main agent, ensuring that the main context window remains unpolluted. Slide 27 — 7:56 (watch) Secondly, sub-agents can be used to parallelize work, allowing your agent to run tasks end-to-end more efficiently through this parallelization. Slide 28 — 8:00 (watch) Finally, sub-agents can utilize any model and any provider, allowing you to align model capabilities with task complexity. Slide 29 — 8:34 (watch) The fourth and final capability of the deep agents harness is support for steering through first-class human-in-the-loop primitives. Human-in-the-loop is essential for two reasons. First, it allows for real-time user feedback on sensitive actions, or tool calls. Second, it enables real-time feedback from the user when needed to unblock the model. There are four common decision patterns we incorporate into deep agents. The first is an approval flow, such as approving an email before it is sent. The second is an edit, like editing a tweet before publication. The third is a reject decision, which involves rejecting a proposed financial transaction. The fourth is the respond pattern, where the agent interrupts to ask the user a question to facilitate its future progress. Slide 30 — 9:06 (watch) We have thoroughly explored the capabilities of the deep agents harness. Now, let's discuss the reasons for using deep agents. Slide 31 — 9:12 (watch) Deep agents are provider agnostic. Slide 32 — 9:30 (watch) You can use any provider or model and swap them at any time, even mixing and matching. Your main agent can utilize a different model than your sub-agents. Major providers include Anthropic, OpenAI, and Google. You can also use local models with Olama, or opt for increasingly performant and cost-effective open-source models, such as those from Fireworks, NVIDIA, and OpenRouter. Slide 33 — 9:48 (watch) Deep agents are highly customizable. Here’s a quick recap of the core agent loop and an overview of the deep agents loop. Slide 34 — 10:02 (watch) Deep agents provide a set of hooks around the core agent loop. We refer to this system as middleware, which allows you to incorporate any custom logic into your agent. Slide 35 — 10:18 (watch) This middleware can accommodate bespoke business logic, deterministic code, policy enforcement such as PII redaction, or dynamic agent control. For instance, it allows for changing the model and tools available at runtime based on the current task. Slide 36 — 10:38 (watch) Even with a capable harness, deploying to production is challenging. Your agent must operate for extended periods, recover from unexpected failures, manage human-in-the-loop interactions and unpredictable behavior, and support bursty traffic. All of this must be done while maintaining a secure posture and adhering to ever-changing interoperability standards. Now, I will hand it off to Vic, who will explain how we simplify this process. Thank you, Sydney. Slide 37 — 11:00 (watch) We just discussed what a deep agent is. Now, let's talk about what it takes to deploy one of these deep agents into production. Slide 38 — 11:12 (watch) I recognize several familiar faces in the audience who have successfully deployed these agents to customers in production. Based on our firsthand experience, today we will discuss the introduction of managed deep agents in private beta, aimed at simplifying this process. Slide 39 — 11:30 (watch) There are four core pillars to managed deep agents. The first is the harness, which Sydney just explained. The second is the runtime, which enables the use of this agent in production. The third pillar is integration with Context Hub. Lastly, we have the ability to execute safe code in a sandbox. Slide 40 — 11:38 (watch) Let's first discuss the runtime. Slide 41 — 12:02 (watch) Managed deep agents are built on top of Langsmith deployment, providing essential primitives for real-time interaction and scalability required by agents in production. This includes endpoints for creating, updating, and invoking agents as needed. We also have a purpose-built task queue and horizontal scaling to manage bursty request loads that agents may encounter. For example, if a support agent is overwhelmed when systems go down, it is crucial to handle that traffic effectively. Additionally, SDKs enable the use of these agents across various platforms, with integrations for CopilotKit, Assistant UI for Gen UI, and more. Slide 42 — 12:26 (watch) The second core pillar of this production runtime is durable execution. Slide 43 — 12:44 (watch) Durable execution may seem unexciting, but it is highly reliable in production. By running deep agents on the Langraph runtime, we can checkpoint each step your agent takes. These checkpoints are stored in durable storage, allowing us to resume and restart from any point. If your agent fails at step 49 out of 50, you can pick up from step 49 without restarting the entire process. Additionally, we can replay and fork the agent from any point in its state. Slide 44 — 13:02 (watch) This enables advanced user workflows, such as forking a conversation from a specific point. Slide 45 — 13:08 (watch) It also enables human approval through a human-in-the-loop approach. Since everything is checkpointed in the database, we can await human input indefinitely. Slide 46 — 13:42 (watch) This enables ambient agent use cases such as Langsmith Engine, which will be discussed in the next session. Security and authentication are crucial for production agents, requiring multiple layers of authentication to support these use cases. The first layer is inbound from your application, ensuring that users are authenticated and authorized to use the agent. The second layer is outbound from within the agent to your external services, where reliable authentication and correct permissions must be assumed at runtime. The third layer involves managing who can create, update, and manage these agents. This includes internal AI engineers or a CTO who may need to make quick changes, necessitating role-based access control (RBAC) and attribute-based access control (ABAC) to define permissions for these actions. Slide 47 — 14:08 (watch) Agent interoperability is increasingly important as you aim to utilize your agents across various use cases. A key feature of Langsmith deployment that we are incorporating into Managed Deep Agents is the capability to access your agent through a remote graph. Slide 48 — 14:24 (watch) You can call your agent built on Managed Deep Agents in a custom LandGraph application deployed on Langsmith with just one line of code. Additionally, we support the A2A protocol. Slide 49 — 14:38 (watch) Many agents currently utilize the A2A protocol for agent-to-agent communication, and we support this functionality out of the box in both our standard Langsmith deployment and Managed Deep Agents. Additionally, we enable you to bring this agent to the environments where you actually perform work. Slide 50 — 14:46 (watch) We build and deploy many agents internally on the Langsmith platform, and we utilize them in our work. Whether it's Deep Agents code or Cloud Desktop, it's essential to bring these agents to the environments where they are needed. Slide 51 — 14:54 (watch) We considered various production use cases so you don't have to. Slide 52 — 15:00 (watch) Features such as double texting and canceling a run mid-flight require significant engineering effort, often taking weeks or even months to develop. We provide all of these functionalities out of the box for you. Slide 53 — 15:10 (watch) The third core pillar is the context hub integration. Slide 54 — 15:26 (watch) The Context Hub is integrated into Managed Deep Agents, allowing us to version and save all the files that your agent operates on. This includes the agent metadata and the skills mentioned by Harrison during the keynote, which are increasingly popular, as well as the memories your agent retains about users. Slide 55 — 15:38 (watch) All of these are saved and versioned within Context Hub, allowing your team to manage various levels of promotion. You can begin in staging and then move to production for different skills. This approach enables you to democratize these skills across different agents. Slide 56 — 16:02 (watch) The next core feature of Context Hub in the Managed Deep Agents primitive is our integration with Langsmith Engine. You will hear more about Langsmith Engine in the next session. Essentially, it will analyze real production usage of these deep agents to make quality improvements and adjustments to your prompts, systems, and skills. This process will lead to better behavior over time as the loop and engine continue to evolve. Slide 57 — 16:14 (watch) The fourth key component of Managed Deep Agents is sandboxes. Slide 58 — 16:26 (watch) As Sydney mentioned, nearly every agent is evolving into a coding agent. Even in research use cases, agents may need to analyze quick statistics and incorporate them into reports. Enabling your agents to perform these tasks in production can lead to more creative outcomes. Slide 59 — 16:50 (watch) We are launching Langsmith Sandboxes, which will be integrated directly with Managed Deep Agents. This Langsmith Sandbox primitive includes several core features. The first is an authentication proxy that securely injects credentials at runtime, ensuring that none of your important environment variables are exposed to the agent or the sandbox itself. The second feature is the ability to snapshot and restore, allowing your agent to maintain the correct execution environment. We will have a session for a deeper dive tomorrow with Mikhail, but these sandboxes are highly effective for most agent use cases. Slide 60 — 17:12 (watch) This encompasses everything you need to take an idea or a working deep agent into production. That's why we're launching Managed Deep Agents, starting the private beta today. We encourage you to join the waitlist, and thank you for your time.

29 slides extracted. Slide 1 — 0:32 (watch) My name is Daniel Szoke, and I am the Rust SDK maintainer at Sentry. I want to explain why I believe Rust is the ideal language for vibe coding. Conventional wisdom suggests that Rust is not typically one of the first languages considered for agentic coding, or vibe coding, as some call it. When I asked ChatGPT about the best language for agentic coding, it indicated that there is no single top language, but that Python is likely the leading choice, followed by a strong contender. It mentioned JavaScript and TypeScript as well. Slide 2 — 1:22 (watch) In my experience, this observation is largely accurate, although I would reverse the order. TypeScript has emerged as the top choice for agentic coding recently. Slide 3 — 1:44 (watch) An article from GitHub, released late last year, indicates that AI has propelled TypeScript to the top language on GitHub based on contributor counts. They confirm that TypeScript is the number one language and strongly suspect this is due to its use in AI-assisted development. Slide 4 — 2:02 (watch) Python, TypeScript, and JavaScript are considered ideal for vibe coding based on conventional wisdom for several reasons. First, they are common and familiar languages, often the first ones learned by beginners in programming. They are easy for humans to understand and also appear to be straightforward for LLMs. Slide 5 — 2:42 (watch) There are many frameworks, libraries, and examples available, which is beneficial when building something new from scratch. This abundance of resources aids both humans and agents. These languages allow for quick scaffolding and execution. JavaScript and Python are dynamic and interpreted languages, while TypeScript involves some light compilation to JavaScript, making it easy to run code and see results quickly, facilitating iteration. For agents, strong typing support helps prevent type misuse, although TypeScript and typed Python can introduce some complexity. Overall, these languages are simple, enabling LLMs to generate runnable code effectively on the first attempt. Slide 6 — 3:58 (watch) The simplicity and few constraints of languages like JavaScript, Python, and TypeScript make them appealing for LLMs to write. However, I believe that many people overlook whether optimizing for this ease of use is truly beneficial. While these classic vibe coding languages are easy for models to generate, this may not be a positive attribute. In fact, I argue that the ease with which models can write these languages is often overstated and can even be detrimental. The dynamic and flexible nature of these languages facilitates writing by LLMs, but this same flexibility increases the likelihood of making mistakes, both obvious and subtle. Slide 7 — 4:36 (watch) Adding typing is a helpful constraint, but it has limitations. While it provides some type safety, this safety is not very strong in TypeScript or Python. Slide 8 — 5:22 (watch) LLMs are inherently fallible due to their design as non-deterministic systems. While we hope they will improve and make fewer mistakes in the future, it is unlikely that errors will ever be completely eliminated. Just as we need to guard against human error, we must also guard against errors from LLMs. A common approach to mitigate this risk, particularly in traditional vibe coding languages, is to add tests. While this is beneficial, relying solely on tests and code review agents presents several challenges. Slide 9 — 6:26 (watch) Firstly, if you don't prompt the agent skillfully, it will often write tests after the implementation, resulting in tests that focus on implementation details rather than properly testing behavior. Even with test-driven development, tests can only demonstrate incorrectness when they fail, as it is impractical to test every possible input combination. In many cases, you cannot prove that every input produces the correct output. Additionally, if LLMs generate the tests, they may make mistakes in writing them, and the same issue applies to code review agents. Slide 10 — 7:20 (watch) On a philosophical level, while we all know AI stands for artificial intelligence, I recently read a book called "Nexus" by Yuval Noah Harari, a historian with a unique perspective on the topic. He discusses the evolution of human information from the Stone Age to the printing press, the internet, and now LLMs. Harari argues that LLMs are unique because they represent the first instance of a non-human entity capable of producing human language. A key point he makes is his discomfort with the term "artificial" in AI, as it downplays the significant differences between LLMs and human thought processes. He prefers to refer to it as "alien intelligence," emphasizing that the internal workings of these systems differ fundamentally from human cognition. Slide 11 — 8:14 (watch) LLMs predict tokens in streams, which is a powerful mechanism, but it differs from human thinking. This difference can lead to unexpected failure modes. If you've coded with AI, you may have encountered situations where the generated code appeared well-structured, with sensible variable names and good comments. However, upon closer inspection, you might find subtle bugs or reliance on heuristics when a more reliable check could have been performed. It's essential to exercise caution with LLM and agent-based development. Slide 12 — 8:58 (watch) This brings me to Murphy's Law, which states that anything that can go wrong will eventually go wrong. When using a language without deterministic guardrails, even with human review, agentic review, and a solid testing process, the absence of a reliable deterministic safeguard will lead to failures over time. Languages like JavaScript, Python, and TypeScript often lack these guardrails, resulting in more frequent failures. In contrast, Rust is a language that incorporates many constraints to help prevent such issues. Slide 13 — 9:52 (watch) For those unfamiliar with Rust, it is a compiled language designed with safety and performance in mind. It aims to match the speed of C and C++, while ensuring memory safety and type safety. Rust's compiler is strict; if your code compiles, you can be reasonably confident that many types of bugs are absent. Slide 14 — 10:32 (watch) The compiler enforces invariants such as type safety, memory safety, and concurrency. Although Rust may seem advanced to those unfamiliar with it, the language is designed to be beginner-friendly. The compiler provides detailed error messages that explain what went wrong and how to resolve the issue, offering valuable context. This feature is particularly beneficial when AI agents compile Rust code, encounter errors, and need to correct them. Slide 15 — 11:02 (watch) Rust provides numerous safety guarantees. The first notable aspect is strict type safety; you cannot bypass it using any type or unchecked casts. Another significant feature is null safety, especially for those coming from other languages. Slide 16 — 11:34 (watch) There is no universal null value in Rust. To represent a type that can be empty, you must define it explicitly as an option type. The compiler enforces that you always check for the presence of a value before accessing the inner value. Additionally, Rust offers fearless concurrency, which means the compiler checks that any shared data between threads is handled in a thread-safe manner. This is just a small list of the many features that the Rust compiler enforces. Slide 17 — 12:04 (watch) I want to provide a quick example of fearless concurrency, as I believe it's very powerful. Here’s a simple code example. Slide 18 — 12:40 (watch) We have a counter that starts at zero, and we will create 100 threads. Each thread will increment the counter by one. Once all threads finish, we expect the counter to equal 100. However, the types used here are designed for sharing mutable data only within a single thread and are not synchronized for multi-threaded safe access. In a language like TypeScript, this code might compile and run, but you would only notice the issue when the counter occasionally returns a value other than 100. Slide 19 — 13:14 (watch) If this is a small part of a larger application, it can be very difficult to debug where the data race is occurring. However, in Rust, this code simply does not compile. Slide 20 — 13:44 (watch) You will receive a compile error stating that the future cannot be sent between threads safely. This means that the async block is not marked as `Send`, which indicates that it is not safe to transfer between threads. While the initial error message may not be very informative, scrolling down will provide more details. This information will be particularly useful for your AI agent, as it specifies that the captured counter value is not `Send` and has the type `Rc<RefCell<i32>>`, which is not thread-safe. Slide 21 — 14:16 (watch) If your AI agent compiles your project and encounters this compiler error, it can immediately change the type to a thread-safe type, of which there are many in Rust. Slide 22 — 14:32 (watch) All these constraints come with a tradeoff. Rust is harder for LLMs to get right on the first try due to the numerous rules they must follow. However, I believe this is beneficial. Slide 23 — 14:48 (watch) It's important to note that LLMs are not the only entities that write code. We integrate LLMs into AI agents that operate in a loop, allowing them to perform tasks autonomously. AI agents are particularly adept at compiling their code, identifying any failures, and correcting them. Slide 24 — 15:04 (watch) Every compile error is a potential bug that you can avoid in your production code. Some people complain that Rust's compile times are slow. Slide 25 — 15:18 (watch) However, I guarantee that it's faster than having an AI agent review your code, which may not catch all the errors that the Rust compiler is guaranteed to identify. Slide 26 — 15:30 (watch) I still believe you should use it, as it provides an additional element of safety. Slide 27 — 15:38 (watch) This is a sponsored talk, and I represent Sentry, so this is our marketing slide. Slide 28 — 15:50 (watch) You should try us out if you haven't already. This QR code provides three months of free access to our business plan. We offer agent monitoring features, and we have a booth downstairs. Slide 29 — 16:02 (watch) Please come by and feel free to ask questions about Sentry. If you would like to discuss the talk, I'm also happy to chat. Thank you.

34 slides extracted. Slide 1 — 0:08 (watch) Thank you for attending my talk. Slide 2 — 0:22 (watch) My name is Adrian Bertagnoli, and I am a founding engineer at Callosum. Today, I will discuss scaling the next paradigm of heterogeneous intelligence. Slide 3 — 0:32 (watch) I will begin by explaining why heterogeneity is important, the specific aspects that facilitate the scaling of AI, how it is currently applied in practice, and how we can leverage it in the future to scale the next paradigm of intelligence. Slide 4 — 0:46 (watch) To provide an intuition about heterogeneous intelligence, I will first explain the current prevailing paradigm of homogeneous intelligence. Slide 5 — 1:04 (watch) Homogeneous intelligence in AI primarily refers to scaling single models across a fleet of identical chips. Slide 6 — 1:36 (watch) This era has been largely shaped by the discovery of neural scaling laws, which indicate that more data and more parameters lead to better models. However, this principle is primarily applicable in the training domain, becoming less relevant as we transition to the inference domain. We are already witnessing a shift toward heterogeneity in our current systems. Architecturally, mixtures of experts are replacing large, dense models. At the workflow layer, single LLM calls are being supplanted by multi-agent systems. Finally, on the hardware level, single chips are being replaced by pre-fill decode disaggregated systems. Slide 7 — 2:12 (watch) Given that we are currently experiencing mild heterogeneity, how can we envision a greater level of heterogeneity? What will that look like? Initially, we are in a state of mild heterogeneity. Slide 8 — 2:52 (watch) Currently, everything operates primarily on homogeneous clusters, though we do see some variety in the prompts. In multi-agent systems, we may utilize different LLMs for various subagents, and we also employ a mixture of experts. As we increase heterogeneity, we could start using different chips for different models, placing distinct LLMs on separate GPUs. These models may interact with one another, incorporating a range of state-based models and diffusion models, all optimized for the best available hardware. The final stage of this heterogeneous paradigm emerges when we achieve co-evolution among systems, hardware, and software, leading to a unified, vertically integrated approach to intelligence and hardware. Slide 9 — 3:52 (watch) Heterogeneity is beneficial because real-world problems are complex, multi-step, and open-ended. These problems decompose into sub-problems that require vastly different types of intelligence. Scaling a singular type of intelligence to address these issues is inefficient and suboptimal. To solve them, we need models of various architectures and sizes working together over extended periods, which we refer to as multi-agent heterogeneous intelligence. Additionally, new generations of silicon are emerging in the market, but there is currently no interface that allows this new hardware to be unified and effectively enhance the existing compute stack. Slide 10 — 4:52 (watch) We aim to change this situation. The benefits of heterogeneity are not just a belief; we have formalized and mathematically proven them. The figure on the left illustrates the principle of maximum heterogeneity, showcasing heterogeneous agents where color represents a distribution over a skill space. When these agents communicate, as indicated by a ring topology, they can create what we refer to as a production function. This production function can effectively meet the demands of one problem while being ill-suited for another. For example, we have a production function that is well-suited for demand A but not for demand B. In a homogeneous approach, you would only be able to scale one peak, or in the optimal case, you would rely solely on generalists with the broadest possible skill set to match the demand function. Slide 11 — 5:40 (watch) Ultimately, a very short cylinder does not adequately meet the production function. We formalized this observation and found that across various domains, including neuroscience, economics, and ecology, these trends persist. Under reasonable constraints, heterogeneous systems consistently outperform homogeneous ones. Slide 12 — 6:08 (watch) To apply this in practice, we focus on optimizing multi-agent systems at three different stages of the workflow. First, we select different hardware based on the computational demands of the agents. Then, we consider how the agents interact and the workflows they construct. Slide 13 — 6:38 (watch) We have demonstrated multiple benefits of this type of orchestration. I want to highlight a couple of them: first, in the workflow, we have a concept we refer to as heterogeneous recursion. Second, in the agent layer, I will discuss multi-modal video action language models. Slide 14 — 7:08 (watch) Heterogeneous recursion refers to recursive language models. For those unfamiliar, a recursive language model is a seminal concept introduced in a paper from MIT last October. The research demonstrated that even when occupying only a small percentage of the context window, significant context shifts can occur depending on the information complexity required from the prompt. Slide 15 — 7:30 (watch) If you're performing a needle-in-a-haystack task, the information requirement remains constant, regardless of the size of the prompt. Slide 16 — 7:44 (watch) Adding up the rows, where you have rows and columns, results in O(N) complexity. As the prompt increases, the informational requirement grows linearly. Therefore, with a constant information requirement, the system scales effectively. Slide 17 — 8:00 (watch) You can utilize the full context window to obtain a good answer. However, when the complexity increases to linear or quadratic, performance degrades to about 30 to 60%. Recursive language models address this issue by treating the context as an environment instead of incorporating everything into the prompt. Slide 18 — 8:44 (watch) In practice, you present the context in a file, allowing the coding agent to interact with it programmatically through the Python REPL. This involves performing keyword searches and other techniques to extract subcontexts. The extracted subcontext is then passed to an identical recursive agent, which can either answer the question or spawn another recursive agent. This is the basis for recursive language models. We extended this concept by mapping the generated subcontext to different chips and models, rather than using a single model on a single chip. This approach emulates performance while being significantly cheaper and faster. Slide 19 — 9:12 (watch) Here are the results on the Oolong benchmark, which is the benchmark referenced in the paper. Slide 20 — 9:50 (watch) GPT-5 and GPT-5.2, the most recent versions, take approximately 2,000 seconds to complete the Oolong benchmark, costing around $3.75 per task. In contrast, our system on Cerebras is seven times cheaper and five times faster, allowing for significant savings in both time and cost. With SambaNova, we achieve even greater efficiency, being 12 times cheaper and three times faster, though this comes at the expense of some latency. These architectural decisions demonstrate that substantial cost reductions can be achieved without solely relying on hardware, while still emulating the intelligence of frontier models. Slide 21 — 10:28 (watch) The next problem we wanted to address is visual web navigation. Slide 22 — 10:36 (watch) We utilized a combination of open and closed video action language models, achieving a performance improvement over the state-of-the-art in video web navigation. Specifically, we surpassed GPT-5.2 and Gemini 2.5 by 18% and 25%, respectively. Slide 23 — 10:50 (watch) We approached the problem by recognizing its heterogeneous nature rather than treating it as homogeneous. Slide 24 — 11:04 (watch) The problem decomposes into multiple steps involving visual reasoning and textual reasoning. Each of these subcomponents requires different models to be successfully completed. Slide 25 — 11:16 (watch) Here, we observe a fundamental shift in the Pareto frontier, where singular models like Kimi K2.5 and GPT-5.2 are outperformed by a heterogeneous set of models. Slide 26 — 11:50 (watch) When we use Qwen3 VL 8B Instruct alongside Kimi K2.5, we achieve a speed increase of 1.3 times compared to using Kimi alone, and we are 18 times cheaper than using GPT-5.2 alone. Combining Qwen3 with GPT makes us three times faster and 3.7 times cheaper. This approach has no downsides; it solely provides benefits. One of our differentiating factors that allows us to surpass the state-of-the-art is that we mapped certain sub-tasks, such as zooming and creating different visual reasoning for the agent, to less intelligent models. For instance, GPT is not necessary for performing zoom functions. Slide 27 — 12:30 (watch) For these sub-tasks alone, we are 11 times faster and 43 times cheaper than using ChatGPT. This contributes to our overall performance, making us 3.7 times cheaper and three times faster. Slide 28 — 12:50 (watch) Looking ahead, we consider the future of compute. The first era of scaling compute was dominated by the CPU, which focused on increasing speed. The second era emphasized massively parallel computing, primarily driven by NVIDIA. Slide 29 — 13:08 (watch) The third paradigm of compute will be heterogeneous, focusing on mapping multi-agentic workloads optimally onto different chips. We are collaborating with ARIA, the UK institute. Slide 30 — 13:22 (watch) We received a £3 million grant to operate the first heterogeneous co-located cluster in the UK. Our goal is to make a significant impact and lead this new era of innovation. Slide 31 — 13:38 (watch) The era of homogeneous scale has delivered extraordinary progress, for which we should be grateful. The next phase is heterogeneous intelligence, where models, workflows, and silicon co-evolve. Each new source of diversity enhances the entire system, making it smarter, faster, and more cost-effective. Slide 32 — 13:52 (watch) This is the worst our infrastructure will ever be. Thank you. Slide 33 — 14:24 (watch) How do you define which tasks to run on the faster, cheaper model, such as a Zoom? Is this hard-coded, meaning if you need to Zoom, you use this model, or do you have a more intelligent system in place? Initially, we made bespoke decisions to map certain simple subtasks to specific models. However, we have since developed an automation layer that detects task complexity and automatically predicts the most suitable model and hardware. Any other questions? Thank you for your attention. Slide 34 — 14:54 (watch) My name is Adrian Bertagnoli. If anyone is interested, we are hiring. Thank you very much.

75 slides extracted. Slide 1 — 0:22 (watch) Welcome, everyone. Thank you for joining the workshop. It's great to see such a full house today. Quick show of hands: who here loves Claude Code? You're in the right place! It's lovely to meet you all. Slide 2 — 0:46 (watch) My name is Arno, and I am an architect on the Applied AI team. Today, I will explain how we Claude Code at Anthropic. We will conduct a workshop where you can code along and earn some credits. A QR code will be available shortly, so please be sure to grab that. There will also be people around to assist you with any technical difficulties you may encounter. Is there anyone in this room who hasn't used Claude Code before? Great! There’s plenty more to explore, and I'm glad you're here. Let's get started. Let me get my clicker. Ah, yes. Excellent. Slide 3 — 1:46 (watch) Please grab the QR code to set yourselves up. There will also be a repository that you can clone, which contains three phases for us to work through. There’s an interesting verification setup that you can follow along with, and it’s quite detailed, so you may want to explore it further later. What we’re covering today is based on the version of the talk that Tariq gave in San Francisco about a week and a half ago. Who here follows Tariq on Twitter? Fantastic. Slide 4 — 2:42 (watch) He published a blog post titled "The Unreasonable Effectiveness of HTML Files," advocating for a shift from markdown files to HTML. We will demonstrate some of that today, and there's a repository available for you to explore throughout the week. Currently, agents are becoming increasingly capable, which is a result of advancements in model capabilities. As models improve, agents can operate for longer periods and handle more complex tasks. However, this necessitates a change in our working habits. Today, we’ll discuss how to adapt your workflow with cloud code to maximize its potential. Allowing your agent to run for extended periods can lead to excessive token usage if it makes mistakes, so it’s crucial to mitigate that risk early on. ...