151 slides extracted.
Slide 1 — 0:16 (watch)
 | Can everyone hear me? Great. Today, I’m Frank McSherry from Materialize, and the title of my presentation is "Bug Batch." When I initially typed the title, I wasn't thinking too deeply about it, but I’ve come to appreciate it. It effectively conveys the topic, particularly the importance of using the right verb tenses, such as the gerund. This presentation will focus on building confidence—what it means and my perspective on it. For me, and I hope for all of you, building confidence is a process rather than a finished product. You are never done building confidence in something. |
Slide 2 — 1:54 (watch)
 | Building confidence is an ongoing process, not a finished product. It involves various elements that help you and those around you understand and evolve the work you are doing. I want to emphasize that what I share today is largely based on personal opinions, although it does include some technology discussions. You are encouraged to evaluate and reflect on these ideas and determine if they resonate with you. Each presentation today has been a personal journey, and I invite you to consider these insights thoughtfully rather than dismiss them outright. I also want to mention that I had a conversation with Gary yesterday, where I mentioned having some strong opinions on certain topics. While I decided not to include those hot takes in my slides to avoid controversy, I will share that my organization has seen significant benefits from neurosymbolic AI. I believe that the processes I have found effective for building confidence align well with the recent advancements in AI tools. These tools may not directly assist in building confidence but can facilitate effective implementation once confidence is established. Now, let me share some of my background. I have worked extensively on timely differential data flow, which involves distributed, scalable, data-parallel systems. These systems can be complex and potentially confusing. Today, I will primarily focus on these topics, using Materialize as a case study to illustrate how we can build confidence in these systems. Materialize has some unique and interesting features that I will discuss throughout the presentation. |
Slide 3 — 3:48 (watch)
 | Materialize is an innovative system that I find fascinating, and I hope to inspire you to see its unique aspects as well. It operates in a nonstandard way by performing incremental view maintenance on the SQL queries you write, even if you're not a distributed systems expert. The goal is to keep your queries continuously updated with sub-second freshness as your data changes. This involves addressing significant challenges in distributed systems, especially given that a petabyte of data is now a common scale rather than an impressive feat. Materialize has about a petabyte of deployed capacity, with live data that is constantly changing, where any record could be updated at any moment. It’s important to note that while I may not be an exceptional programmer, the effectiveness of Materialize is not solely due to programming skills. For example, I have never learned to use a debugger and I don’t particularly enjoy writing tests. However, I want to emphasize that the team at Materialize includes many talented individuals who excel in these areas. |
Slide 4 — 5:58 (watch)
 | Materialize is a component in data infrastructure that mirrors data from upstream source databases, such as Postgres, SQL Server, MySQL, and Kafka. It uses change data capture to bring data into Materialize and propagates changes through the SQL you write. This allows you to integrate various data sources, such as joining dimension tables from Postgres with fact tables from Kafka. The results are continually current, targeting subsecond freshness, although this can vary based on the SQL you write. Materialize ensures that every result is consistent and reflects the correct answer at a specific moment in time. It offers strict serializability by default, meaning it is impossible to achieve anything weaker than serializable isolation. The system uses standard SQL, making it accessible to users who may not want to deal with the complexities of distributed or streaming systems. Now, we'll move on to a demo to illustrate these concepts. |
Slide 5 — 8:06 (watch)
 | If you're on that side of the room, you're in for an amazing experience. I will narrate the demo to clarify what’s happening, especially if some aspects are less clear. This screen offers a better view than the one further along, but I will ensure to highlight the main points. There are three key aspects of Materialize that I want to emphasize during this demo. The first is responsiveness. This is similar to a standard database where you can build indexes on your data. In this case, we are working with auctions and bids, and we will create indexes to quickly retrieve information about them. You will notice that the responses come back quickly. This demo is recorded; I was advised against doing a live demo, which turned out to be wise, as it took considerable time to get this right even in a recorded setting. The data processing is happening in Virginia while I am in New York, illustrating how data centers operate. We are pulling data from our auctions and bids, which includes about 11 million bids, and the retrieval is quite fast. However, this may not seem particularly exciting yet. |
Slide 6 — 9:20 (watch)
 | Indexes have been used on data for a long time, so there’s nothing particularly fascinating about that at this stage. However, Materialize takes a different approach by recognizing that most people find data boring; they are merely inputs for more interesting business logic. For example, we have a winning bids view that combines auctions and bids through joins and aggregations to determine when an auction closes and identify the highest bidder who submitted their bid before the end. This process involves complex business logic to determine the auction winner. While optimizing indexes can improve query performance, Materialize allows you to create an index on the results of this view, keeping them indexed in memory and continuously updated. In just a few milliseconds, we can retrieve the current results, which are live and changing. |
Slide 7 — 10:02 (watch)
 | It's synthetic data, but it is live and changing. I apologize for racing ahead in the demo to show you what is happening behind the scenes. |
Slide 8 — 10:42 (watch)
 | We are serving results to you, but behind the scenes, we have a continually evolving changelog. On the left side of the screen, you can see timestamps in milliseconds since 1970, which is the correct way to represent time. The plus and minus signs indicate inserts and deletes, respectively. The first five elements in this list are plus ones. Since this query limits the results to five, there are only ever five items in the collection at a time. If you examine it more closely, I can show you this offline for a deeper understanding. The changes are coupled; plus ones and minus ones occur simultaneously as a record is inserted while another is ejected at the same time on the left side. |
Slide 9 — 11:34 (watch)
 | This is essentially a high-fidelity changelog that reflects the system's activities. We will present this to you as a live feed of business logic in real-time. This serves as the backbone of our technology, enabling us to communicate and manage fresh data while maintaining correctness and consistency. The second aspect is freshness. It's impressive to see data streaming by, but we need to ensure it's not delayed or pre-recorded. We'll demonstrate this by interactively navigating the data, focusing on winning bids and identifying the top bidders. We have some concerns about potential fraudulent accounts, so we will introduce a new table for fraud accounts. Any accounts listed there will be excluded from our big winners list. Please pay attention to how quickly this updates as we insert records into the fraud accounts table. |
Slide 10 — 12:44 (watch)
 | Do not count fraudulent accounts in our list of big winners. Watch how promptly the system updates. In the lower left, we will insert records into the fraud accounts table, which will trigger a change in the change log. Currently, the change log is not updating because we are viewing the top winners of bids by their account. It will update, but it is not live at the moment. The change does not occur simply because an auction has closed; the individuals with seven and eight bids are not being replaced by the latest auction winners. We will take the account with eight bids, 55626, and add it to the fraud accounts. You may have missed it, but it was immediately removed from the list. The difference is that 55626 is out, and 21259 is in. We will demonstrate this again to show that each time we add a new value to the fraud accounts, the change log updates immediately. |
Slide 11 — 13:10 (watch)
 | Each time we add a new value to the fraud accounts, the change log updates immediately. If we determine that this experiment was unnecessary, we can delete all three entries in the fraud accounts, removing all overrides. |
Slide 12 — 14:36 (watch)
 | We discard the most recently inserted element and revert to the original value of 55626. This is a transactional cutover from the end of our previous solutions back to the beginning. This leads us to the concept of materialization and consistency. In distributed systems, rapid updates can lead to inconsistencies; numbers may not add up correctly due to timing differences in updates. Materialize addresses this issue. We have defined views for credits and debits: individuals who win auctions incur debits, while sellers receive credits. Although credits and debits are processed on the same machine, they could also be on separate machines. When querying the sums of credits and debits, discrepancies may arise, which is expected given the dynamic nature of the data. If we combine the two sums, we find that the difference is actually zero, which aligns with Bookbash's promotional material stating there is zero slop. |
Slide 13 — 15:44 (watch)
 | I did not realize this when I prepared the demo, but we have a column labeled "slop," and it is zero. It will always remain zero. We will run this query a few more times. |
Slide 14 — 16:04 (watch)
 | The value will always be zero, and I am confident in this. We will set up a subscription to display the full changelog. I initially thought I needed to pause it, but I don't; it will never change. At this point, there was a single record with a value of zero. Let's observe it for a moment. |
Slide 15 — 16:28 (watch)
 | If this ever changed, it would be a P0 level bug at Materialize, indicating a fundamental correctness issue with the underlying infrastructure. Fortunately, I stopped the recording before that occurred. |
Slide 16 — 16:42 (watch)
 | The goal of this presentation is to pique your interest in how you can be confident in the correctness of the information that Materialize produces. Materialize's motto is "time to confident action." This refers to the time it takes for us to provide you with information that prompts you to take an action, which can be difficult to reverse once executed. |
Slide 17 — 17:10 (watch)
 | Systems that function effectively do so for specific reasons. Their success is not solely due to extensive testing; there must be a foundational rationale for why the system is designed to work. |
Slide 18 — 17:38 (watch)
 | Tests are a helpful component in ensuring that your implementation aligns with your intended reasons, but they are not the source of those reasons. They contribute to absolute tests and the overall confidence narrative, but the foundation of that confidence lies in a clear rationale. I view this rationale as an abstraction that explains why things should work. |
Slide 19 — 18:28 (watch)
 | This quote is often cited by practitioners who enjoy challenging theoretical perspectives. As someone who started as a theoretician, I recognize that while theory and practice can seem similar, they diverge significantly in application. In the physical sciences, if your theory of gravity does not align with its practical outcomes, the theory is flawed and needs revision. However, computer systems, particularly distributed systems, operate differently. The practice we develop may not always align with theoretical expectations, and that discrepancy can indicate issues within the system itself. There is no inherent truth dictating how a distributed system must function; it could simply be poorly designed. I gained valuable insights during my time at Microsoft Research in Silicon Valley, a hub for both theorists and practitioners in distributed systems. They believed that any gap between theory and practice indicated an area for improvement. I often approached the systems with a naive theoretical perspective, pointing out inconsistencies, and the practitioners would acknowledge that my expectations were valid. This process of narrowing the gap between theoretical expectations and actual performance is crucial for establishing and building confidence in these systems. |
Slide 20 — 19:30 (watch)
 | Materialize's reason for this discussion is the unexpected prevalence of virtual time among attendees. This concept, introduced in a 1985 paper by David Jefferson, will be explored further. |
Slide 21 — 19:44 (watch)
 | The abstract identifies two uses for virtual time. The first is in discrete event-based simulations, which many people are currently utilizing. The second use case is for concurrency control. |
Slide 22 — 19:56 (watch)
 | We will use virtual time for concurrency control. I will explain what that means, but in short, it is essential for our approach. |
Slide 23 — 20:02 (watch)
 | Materialize and its underlying stack function as a large-scale simulator. |
Slide 24 — 20:10 (watch)
 | It simulates potential computations rather than simply reacting to external stimuli. |
Slide 25 — 20:16 (watch)
 | It is performing a prescribed sequence of computations. |
Slide 26 — 20:20 (watch)
 | We previously observed this on the screen. |
Slide 27 — 20:30 (watch)
 | The backbone of Materialize consists of continually changing and evolving changelogs. These changelogs evolve only in the sense of appending new information. We are gaining a deeper understanding of our underlying data. |
Slide 28 — 20:40 (watch)
 | We document them as triples, which may vary slightly. In this case, we focus on time difference data, as displayed on the big screen. |
Slide 29 — 20:44 (watch)
 | Time stamps are recorded as plus or minus ones, along with some record payloads. |
Slide 30 — 20:50 (watch)
 | As these collections evolve and grow in length, they provide a specific understanding of their contents at particular moments in time. |
Slide 31 — 21:00 (watch)
 | You aggregate all the changes up to the specified time, summing the depths for each data point. Any data point with a non-zero value reflects that multiplicity in your dataset. |
Slide 32 — 21:12 (watch)
 | The backbone of Materialize is its continual changelog, which allows for the addition of negative numbers. While it may seem unusual, this approach is entirely feasible. |
Slide 33 — 21:16 (watch)
 | Is this just an accounting technique? |
Slide 34 — 21:20 (watch)
 | This process is not simply about writing things down and being finished. |
Slide 35 — 21:28 (watch)
 | Materialize uses changelogs as the nouns, while the verbs act on these changelogs to transform them. For example, we have a changelog describing some input data, referred to as input zero, and a user may decide to apply a filter to it. |
Slide 36 — 21:44 (watch)
 | This could be a dataset containing information about individuals, and the filter specifies that we want to retain only those individuals whose ages are even. |
Slide 37 — 22:02 (watch)
 | There is a computational problem when given an input changelog with explicit timestamps, including additions and deletions of data. The filter operator must produce a corresponding changelog in its output that reflects a frozen state at every moment. It integrates the input, retains only those individuals whose ages are even, and then differentiates the output to identify the changes that recreate this filtered state. This process results in the appropriate output from the filter. |
Slide 38 — 22:22 (watch)
 | You can apply this approach to various types of operators, such as map, join, and reduce, which are fundamental to data parallelism in computation. The exciting aspect is that there is a precise high-fidelity match between the input changelog and the output changelog, allowing these components to function like Lego bricks. |
Slide 39 — 22:30 (watch)
 | You can easily connect them together. |
Slide 40 — 22:34 (watch)
 | If you filter input zero and then map a function onto input one, you can join the two together and reduce them. This process is akin to going through input one and input zero, freezing time at each moment, integrating the results, applying filter, map, join, reduce, and then differentiating the output. |
Slide 41 — 22:46 (watch)
 | That property holds true regardless of your expertise with distributed systems. |
Slide 42 — 23:04 (watch)
 | Even if you are not experienced with distributed systems, as a data scientist or business analyst, you can use SQL to assemble relevant data without worrying about correctness or the complexities of asynchronous computation. |
Slide 43 — 23:18 (watch)
 | This feature is very useful. The certainty regarding the equivalences between inputs and outputs allows you to confidently assemble various components. |
Slide 44 — 23:26 (watch)
 | Moreover, the SQL plans themselves indicate that computations at any scale can also be composed effectively. |
Slide 45 — 23:38 (watch)
 | If you have a piece of business logic and someone else in a separate room has a different piece of business logic, both of you can execute your logic simultaneously. A third person can then take your results, combine them, and ensure consistency between them. |
Slide 46 — 23:48 (watch)
 | It will always be as if the input data was frozen, both of your bits of logic were executed, and then the third person's logic was applied to produce the change log for the results. |
Slide 47 — 24:08 (watch)
 | This approach simplifies the process for users who want to perform data analysis using SQL or other languages. The goal is to build confidence in the system, which is not solely based on speed or a complex locking discipline. |
Slide 48 — 24:24 (watch)
 | From a systems perspective, in large and complex distributed systems, we discussed determinism and non-determinism yesterday. One way to understand this is that Materialize addresses much of the non-determinism at the system's boundary. |
Slide 49 — 24:34 (watch)
 | Materialize allows for extensive functionality within the system's confines. You can choose how to assemble the various components, and they are predominantly deterministic at that stage. |
Slide 50 — 24:46 (watch)
 | The computation results are determined by the resolution of change logs at the boundary as data is brought in from Postgres, MySQL, and other sources. While the process may not be entirely deterministic, the outcome is influenced by these factors. |
Slide 51 — 24:56 (watch)
 | From a system perspective, it effectively eliminates logical contention. |
Slide 52 — 25:12 (watch)
 | It eliminates complicated coordination and synchronization issues that could have multiple outcomes at runtime. This simplifies the process, making it primarily a performance question focused on how to optimize speed. |
Slide 53 — 25:22 (watch)
 | While there is a potential for errors, the design aims to minimize user mistakes. Ideally, users should not be able to make significant errors in this system. |
Slide 54 — 25:34 (watch)
 | It's important to clarify the distinction between virtual time as a concurrency control mechanism and logical clocks. Many people associate time with clocks and may think they understand logical clocks. |
Slide 55 — 25:48 (watch)
 | You're likely thinking of vector clocks or Lamport clocks. |
Slide 56 — 25:58 (watch)
 | The fundamental difference is that virtual time is a prescriptive technique, while logical clocks are descriptive. Virtual time introduces a structure that limits the available options. |
Slide 57 — 26:06 (watch)
 | The system now has significantly fewer degrees of freedom than it previously did. However, this reduction also simplifies the overall complexity. |
Slide 58 — 26:12 (watch)
 | Users of the system no longer have to manage certain issues that could have arisen, which can be a significant relief. |
Slide 59 — 26:22 (watch)
 | Logical clocks, in my view, do not eliminate complexity; rather, they transcribe it. |
Slide 60 — 26:38 (watch)
 | Logical clocks remove ambiguity by clarifying what events did not occur. This can be reassuring, but programming with logical clocks can be challenging. If you were given a logically clocked change log and asked to derive the correct answer, you would need to understand logical clocks, which are inherently complex. I wanted to highlight this distinction as it leads into the next topic about building competence. |
Slide 61 — 27:00 (watch)
 | I spent a considerable amount of time thinking that my goal was to be perceived as a really smart person. I focused on presenting myself in a way that reflected that intelligence. |
Slide 62 — 27:16 (watch)
 | Ultimately, what are computer scientists uniquely good at? Where do we provide the most value? |
Slide 63 — 27:24 (watch)
 | In my personal experience, effective abstraction has been a key strength. There are many opportunities for talented individuals in this area. |
Slide 64 — 27:30 (watch)
 | One of our strengths is our ability to take complex concepts and reduce their surface area, making them easier and more accessible for those who prefer not to engage with that complexity. |
Slide 65 — 27:42 (watch)
 | In mathematics, a 150-page paper can be met with enthusiasm, and a 200-page paper may receive even more praise. |
Slide 66 — 27:50 (watch)
 | In reality, none of our customers want to read a 200-page document. They prefer simplicity and ease of use. For example, my computer contains a lot of complex silicon that I do not understand. |
Slide 67 — 28:06 (watch)
 | The operating system is complex. While its intricacies are fascinating for those who wish to explore them, I prefer to focus on delivering a presentation. It is powerful to conceal that complexity, allowing people to engage without having to worry about it. |
Slide 68 — 28:16 (watch)
 | It's not necessarily as interesting. Many of you may feel that you want complexity. |
Slide 69 — 28:24 (watch)
 | We aim to remove complexity for others, which is a key aspect of our service. Often, we are compensated for successfully achieving this goal. |
Slide 70 — 28:38 (watch)
 | Virtual time is a valuable abstraction. When implemented correctly, it is not overly complicated, but it can be challenging to misuse. |
Slide 71 — 28:50 (watch)
 | If someone gives you a collection of pieces, like Lego bricks, and asks you to build something, you would likely use them appropriately. You wouldn't resort to using a hammer to smash them together or a blowtorch to melt the pieces. |
Slide 72 — 28:56 (watch)
 | The pieces fit together nicely and logically. |
Slide 73 — 29:00 (watch)
 | You obtain a result that you understand. |
Slide 74 — 29:04 (watch)
 | You may not fully understand how or why it works, but you receive the correct answer, which is satisfying. It's a relief not to have to learn about distributed systems to use this effectively. |
Slide 75 — 29:22 (watch)
 | We will now discuss several vignettes related to the abstraction of virtual time. This abstraction is beneficial not only for users but also for various system-building functions. |
Slide 76 — 29:32 (watch)
 | Several aspects are straightforward and materialized, but they are simpler than they would be in a complex system that exposes all its internals. In such a system, careful handling is required for everything to function properly. |
Slide 77 — 29:38 (watch)
 | We can conduct a brief demonstration of active replication at the end of the previous slide. |
Slide 78 — 29:46 (watch)
 | All forms of parallelism are generally straightforward, though I hesitate to say they are easy. |
Slide 79 — 29:52 (watch)
 | Task parallelism is relatively straightforward, though it can be challenging to define. |
Slide 80 — 29:56 (watch)
 | If five people want to use the same changelog, they can proceed without hesitation. This approach will yield consistent results at the end. |
Slide 81 — 30:10 (watch)
 | Differential data flow relies on data parallelism, allowing workers to operate independently without constant coordination. This enables them to work efficiently and focus on their tasks without confusion about what to do next. |
Slide 82 — 30:24 (watch)
 | Pipeline parallelism is an excellent example of where virtual time excels. In a sequence of tasks A, B, and C, task B cannot begin until task A has completed a portion of its work. |
Slide 83 — 30:34 (watch)
 | Virtual time records the intended start time for tasks, even if there is a delay before they can actually begin. |
Slide 84 — 30:40 (watch)
 | Pipelining is not easy, but it is relatively straightforward. Queries, which are another form of interaction with the system, receive virtual times. |
Slide 85 — 30:52 (watch)
 | This setup provides serialization by default. The system maintains a total order on time, meaning that every query you make is associated with a specific time, and you receive the answer exactly as of that time. |
Slide 86 — 31:02 (watch)
 | If you have additional constraints on time, such as a specific lower bound, you can achieve strict serialization. |
Slide 87 — 31:08 (watch)
 | A notable property is that you obtain something stronger than serialization; it is composable strict serializability. |
Slide 88 — 31:16 (watch)
 | By revealing these timestamps, multiple users can combine two strict serializable databases to create a third one. However, I prefer not to have to do that. |
Slide 89 — 31:32 (watch)
 | With virtual times, their structure is revealed in a way that allows for composition, enabling the creation of even more fascinating composed systems. |
Slide 90 — 31:44 (watch)
 | Errors, which often manifest as behavior, are simply data within these systems. |
Slide 91 — 31:54 (watch)
 | An error does not disrupt the system or prevent you from performing an action. For instance, if you attempt to divide by zero, the system will register an error indicating a divide by zero situation at that virtual time. If you remove that record from the input, the error will disappear immediately upon removal. This feature is quite useful. |
Slide 92 — 32:10 (watch)
 | Integrity constraints are often overlooked in streaming systems or cloud data warehouses because it's uncertain when the data actually arrives in the environment. Primary and foreign key relationships do not hold, so it's advisable to use left joins everywhere. However, in systems like Materialize, these constraints are maintained, allowing you to confidently convert all outer joins to inner joins based on these properties. |
Slide 93 — 32:24 (watch)
 | Finally, active replication, which I will demonstrate, ultimately results in deduplication. |
Slide 94 — 32:34 (watch)
 | When multiple people are working on the same task, you only need one of them to provide the answer. Once they inform you of the next item in your change log, you can proceed with that information, even if one of them is unavailable. |
Slide 95 — 32:50 (watch)
 | As long as one person is responsible for the work, it will get done. I had a few other examples, but the key takeaway is that simplifying your processes makes everything easier. Some tasks can be challenging, and at the end of the talk, feel free to ask me about those challenges; I may need to navigate that question carefully. |
Slide 96 — 33:00 (watch)
 | Here’s a quick demonstration of active replication that will take about a minute or two. |
Slide 97 — 33:20 (watch)
 | We are performing the same operation as before, focusing on the winning bids. We will use a modifier for this command to demonstrate the presence of several nulls, indicated by the slashes. This modifier ensures a steady heartbeat in the background, even if no changes occur. This ticking sound signifies that time is progressing, which is important as we will intentionally break some components to observe the outcomes. While this process continues, Materialize maintains what are known as cluster replicas behind the scenes. |
Slide 98 — 33:40 (watch)
 | A cluster defines the computation you want to perform, while the replicas are the engines that generate the results. |
Slide 99 — 33:48 (watch)
 | You can have any number of replicas, including zero. One is a common choice, and two is also used frequently. Having zero replicas is typically a mistake, but we will proceed with that configuration. |
Slide 100 — 34:02 (watch)
 | We have a default cluster running, and we just dropped a replica. As a result, the highlighted line has stopped changing, which is unusual. |
Slide 101 — 34:12 (watch)
 | This is exactly the desired behavior in a virtually timed system. We cannot predict what will happen next. |
Slide 102 — 34:18 (watch)
 | In particular, we cannot assume that nothing has changed. While we cannot definitively say that this is the wrong answer, we are not certain that it is the correct one either. |
Slide 103 — 34:22 (watch)
 | We need to pause the feed. |
Slide 104 — 34:28 (watch)
 | It needs to stop, and it is clear that it has. As consumers, we are left wondering what comes next, but the system is unable to provide an answer. |
Slide 105 — 34:52 (watch)
 | We are currently creating new replicas and bringing them back online. This process involves seeing how a replica resumes its functions. Additionally, I am starting a second replica. I want to give you a heads-up that we will perform a follow-up action shortly. As the first replica comes back online, it needs to rehydrate its data flows and start pulling down the change logs to get back up to speed. It has just resumed, and I paused it at this highlighted line. |
Slide 106 — 35:14 (watch)
 | If you examine the numbers closely, you may find them difficult to see, but the change log is essentially uninterrupted. |
Slide 107 — 35:22 (watch)
 | Although we experienced an interruption, the change log remains essentially uninterrupted, as if we had continued running without pause. |
Slide 108 — 35:32 (watch)
 | If we had the same computation running on a different cluster without interruption, we would observe the same information in the same order, aside from the minute-long pause during which no activity occurred. |
Slide 109 — 35:58 (watch)
 | Currently, we have two replicas running: a 400cc and an 800cc, which are our internal units. The 400cc is half the size of the 800cc. We are going to drop the 800cc, but this time we have a backup with the 400cc operational. It takes me a moment to gather the confidence to press the button, but once we stop the 800cc, the process continues. We have successfully performed a live migration from the 800cc to the 400cc without interruption, aside from the brief moment when I demonstrated the interruption. This transition allows us to reconfigure the physical part of the system and switch from one class of machine to another seamlessly. |
Slide 110 — 36:24 (watch)
 | Zero downtime physical replication is a principle that applies in various other settings as well. |
Slide 111 — 36:40 (watch)
 | Logical reconfiguration allows you to change the business logic of your view. Transitioning from one configuration to another can be challenging, but it is not unexpected. |
Slide 112 — 36:54 (watch)
 | The final point I want to address is brief, as this is the last slide. |
Slide 113 — 37:08 (watch)
 | There are fewer demonstrations in this section, and while "hand wave" isn't the right term, I feel strongly about these concepts. It is relatively straightforward to identify reasons why technology should work and to create abstractions. This process helps build your own confidence in the technology. |
Slide 114 — 37:18 (watch)
 | Building confidence involves figuring out how to transfer that confidence to others. Sharing your personal conviction is interesting, but not my primary focus. |
Slide 115 — 37:32 (watch)
 | As a user or participant in your technology, I want to understand its purpose. Convince me, not just yourself. Ideally, everyone you interact with should ask the same question: "Convince me." I want to be involved and comprehend what’s happening, or know when I don’t need to understand. |
Slide 116 — 37:44 (watch)
 | I recommend three key actions, though this is not an exhaustive list. |
Slide 117 — 37:58 (watch)
 | Drug footing is crucial. Utilize the system you are developing; if you don't engage with it, the validity of your statements is questionable. |
Slide 118 — 38:08 (watch)
 | Using these systems reveals how your initial assumptions may have been incorrect or how you might have overlooked the real challenges associated with the technology. It is essential to engage with these tools actively to validate your theories about their effectiveness and their potential to simplify processes for everyone involved. |
Slide 119 — 38:22 (watch)
 | Benchmarking is something I strongly advocate for. |
Slide 120 — 38:30 (watch)
 | I often engage in performance-related work. If this area is not your focus, it may seem less relevant. However, I encourage you to test the limits of your system. |
Slide 121 — 38:40 (watch)
 | In structural engineering, professionals understand the limits of materials, such as the point at which steel bends. Similarly, if you do not know the limits of your system, I doubt it will withstand significant stress. |
Slide 122 — 39:00 (watch)
 | Challenge yourself by actively testing the limits of your system. Attempt to break things and identify failure points. Avoid relying solely on simple benchmarks; instead, engage in more complex evaluations that may complicate your work. |
Slide 123 — 39:04 (watch)
 | The advantage of thoroughly testing is that you uncover hidden issues not only in your own system but also in others. |
Slide 124 — 39:16 (watch)
 | If your goal is to tell a compelling story, you'll create an extensive checklist of challenges to overcome. It can be difficult to achieve these goals, and if you succeed, it will be impressive, prompting others to learn about your methods. |
Slide 125 — 39:26 (watch)
 | It's important to thoroughly test your ideas and identify where they may fail. Additionally, I strongly recommend maintaining open communication with others about your progress and challenges. |
Slide 126 — 39:38 (watch)
 | I have historically written many blog posts, and often, while writing about a fascinating idea, I realize halfway through that it's not a good concept. I find myself thinking that the idea is trash and horrible. |
Slide 127 — 39:54 (watch)
 | I attempt to explain the value of the idea, which requires balancing numerous elements at precise angles. Often, I end up deleting the post and going back to improve it. |
Slide 128 — 40:00 (watch)
 | The exercise of communicating and bringing others on board is essential. |
Slide 129 — 40:10 (watch)
 | Explaining how simple and easy this process can be is both challenging and informative. It tests the theory that this information is valuable and can genuinely simplify people's lives. |
Slide 130 — 40:26 (watch)
 | The final takeaway is that building confidence is something you provide to others. It is not solely a technical issue; confidence cannot be solved by a piece of software. |
Slide 131 — 40:36 (watch)
 | Building confidence is a process that involves many components working together to achieve a successful outcome. |
Slide 132 — 41:04 (watch)
 | Twenty years ago at Microsoft, a colleague of mine focused on theoretical computer science, algorithms, and data structures. He suggested to Jim Gray that more people should work on fundamental algorithms and data structures, emphasizing their importance. Jim responded, conveying that everyone is indeed working on these areas, but my colleague was the one without a clear purpose. This raises the question: why are you pursuing this work? Are you building confidence for a specific reason? |
Slide 133 — 41:22 (watch)
 | Is your study for others, or is it for yourself? Is it a self-indulgent pursuit? |
Slide 134 — 41:32 (watch)
 | Software reliability can easily lead to the misconception of creating highly reliable systems in the abstract, without a clear understanding of what reliability truly entails. |
Slide 135 — 41:40 (watch)
 | Reliability has different implications for a self-driving car compared to a pacemaker. |
Slide 136 — 41:46 (watch)
 | One of these systems should shut down carefully and in a controlled manner, while the other should not shut down at all. One of the main reliability questions we have not addressed is related to performance. |
Slide 137 — 41:54 (watch)
 | With Black Friday approaching, will the system remain operational as I conduct business? |
Slide 138 — 42:04 (watch)
 | The software is not expected to crash, but we are encountering fewer glitches related to computed results. Instead, we are facing issues with non-functional requirements. For example, we did not anticipate that a specific collectible doll would become critically important, which led to significant problems with the system. |
Slide 139 — 42:16 (watch)
 | Design bugs are a crucial aspect of reliability, often more significant than pointer access bugs or similar issues. |
Slide 140 — 42:30 (watch)
 | Understanding reliability involves considering its broader implications beyond your own perspective. It's essential to identify who needs to be convinced and what specific aspects of reliability are important to them. |
Slide 141 — 42:38 (watch)
 | I will pause here. We have time for one or two questions. If the questions are particularly challenging, we may have less time to address them. |
Slide 142 — 43:00 (watch)
 | Abstraction is a powerful concept that resonates strongly. The virtual time abstraction is particularly effective, and it raises questions about monotonicity. How did you develop the virtual time abstraction? Additionally, what tips do you have for creating effective abstractions? What is the process involved? These are challenging questions. |
Slide 143 — 43:20 (watch)
 | To clarify, I did not originate the concept of virtual time; that was developed by Jefferson in the 1980s. |
Slide 144 — 43:40 (watch)
 | I arrived at the concept of virtual time primarily through my experiences with streaming. In that context, the term "event time" is used instead of "virtual time." Streaming has a unique property compared to databases: it allows an external source to inform you about the sequence of events. In a streaming system, events arrive with timestamps indicating when a particular sensor recorded specific data. |
Slide 145 — 43:54 (watch)
 | I apologize for the delay in communicating this, but this is how the system needs to function. |
Slide 146 — 44:00 (watch)
 | In the background of much of the streaming work, event time has been a key concept. When you unpack it, you realize that it essentially represents virtual time. This idea has existed already; it is not a fundamentally new concept. |
Slide 147 — 44:44 (watch)
 | Much of my work has focused on the concept of composability. If a concept only works in a specific context and must be maintained precisely, I find it lacks lasting value. Differential privacy illustrates this well; while there are many definitions of privacy, they often lack composability. When you combine two definitions, they may lose their individual properties, which is unfortunate. One appealing aspect of differential privacy is its composability; even those without expertise can use it effectively and piece it together without a deep understanding. This serves as an important litmus test for evaluating the quality of an abstraction. However, it’s not a foolproof recipe. If I had a definitive recipe, I wouldn’t share it; I would be collaborating with Claude to explore every field of study. There are ways to assess why an abstraction might not succeed. You can ask questions like: Why did I think this was a great abstraction? What challenges did it face? Did it fail to address the problem people had? Can users not figure out how to use it? These rhetorical questions can help identify where things went wrong. |
Slide 148 — 45:18 (watch)
 | The cost paper discusses evaluating distributed systems against inadequate baselines, specifically a strong single-threaded baseline. We need to consider how to adapt this concept to software correctness. What should developers prioritize before engaging in elaborate testing? Is it implementing a strong type system? What constitutes the baseline in this context? |
Slide 149 — 46:54 (watch)
 | This is an important question, and while I don't have a quick answer, I want to emphasize a key aspect that is often overlooked: the role of formal methods in defining what a program is supposed to do. A common issue is that developers spend a significant amount of time programming and testing, but they often neglect the critical thinking phase where they define the specifications. It's essential to articulate what the software is intended to achieve. In my experience, and as supported by common wisdom, engaging in formal modeling can help clarify these specifications. Often, just the process of writing down the intended behavior reveals misunderstandings about the software’s functionality. This realization can lead to identifying bugs before they become problematic. For me, using Rust has been particularly beneficial. Its type and trait systems have helped me be more precise in my programming. The type system can indicate discrepancies, such as when a declared type does not align with the intended functionality. Therefore, it’s crucial to utilize tools that ensure the components of your program fit together correctly before diving into complex implementations. While there may be a temptation to rely on quick fixes or “pain medicine” for coding challenges, it’s more effective to step back and reassess what went wrong in your understanding of the requirements. Transitioning to a stronger programming language can often resolve many issues, so I recommend moving away from C and exploring better alternatives to see if that alleviates your problems. |
Slide 150 — 47:58 (watch)
 | Thank you. |
Slide 151 — 48:14 (watch)
 | Thank you. |