Hannes Mühleisen

Key Insight: The data architecture should be flipped upside down: instead of powerful central servers serving dumb clients, empowered client devices with local query engines should do the compute while sharing data through commodity storage and open lakehouse formats.

Hannes Mühleisen traces the evolution of data architecture from the 1985 three-tier model through cloud data warehouses to a proposed inverted architecture where compute happens on client devices. He argues that the traditional top-down pyramid—where a central data warehouse holds all power—has barely changed despite moving to the cloud, and that modern single-node query engines like DuckDB combined with lakehouse formats can flip this model. He demonstrates DuckDB processing 6 billion rows on a laptop with only 2GB of memory, and shows examples of running analytics on 13-year-old laptops, phones, satellites, and cars. The core proposal is that clients should be empowered with local compute while sharing data through commodity storage and open metadata formats.

Key Insight: By replacing file-based metadata with a SQL database, DuckLake dramatically simplifies the lakehouse stack while enabling features like instant small writes, cheap snapshots, and file-level encryption that are difficult or impossible with existing formats.

Hannes Mühleisen and Mark Raasveldt discuss DuckLake, their reimagination of lakehouse formats that replaces file-based metadata (JSON, Avro) with a SQL relational database for all metadata management while keeping data files on object stores. The project emerged from implementing Iceberg and Delta support in DuckDB and realizing the catalog database could handle everything more simply. DuckLake offers unique features like data inlining for small writes, file-level encryption with keys stored in the catalog, and cheap snapshots. The format scales from a single local DuckDB file to thousands of compute nodes, and can serve as a backend for the Iceberg REST catalog. They position it as DuckDB's multiplayer mode, fundamentally flipping the traditional architecture to make client-side compute the primary player.

Key Insight: By replacing file-based lakehouse metadata with a standard SQL database, DuckLake eliminates enormous complexity while preserving all the benefits of open table formats, enabling deployment scenarios from a single laptop to thousands of distributed nodes.

Hannes Mühleisen and Mark Raasveldt introduce DuckLake, a reimagined lakehouse format that replaces file-based metadata (JSON, Avro) with a SQL relational database for all metadata management while keeping data files on object stores. The project emerged from critically examining the Iceberg and Delta tech stacks while implementing support in DuckDB, realizing the catalog database could handle everything. DuckLake dramatically reduces complexity, cutting query round trips from ~7 to 2, and enables features like data inlining (small writes go directly to the metadata store), per-file encryption with a single boolean flag, and cheap snapshots. The format scales from local single-file setups to thousands of distributed compute nodes, and can serve as a backend for Iceberg REST catalogs.

Key Insight: The enduring abstractions of tables, SQL, and ACID are fundamental civilizational technologies whose elegance should not be confused with the limitations of their oldest implementations.

Hannes Mühleisen argues that database abstractions—tables, SQL, and ACID—are fundamental civilizational technologies deserving of enthusiasm rather than dismissal. He traces tables back 5,000 years to Mesopotamian clay tablets, predating written text by a millennium. He contends that criticisms of databases conflate old implementations (like Oracle) with the underlying abstractions, which remain elegant and powerful. He uses MongoDB as a cautionary tale: by initially rejecting tables, SQL, and ACID, it forced developers to reimplement those concepts poorly in application code. He concludes with a technical demonstration of DuckDB's vectorized execution achieving near-theoretical hardware limits.

Key Insight: The lakehouse world spent years trying to eliminate databases from metadata management only to add them back via catalogs, proving that a simple SQL database for metadata plus object storage for data was always the right architecture.

Hannes Mühleisen presents DuckLake, a new lakehouse format that replaces Iceberg's complex file-based metadata architecture with a simple SQL database for catalog management. He traces the evolution from single-node databases through shared-nothing architectures to separated storage and compute, showing how Iceberg ironically reintroduced the database it tried to eliminate via its REST catalog. DuckLake acknowledges this reality by using a proper database for metadata alongside object storage for data files, eliminating layers of Avro and JSON metadata files. The result is dramatically simpler setup, fewer round trips, and 4x better performance than Iceberg in benchmarks.

Key Insight: If you already need a database to manage your data lake catalog, you should redesign the entire metadata layer around that database rather than bolting one onto a file-based format that was designed to avoid needing one.

Hannes Mühleisen discusses DuckLake, a new open data format he and Mark Raasveldt created as an alternative to Iceberg and Delta Lake. He argues that Iceberg's fundamental problems are conceptual, not implementation-based, particularly the bolted-on catalog server that broke its original file-only design philosophy. DuckLake's key insight is using an existing database (like Postgres) to store all metadata—tables, columns, files, statistics—eliminating the need for Avro, JSON, REST APIs, and Java catalog servers. The approach enables sub-second query planning even at petabyte scale with 100 million snapshots, high-frequency transactional updates, and dramatically simpler implementation for connector authors.

Key Insight: Open table formats unnecessarily reinvented metadata management outside databases due to vendor lock-in fears, and DuckLake corrects this by putting metadata back where it belongs—in a SQL database—while keeping the good parts of decoupled blob storage.

Hannes Mühleisen introduces DuckLake, an evolution of open table formats that replaces complex metadata layers (Avro files, JSON, REST APIs) with a standard SQL database for metadata management while keeping Parquet files on blob storage. He explains how examining Iceberg's specification revealed unnecessary complexity born from an emotional reaction to vendor lock-in rather than sound technical reasoning. DuckLake separates storage, compute, and metadata into three independently scalable axes, enabling features like cross-table commits, efficient small writes, per-file encryption, and time travel without the snapshot management overhead of existing formats. The system works with any SQL database supporting transactions and primary keys as the metadata store.

Key Insight: Since every lakehouse format already requires a database for its catalog server, the logical step is to store all metadata directly in that database, eliminating the complex file-based metadata layers and unlocking multi-table ACID transactions, fast small commits, and zero-trust encryption.

Hannes Mühleisen and Mark Raasveldt introduce DuckLake, a new lakehouse format that replaces the file-based metadata layers of Iceberg and Delta with a standard database (like Postgres or DuckDB) for metadata storage while keeping Parquet files on blob storage for data. They argue that existing lakehouse formats made poor design decisions by trying to avoid databases, only to eventually require catalog servers backed by databases anyway. DuckLake simplifies the architecture to just two components—a metadata database and file storage—enabling multi-table transactions, fast small commits, data inlining, and zero-trust encryption. The format is an open standard using simple SQL, and the DuckDB extension implementation is MIT-licensed.

Key Insight: The most impactful design decisions in DuckDB were acts of deliberate simplification — single-node, no GPU, no SIMD, no JIT — which maximized portability and enabled analytics to run anywhere, from browsers to Raspberry Pis.

Hannes Mühleisen recounts the origin story of DuckDB, born from frustration with the state of analytical data management in 2015, when options were expensive enterprise systems, Hadoop, immature cloud services, or bolted-on columnar extensions. He and Mark Raasveldt decided to build 'the SQLite for analytics' at CWI Amsterdam, making key design decisions: single-node over distributed, vectorized execution over JIT compilation, CPU-only with no GPU support, no hardware-specific SIMD code, and MIT-style open source licensing. He demonstrates DuckDB's out-of-core hash join capabilities and argues that the embedded architecture liberates data management by allowing DuckDB to run anywhere from browsers to Raspberry Pis to outer space.

Key Insight: The future of data architecture is flipping the pyramid — empowering edge devices with capable local query engines while using commodity lakehouse formats for coordination, eliminating the need for expensive centralized data warehouses.

Hannes Mühleisen traces the evolution of data architecture from the 1985 three-tier model through cloud data warehouses to a new client-empowered paradigm. He argues that the traditional top-down architecture, where a central data warehouse holds all power, has barely changed despite moving to the cloud. Demonstrating that modern hardware can handle surprisingly large datasets on single machines, he makes the case that compute should move to the edge. He proposes flipping the architecture so clients do their own processing while lakehouse formats and commodity storage provide shared coordination.

Key Insight: DuckDB's power comes from radical simplicity and portability—avoiding hardware-specific optimizations and AI hype in favor of being a universal, embeddable database component that works everywhere.

Hannes Mühleisen discusses DuckDB's origins, philosophy, and growth on The Data Engineering Show. He explains how DuckDB emerged from CWI research as the first in-process analytical database, designed for universal deployment from phones to space satellites. He emphasizes DuckDB's focus on portability over hardware-specific optimization, explains why they built their own Parquet reader, and shares strong skepticism about AI's role in database internals while acknowledging its value for user-facing assistance like text-to-SQL.

Key Insight: The era of big data infrastructure is largely unnecessary for most workloads, as single-node hardware with efficient software like DuckDB can handle 99.9% of real-world analytical datasets.

Hannes Mühleisen opens DuckCon #6 in Amsterdam, reaffirming DuckDB's mission to make large datasets approachable rather than intimidating. He presents evidence that single-node computing power has outpaced the growth of useful data sizes, citing analysis showing P99.9 of data warehouse sizes is under 300GB. He shares adoption metrics including 32 million monthly extension installs and 1.8 million monthly website visitors. The talk transitions to Mark discussing upcoming v1.2 features and the lakehouse focus for the year ahead, followed by Q&A covering architecture, Wasm, partitioning, and extension development.

Key Insight: DuckDB's success came from observing that most data practitioners were struggling with unnecessarily complex tools, and that a simple in-process database could replace distributed systems for the vast majority of real-world analytical workloads.

In this Posit Data Science Hangout, Hannes Mühleisen discusses DuckDB's origins, architecture, and growing adoption. He explains how DuckDB emerged from frustrations teaching Spark courses and conversations with the R community about real data problems. He emphasizes that DuckDB's in-process architecture eliminates client-server complexity, runs anywhere from laptops to browsers, and handles mid-range data that previously required big data solutions. He addresses corporate adoption, noting DuckDB is widely embedded in BI tools and data platforms, and discusses the project's culture of friendliness and controlled governance.

Key Insight: The database industry wasted a decade solving problems only tech giants have, while the real opportunity was packaging proven database technology into something ordinary users could actually install and use.

Hannes Mühleisen discusses DuckDB's origins, philosophy, and explosive growth as an embedded analytical database. He explains why DuckDB prioritizes user experience, zero-dependency installation, and state-of-the-art query processing in a simple package. He critiques Lakehouse formats as technically inferior recreations of data warehouse features driven by market forces rather than technical merit. He argues the industry has been solving 'whale problems' while neglecting the 99% of users whose data fits on a single node, and advocates for transactional semantics in analytics without performance penalties.

Key Insight: ACID transactions, long considered unnecessary overhead for analytics, are essential for data confidence—and DuckDB proves they can be delivered without performance compromise in an embedded, zero-dependency package.

Hannes Mühleisen presents DuckDB's ACID transaction capabilities as a solution to common data reliability problems in the Python ecosystem. He argues that data practitioners live in fear of their data because tools like pandas lack transactional guarantees, leading to issues like accidental data loss, partial file writes, and race conditions. He walks through each ACID property (Atomicity, Consistency, Isolation, Durability) with practical analytics examples, showing how DuckDB's in-process architecture combined with full transactional support lets analysts change data with confidence. He demonstrates that DuckDB's analytics-optimized MVCC delivers these guarantees without sacrificing performance.

Key Insight: Since hardware growth is outpacing data size growth and most analytical queries scan under 300GB, single-node ACID-compliant databases can replace both distributed big data systems and Lakehouse format workarounds.

Hannes Mühleisen argues that the era of big data requiring distributed systems is effectively over, as research from Redshift and Snowflake shows median query scans use only 100MB and P99.9 scans fit under 300GB. This means single-node systems like DuckDB can handle virtually all analytical workloads. With smaller effective data sizes, we can reclaim ACID transaction properties that were sacrificed during the Hadoop era. He demonstrates how atomicity, consistency, isolation, and durability bring real value to analytics pipelines. He also expresses skepticism about Lakehouse formats like Iceberg, viewing them as limited workarounds for change management on object stores.

Key Insight: By refusing to distribute and betting everything on single-node performance, DuckDB exploited algorithmic advantages unavailable to distributed systems and captured the 99% of users who never needed Google-scale infrastructure.

Hannes Mühleisen discusses how DuckDB was built as a radical bet on single-node, in-process analytics over distributed scale-out architectures. Born from research at CWI Amsterdam, DuckDB combined state-of-the-art vectorized query execution with SQLite-like ease of use, after an earlier experiment (MonetDB Lite) proved the in-process model had demand. The 1.0 release focused on storage format stability and backward compatibility. Hannes explains how DuckDB's SQL dialect innovations like GROUP BY ALL have been widely copied, and discusses the emerging lakehouse ecosystem with mixed feelings about potential re-centralization of data access behind APIs.

Key Insight: The future of data processing is not centralized server architectures but embeddable engines that can run anywhere — in pipelines, application servers, browsers, and mobile devices — eliminating the friction and scaling problems of traditional data infrastructure.

Hannes Mühleisen presents how DuckDB, an in-process analytical database, is being used in unexpected ways beyond its original interactive analysis purpose. He explains DuckDB's core design philosophy of end-to-end user experience and 'data confidence' — removing the terror of encountering messy datasets. He highlights three surprising deployment patterns: DuckDB as a pipeline component (e.g., Fivetran, Lambda functions), DuckDB on application servers replacing separate analytics infrastructure, and DuckDB on clients via desktop apps, mobile, and WebAssembly in browsers. His central thesis is that DuckDB liberates data processing from centralized server architectures, enabling data engines to run anywhere.

Key Insight: DuckDB's in-process architecture represents a paradigm shift where state-of-the-art analytical query processing is unshackled from centralized servers and can be embedded at every tier of a data architecture, from operational databases to browsers.

Hannes Mühleisen presents how DuckDB can be deployed at every tier of a data architecture, replacing the traditional centralized database server model. He argues that DuckDB's in-process, embeddable design allows it to run inside operational servers (via PG-DuckDB), in serverless Lambdas replacing Spark clusters, within app servers eliminating database protocol overhead, and even in browsers via WebAssembly. He frames this as a paradigm shift: state-of-the-art query processing is no longer bound to a fixed server or cluster but can live anywhere creativity allows.

Key Insight: DuckDB's core mission is replacing the fear and hostility people experience with data tools by making data as approachable as opening a movie file in VLC.

Hannes Mühleisen opens DuckCon #5 by reflecting on DuckDB's mission and extraordinary growth. He frames DuckDB's core purpose as replacing fear of data with confidence, comparing the goal to how VLC made video files approachable. He outlines three main use cases: interactive analysis, pipeline components, and creative architectures like mobile devices and satellites. Adoption metrics are staggering — 6 million Python downloads per month, 17 million extension installs, and 600K unique web visitors. He emphasizes DuckDB is not VC-funded, sustained instead by commercial support contracts through DuckDB Labs.

Key Insight: By focusing on the 'tail scale' — solving data problems for the 99% who don't need distributed systems — DuckDB delivers a nimble, zero-configuration analytical engine that queries data where it lives without ingestion.

Hannes Mühleisen discusses DuckDB's design philosophy, inspired by SQLite's ease of use but built for analytical workloads. He explains why DuckDB was written in C++, how it can query data directly from CSV and Parquet files without ingestion, and its three main use cases: prototyping, local data analysis, and enterprise data pipelines. He describes the technical challenges of building a database from scratch over six years, including transactions for columnar bulk updates, graceful memory degradation, and storage format backwards compatibility. He also discusses DuckDB Labs' unusual self-funded business model and the future vision of DuckDB as an extensible platform.

Key Insight: DuckDB's Delta Lake integration via the shared delta kernel exemplifies how open ecosystem collaboration reduces duplicated effort and benefits everyone, especially small teams building in-process analytics tools.

Hannes Mühleisen announces official DuckDB support for Delta Lake and the Unity Catalog at the Data + AI Summit 2024. He positions DuckDB as filling the gap between Excel and Spark for last-mile data analysis. The Delta Lake integration leverages Databricks' delta kernel rather than building from scratch. He also announces DuckDB community extensions, allowing anyone to publish and share extensions easily.

Key Insight: The history of data management is a recurring cycle of abandoning and rediscovering the same fundamentals—tables, SQL, and ACID—and modern hardware has made distributed systems unnecessary for most workloads.

Hannes Mühleisen traces data management from 5,000-year-old Sumerian clay tablets to modern cloud databases, arguing that tables, SQL, and ACID transactions form an unbreakable 'holy trifecta.' He recounts how the NoSQL era saw companies abandon SQL for MapReduce due to hardware constraints, only to bring it back within years—MongoDB added schemas, Cassandra added ACID, and Databricks restored integrated storage. He predicts relational systems will absorb key-value stores, document databases, time series, vector databases, and graph databases, since all reduce to trivial table representations. Finally, he argues 'big data is dead' because modern single-node hardware is so powerful that distributing computation often makes things slower.

Key Insight: Building serious software in academia requires gambling your career on impact over papers, and sustaining it requires escaping institutional constraints through bootstrapping rather than surrendering control to venture capital.

Hannes Mühleisen traces DuckDB's origin from CWI, the Dutch National Research Lab where Python was also invented, explaining how academic environments can be both safe spaces and traps for software development. He describes DuckDB's core philosophy: an in-process, zero-dependency analytical database that prioritizes the full user experience over just the query engine internals. He draws parallels to Python's origin story at CWI, including similar institutional tensions when research software gains real-world traction. The talk concludes with hard-won lessons about funding open-source companies, arguing for bootstrapping over venture capital to maintain project integrity.

Key Insight: By starting from usability first principles rather than traditional database architecture, DuckDB arrived at an in-process, vectorized, morsel-parallel design that brings state-of-the-art academic database research to the 99% of data practitioners underserved by existing tools.

Hannes Mühleisen explains the technical foundations of DuckDB, an in-process analytical database built on vectorized query execution and morsel-driven parallelism. He describes how DuckDB originated from academic research at CWI Amsterdam, driven by the observation that no one had designed a data management system prioritizing usability and ease of data import/export. The conversation covers how vectorized execution processes data in 2,048-row column-oriented chunks to stay in CPU cache, and how morsel-driven parallelism improves on older exchange-based approaches by making operators parallelism-aware and reactive. Hannes argues that most data problems don't need distributed systems since hardware scales faster than human-generated data, and criticizes academia for solving 'whale problems' instead of helping the 99% of practitioners stuck on tools like Pandas.

Key Insight: DuckDB's path to 1.0 prioritizes stability and backwards-compatible file format permanence over new features, aiming to become the de facto standard for storing tabular data.

Hannes Mühleisen opens DuckCon #4 by sharing DuckDB's explosive adoption metrics, including 5 million monthly extension installs, half a million unique website visitors, and a rise to #45 on the DB-Engines ranking. He announces the upcoming 0.10 release and outlines the path to 1.0, which will follow a 'Snow Duck' philosophy inspired by Apple's Snow Leopard—focusing entirely on stability with no new features. The centerpiece of the 1.0 vision is backwards-compatible database file format, which Hannes positions as a potential new de facto standard for storing tabular data. Mark Raasveldt then details 0.10's features including multi-database attach for Postgres/MySQL, a 3x faster CSV reader, fixed-length arrays, unified memory manager, ALP floating point compression, and storage format improvements.

Key Insight: Most real-world data problems involve modest data sizes that are poorly served by distributed systems, and building a single-node in-process database with obsessive attention to developer experience fills an enormous gap.

Hannes Mühleisen discusses the origins of DuckDB, which started from observing that big data tools were overkill for most users' actual needs. He explains DuckDB's key architectural decisions: single-node execution and in-process design, which were controversial in 2018 but proved correct. He emphasizes that developer experience is paramount, describing how the team invested heavily in things like CSV parsing rather than just query optimization. He also discusses the partnership model with MotherDuck as separate companies, allowing DuckDB Labs to focus purely on building the best database without commercial pressures.

Key Insight: Single-node compute is growing faster than useful data creation, making distributed big data systems unnecessary for 99% of workloads, and DuckDB's in-process architecture proves databases can be neither slow nor frustrating.

Hannes Mühleisen presents DuckDB as the answer to the long-standing frustration that databases are slow and difficult to use. He demonstrates how traditional databases like Postgres require painful setup and CSV ingestion compared to data science tools, then shows how DuckDB eliminates this friction as an in-process analytical database. He argues that single-node computing is increasingly sufficient for virtually all datasets, backing this with a thought experiment about data creation rates versus hardware improvements, and demonstrates DuckDB outperforming a 1,024-core Spark cluster on a single machine.

Key Insight: The key innovation of DuckDB was recognizing that database frustration stems not from the engine itself but from the end-to-end user experience — and that in-process architecture eliminates most of that friction while preserving performance.

Hannes Mühleisen presents DuckDB's origin story and rationale at PyData Amsterdam 2023. He explains how a provocative quote from Hadley Wickham about databases being 'slower and more frustrating' motivated him and co-founder Mark Raasveldt to rethink the database user experience. DuckDB was designed as an in-process analytical database (like SQLite but for analytics) that eliminates installation friction, server management, and data transfer overhead. He demonstrates zero-copy querying of Pandas DataFrames and previews DuckSpark, a compatibility layer that runs PySpark scripts on DuckDB.

Key Insight: Most analytical data problems are far smaller than the infrastructure people use to solve them, and an in-process database on a laptop can replace cluster-scale systems for the majority of exploration and analysis work.

Hannes Mühleisen presents DuckDB at the Data + AI Summit 2023, explaining how he and Mark Raasveldt identified a fundamental mismatch in the data world: people were using massive distributed systems to solve relatively small data problems. He argues that the separation of storage and compute, combined with efficient formats like Parquet, means compute can scale down dramatically—often to a single laptop. DuckDB was built as an in-process analytical database with no dependencies, a 20MB footprint, and MIT licensing. He demonstrates querying a billion-row dataset in 160ms and 90GB of Parquet data in under 10 seconds on a MacBook, making the case that client-server architecture has been unquestioned since 1984 but is unnecessary for most analytical workloads.

Key Insight: DuckDB bridges the gap between easy-to-use DataFrame libraries and powerful analytical databases by running an academically rigorous query engine in-process with zero-copy Python integration.

Hannes Mühleisen presents DuckDB at EuroPython 2023, explaining how it combines the ease of use of DataFrame libraries with the analytical power of traditional database systems. He describes DuckDB's origins at CWI Amsterdam, inspired by SQLite's embedded approach but designed for analytics rather than transactions. The talk covers DuckDB's key technical advantages: columnar storage, vectorized execution, query optimization, parallelism, and beyond-memory execution. He demonstrates DuckDB's Python integration, showing it outperforming Pandas and PySpark on analytical queries while maintaining a tiny 16MB footprint.

Key Insight: DuckDB's architecture is evolving to handle datasets larger than memory through out-of-core operators and compressed materialization while maintaining its core philosophy of simplicity and pluggable extensibility.

Hannes Mühleisen presents the State of the Duck at DuckCon #3, covering DuckDB releases 0.7 and 0.8 from the preceding four months. Key features include multi-database attach support with pluggable storage backends, parallel JSON reading, hive-partition writes, pivot/unpivot operations, and as-of joins. He previews version 0.9 plans including out-of-core hash aggregation, compressed materialization, Apache Iceberg support, and better cloud platform integration. Adoption metrics show 1.3 million Python installs per month and 10K GitHub stars.

Key Insight: DuckDB succeeded by focusing on the end-to-end data experience rather than raw query speed, and its scale-up philosophy — initially dismissed by the industry — was validated when practitioners with big-data credentials confirmed that most real-world data isn't actually big.

Hannes Mühleisen describes how DuckDB was created at CWI in Amsterdam as an in-process analytical database, born from frustration that academic database research wasn't having real-world impact. He explains how talking to practitioners revealed that end-to-end experience mattered more than raw algorithm speed, and how restrictive IT environments shaped DuckDB's zero-dependency design. The conversation covers how Jordan Tigani discovered DuckDB and founded MotherDuck to commercialize it, with Hannes noting that Jordan's credibility from BigQuery validated the scale-up approach they'd been ridiculed for. Hannes reflects on the transition from academic researcher to running DuckDB Labs.

Key Insight: DuckDB was born from discovering that data practitioners refused to use databases due to client-server complexity, revealing an entirely missing category: the in-process analytical database.

Hannes Mühleisen recounts the origin story of DuckDB, which grew out of research at CWI in the Netherlands. After engaging with R community data practitioners, he discovered they avoided databases entirely despite working with large datasets, due to slow client protocols and the burden of managing database servers. A prototype called MonoDB Lite proved the concept of in-process analytical databases but hit architectural limits, leading Hannes and Mark Raasveldt to build DuckDB from scratch—a multi-year effort that created an entirely new class of database system.

Key Insight: DuckDB's strategy of squeezing maximum performance from a single machine through lightweight compression, out-of-core execution, and pervasive parallelism eliminates the complexity of distributed systems for the vast majority of analytical workloads.

Hannes Mühleisen and Mark Raasveldt present the State of the Duck at DuckCon #2, covering DuckDB's progress over the past year. Major achievements include persistent indexes, parallel data loading, advanced lightweight compression algorithms, out-of-core query execution for joins and sorting, and significant storage format improvements. DuckDB Labs has grown to 15 staff in Amsterdam, with adoption reaching one million Python installs per month. The talk outlines upcoming features including multi-database support, JSON reading, Hive partitioned writes, and plans for storage format finalization and V1.0.

Key Insight: The database industry's obsession with distributed systems has caused it to neglect single-node performance so badly that an in-process analytical engine on a laptop can outperform clusters costing 32 times as much.

Hannes Mühleisen explains DuckDB as the first in-process OLAP database, designed to bring analytical SQL processing directly into applications like Python without a client-server architecture. He describes how DuckDB's vectorized execution engine processes data in chunks of 1,024 values to optimize CPU cache usage and amortize interpretation overhead. He contrasts DuckDB's single-node efficiency against distributed systems like Spark, arguing that most datasets fit on a laptop and distributed coordination often dominates execution time. He also discusses DuckDB's WebAssembly compilation for browser-based analytics, the 'never give up, never surrender' out-of-memory project, and the bootstrapped business model of DuckDB Labs.

Key Insight: Most analytical workloads are massively over-provisioned with distributed infrastructure when a single-node in-process engine can deliver equivalent or superior performance with vastly less complexity and cost.

Hannes Mühleisen discusses DuckDB, an in-process SQL analytical database engine he co-created at CWI in Amsterdam. Born from conversations with statisticians who needed powerful analytics without server setup complexity, DuckDB fills the gap between SQLite (transactional) and heavy distributed systems like Spark. He explains its vectorized query processing architecture, zero-copy integration with Python/R/Arrow, and the philosophy that most 'big data' workloads can run on a single machine. The project is governed by a non-profit foundation with MIT licensing, while DuckDB Labs provides commercial support.

Key Insight: The data analytics ecosystem reinvents data management poorly; DuckDB applies the SQLite embedding model to analytical workloads, bringing decades of database engineering to where data scientists actually work.

Hannes Mühleisen introduces DuckDB at FOSDEM 2020, positioning it as an embeddable analytical database analogous to what SQLite is for transactional workloads. He argues that the current state of data analytics—with CSV files, crude query engines like Pandas, and hand-rolled binary formats—is a mess that database technology solved decades ago. DuckDB brings proper data management to analytics by running as an in-process library with zero external dependencies, a single-file storage format, and a vectorized execution engine that keeps data in CPU caches. He demonstrates simple integration with Python, R, and C++, and highlights rigorous quality assurance including millions of test queries and continuous benchmarking.