What MongoDB teaches us about database trends

Image: Various Photographs/Adobe Stock

This week is MongoDB World 2022, and for those of us who were there for MongoDB 2014 (like me), it’s fair to say that a lot has changed in the last eight years for the company and the industry.

Although I’m clearly biased – I worked for MongoDB in 2014, and after a few years with AWS and Adobe, I joined mid-2021 – it’s interesting to see how the database market has evolved in less of a decade. But it’s perhaps just as interesting how some things have stayed the same.

SEE: Recruitment Kit: Database Engineer (TechRepublic Premium)

Not Just Relational Data: A Key Shift in the Database Market

Return in June 2014the five most popular databases were exactly the same as June 2022: Oracle, MySQL, Microsoft SQL Server, PostgreSQL and MongoDB.

The difference is their relative popularity: PostgreSQL and MongoDB are gaining popularity over relational incumbents. PostgreSQL, for example, won at the expense of Oracle, and now accounts for about half of Oracle’s score (measure in terms of things like job postings and LinkedIn profile mentions), up from about 16% as popular as Oracle in 2014.

There are several reasons for this market development. I’ve written before about the rise of PostgreSQL. Here I will focus on MongoDB and what it means.

In his keynote, MongoDB CEO Dev Ittycheria shared statistics showing that MongoDB has become mainstream data infrastructure for over 35,000 customers, from Fortune 500 companies to garage startups.

Customers are one metric to measure adoption, but MongoDB’s footprint is even broader. Although downloads are no longer a primary metric for this primarily cloud-focused company (60% of the company’s revenue comes from Atlas, its cloud service), in 2014 the company had tens of thousands of downloads . Today, that number stands at 265 million, with more people downloading MongoDB’s community product in 2022 than in the first 11 years of MongoDB combined.

That’s a lot of adoption for a product that in 2014 still had web-wide roll your eyes. The video was funny, even though MongoDB never really struggled for scale. On the contrary, this video captured the feeling that relational databases could support most application requirements. As such, the challenge for MongoDB in 2014 was convincing developers to consider a world beyond relational data and tabular data structures.

MongoDB has always handled data relationships very well; it treated them differently than a relational database. Thus, at the time, the company accepted the NoSQL labeldespite its issues (who wants to be defined by what it’s not?), because it helped developers think beyond tabular data structures.

Since then, there has been an explosion of non-relational or multi-model databases. Today, DB-Engines includes nearly 400 databases, but less than half of them are relational databases. From document to time series to chart to column to key-value to [insert new database type here]the industry continued to use relational databases even as it found a home for a wide variety of new databases.

As RedMonk analyst Steve O’Grady wrote: “The era of a single category of general-purpose databases has given way to a period of specialization, with database types selected according to workload and need.”

In 2018, Amazon CTO Werner Vogels captured this move in a blog post:

“For decades, because the only choice of database was a relational database, whatever form or function the data had in the application, the data was modeled as relational,” Vogels said. “Instead of the use case driving the database requirements, it was the other way around. The database drove the data model for the application use case. A relational database is it specifically designed for denormalized schema and to enforce referential integrity in the database Absolutely, but the key point here is that not all application data models or use cases fit the relational model .

Vogels delivered this first MongoDB World in 2014. To capitalize on customer interest, he then helped AWS launch more than a dozen new, “purpose built“Database services over the next eight years.

The Rise and Rise of General Purpose Databases

More recently, we’ve seen the database market “revert to the mean,” so to speak, with developers returning to general-purpose databases like MongoDB and PostgreSQL. The reason, O’Grady explained, is simple – or rather, it’s a matter of simplicity: “Today’s overhead of having to learn and interact with multiple databases has become more of a burden than a bargain.”

He went on to say that companies have pushed database vendors to increase their capabilities because they don’t want to “context switch between different data stores, because they want to be able to do things like analytics on a given in-place dataset without having to migrate it, because they want to consolidate the sheer volume of vendors they deal with, or a combination of all of the above. »

SEE: Job Kit: Database Administrator (TechRepublic Premium)

In 2014, MongoDB helped spark an industry trend toward specialization; in 2022, this is part of a move away from specialization. The irony is that MongoDB never touted specialization and instead marketed itself as a general-purpose database from the start. Why? Because, as O’Grady explained, “general purpose” makes life easier for developers, and MongoDB has always focused on convenience for developers.

That’s why some of the biggest news in MongoDB World 2022 shouldn’t be news at all: the company is increasingly positioning MongoDB as a data platform for developers, not a database.

Databases become data platforms

Again, let’s look at this in the context of the industry: a series of companies are trying to provide one-stop-shops for data scientists, business analysts or other groups, and therefore market clouds of data and data platforms. What seems unique about MongoDB’s focus, however, is the company’s focus on developers.

Even though MongoDB can credibly claim that its document model has dramatically improved developer productivity, application demands constantly force developers to take on the uncomfortable responsibility of connecting a sprawling array of backend data systems, including search, real-time analysis and more. These services, in turn, require management such as logging and alerts. Guess who has to sew them all together? Developers. As MongoDB CTO Mark Porter joked, this leaves developers with “more glue than model”.

Indeed, throughout the keynotes, MongoDB executives repeatedly reiterated the company’s focus on developers. But now, instead of being on a flexible scheme or horizontal scale, the company has touted a sleek developer experience that spans an increasingly broad set of services to support a full data lifecycle, taking thus supporting a wider range of use cases, from transactional to operational to analytics. .

By largely abstracting the movement of data between services or products, MongoDB’s Developer Data Platform aims to help enterprises significantly reduce their middleware investments, in terms of people and software/systems, while reducing the need to reconcile data between systems and helping organizations ensure a single source of truth.

The idea of ​​a data platform, or data cloud, is not new and is not unique to MongoDB. As I said, this is an industry trend towards vertical integration to make life easier for developers (or data analysts, in the case of data warehouse vendors). But what’s different, and seems completely unique to MongoDB, is this idea of ​​a data platform for developers: something that makes developers much more productive with data.

Clearly a key part of this for MongoDB is analytics, but not those analytics. Even if I didn’t work in the company, it would be hard to leave MongoDB World thinking that MongoDB was planning to compete for data warehouse workloads.

Instead, the keynotes revealed many thoughts on analytics workloads that drive engaging in-app experiences. As? Well, like personalization apps that determine which promotions to display at checkout based on what was recently displayed. Or, like security apps that analyze network activity to separate good domains from bad ones.

Traditionally, analytical systems suited to these workloads were separate from operational systems. While separation sounds good, it really isn’t, as it introduces cost and complexity through things like ETL, distancing applications from the back-office data that feeds them data.

This batch-oriented world may be the status quo, but it creates poor application experiences for customers. MongoDB clearly agrees, and said so repeatedly at the event through a variety of announcements.

Which brings me back to how different our industry is today than it was in 2014, and how much the same it is. We still rely on relational databases and will for some time to come, as I wrote in 2016. And yet, it’s equally true that businesses increasingly depend on non-relational databases like MongoDB.

In both camps, we’ve seen developers flirt with special-purpose databases, while investing more in general-purpose databases and, more recently, general-purpose data platforms. As the Talking Heads could sing: “Same as it ever was”.

Disclosure: I work for MongoDB, but the opinions expressed here are my own, not those of my employer. Just ask: they often disagree with me.

Maria H. Underwood