The SQLite Takeover with Turso’s Glauber Costa

Welcome to Syntax. Today, we have a really good one. We're talking to Glauber Costa about Terso and SQLite. It's a distributed SQLite platform that is really just getting a lot of attention from a lot of developers who've tried it. So we're gonna be diving into distributed databases in general. We're gonna be talking about SQLite and why people are using it these days. And a little bit kinda kind of a little bit of a glow up recently in terms of, like, its public perception, it feels like. So we're gonna be talking all about that and more with me JS always is Wes Bos. What's up, Wes?

Hey. Not much. Excited to talk about SQLite and Terso and everything. The, the dudes from Drizzle had good things to say about you. Probably half because they like it and half because you, I think you sponsor the development of Drizzle quite a bit.

Yeah. Yeah.

Well, first of all, thanks so much for having me. It's a pleasure to be here. I've been a fan of the podcast. And, yeah, like, we all we were, in fact, the 1st corporate sponsors of Drizzle when right when they were just getting started. We're very happy with the with the partnership. We're very happy with them.

And I think it's safe to say they'll correct me if if they disagree here, but it's not just a financial thing. Like, we really believed in in what they're doing.

And, you know, very early on and I'm happy to go into into more details there as well. But, like, very early on, we had this issue in in which people were very excited about the idea of Trussell. That's when we were releasing our private beta.

But the one thing that we got from the private beta users is this is really hard to consume because it doesn't work with the ORMs. Right? Mhmm. And then we had this this Drizzle was just popping out in the scene at the moment, and we really had that symbiotic win win relationship in which, hey. If you guys can support us, we will be happy to sponsor you.

And, like, this, you know, this is how the relationship started, but we've been very impressed with what they're doing. I think they've been very impressive what we're doing. And Yeah. You know, we we keep sponsoring. We we wanna keep helping. We understand that they're not a company. They are a group of dudes from Ukraine Yeah. Which makes the whole thing even more impressive, right, Justin?

Totally. Yeah. We just we just talked about that. Cool. Well, let let's talk about Terso. What is Terso?

Terso is a managed version, managed offering, of a fork of SQLite called Leapsychl. So this is, you know, the the elevator pitch, and then we go into the details of that. But Tursu started with us.

We had this company that was creating a product that was not a database, but it was a product that used the database like many other products and and still in the data infrastructure space. And we used SQLite very heavily.

And just one day, we decided to fork SQLite.

And the reason we decided to fork SQLite is that we figure that SQLite is a great database. We love Sanity. And and, but but to use it as a as a production database for server workloads. I mean, SQLite has traditionally been used in in appliances, mobile devices, and your phone probably has like 10,000 SQLite databases if if not more.

My toaster, like, you run SQLite and your Apple Watch runs SQLite. I mean, it's everywhere.

We were seeing a lot of people trying to run this on the server, us included. We were using this in in our product.

And we weren't really happy with the solutions that existed at the time to run SQLite as a server side production database.

We, we felt like we had the technical capabilities to make the changes needed to SQLite to make that happen.

Becca and I worked in the Linux kernel for 10 years, each of us. Becca being my cofounder. We then wrote a database from scratch called Sanity DB later, and in a company that that we joined that is is a database focused on the NoSQL, petabyte scale workloads.

So we looked at the code and said, hey. Look. I mean, just we we Sanity really a bunch of changes that we can make sure this database that would allow us to run this on par if if not better than Postgres, MySQL, Mongo, and and any other offering really. But SQL like the project doesn't take contributions from from anybody.

So we decided to fork it. That's how, the open source project was built. And then we built the managed service, around it, registering, which is what Duracell is today. And if you want to see all of the errors in your application,

you'll want to check out Sentry at century.ioforward/ syntax.

You don't want a production application out there that, well, you have no visibility into in case something is blowing up, and you might not even know it. So head on to reduce entry.ioforward/ syntax. Again, we've been using this tool for a long time, and it totally rules. Alright.

So do you have a a lot of folks using Lib SQL as just Lib SQL without, you know, using Terso at all?

I think it depends on what you mean by how lots because SQLite is used by potentially trillions of of devices.

Mhmm. And and so we're nowhere near that Scott. But there are a couple of companies that we know of that use Libsico without Ursula. That and and I'm not gonna unfortunately, there are 2 companies in particular that I know that may have use of that. I don't think I am at liberty to disclose who they are, but but we know of some. But I also wanna clarify that we also don't put a lot of effort into making that happen.

Mhmm. So torso as a company, this all happened in October 2022. October 2022, we launched this fork of sequelite. And then by February 2023, we had Turso demand Sanity service Scott because we Yarn, unlike the Drizzle folks, we are a VC backed company. So if fairly early on, we we had to give this answer to ourselves and our investors and our employees and everything. Like, how do you make money out of this? And it's not by being a just a open source SQLite clone or fork or whatever they run on device. You have to have something.

And so all of our efforts, again, it's open source, so people use it. And we know Node people using it as just a replacement for SQLite. We add a couple of things to that fork that have nothing to do with the TURso platform.

One of them being vector search. So you have native vector search in the Libs SQL build. Oh, cool. Yep. And, you know, I know of some people using it. But Wow.

I wanna explain real quick for people listening. So vector search is the ability to query vector vector, what do you call them, vector objects that are are close to each other, which is pretty cool. I know that it's probably a lot more than that, but the idea is with with AI right now, a lot of people are creating what's called embeddings.

And embeddings are, sort of a numerical representation of the data that's in it. Right? Mhmm. And if you want to, I could take, like, a whole bunch of photos or a whole bunch of text and convert them to embeddings. And if you wanna find similar text, not just based on the words inside of it, but The meaning. The semantic meaning of what those things are. Yeah. Semantic meaning. Exactly. Or, like, pictures of of what is in the photo, etcetera.

That is really cool. And very early on, we saw a lot of just, like, custom vector databases.

And I think you could tell me. Are they getting their lunch eaten by just, like, regular databases, like SQLite adding it as a feature?

I think the answer is unequivocally Wes. And and, yeah, I'm I'm I'm hoping, you know, obviously, people will disagree. And if you talk to, somebody from a vector database, I will expect them to disagree. That's the they they they're financially motivated to disagree, which which is fine. Yeah.

But but I don't think vector is look. I I worked as an in in a Node database for 8 years before that, which is a columnar store. I'm fairly familiar with with NoSQL Node SQL databases by a large group together as NoSQL.

And some of the things that those databases do can be quite complex. Think for example, in time series.

To have a good time series database, you have to have a completely different on disk representation.

Things are not represented as relationships. They're represented as series, and then you you you have to be able to very quickly do aggregations across those series, which is a completely different set of queries than a relational database would do. And those things are very, very, very complicated. You have companies, well, Datadog being one example. At the end of the day, they are just a giant time series databases.

You have things like InfluxDB.

There are very complicated technologies that allow you to query time series.

Vector is not like that. Vector is relatively simple.

And and I think now that everybody is looking into that because of the AI boom, turns out JS not that complicated. So if it's not that complicated, what happens is that all of the databases just adopted like the same way they they adopted JSON as a as a format. And this happening with Postgres. Postgres is doing a lot of vector. This is happening with us. So we allow, for example, without any extensions. You don't need any extensions. You just build Libs SQL.

And our build of Libs SQL has vector, which is now available in the Thurso platform, but it came first through the open source.

And people have been using, one of the cases that I am aware of for people using Libs SQL, nothing to do with Tersu is exactly for vectors. So you have this vector thing that is just there. Right?

Interesting. You had mentioned that, you know, your prior experience was in a NoSQL, a NoSQL database, and now you're working on SQLite.

Why the shift to relational databases? I

I'm not gonna say that I don't believe in NoSQL anymore. It's just that I don't believe in NoSQL anymore as your default choice.

Mhmm. And so so what I what I found in the market is that around 2010, you just had like, if if you had anything remotely serious, you would go for NoSQL. And the reason for that is that the SQL databases didn't scale very Wes, and you didn't really have a lot of hardware. Your phone today probably has more compute and storage capacity than a server had in in in 2010.

So you had to scale out. People sometimes make fun of that and and say, no. You didn't really. But reality is that you had if you had any website in 2010 that had any mild capacity, you had to scale it out. And Mongo was at the time the best choice and and to do that. Like the SQL databases, you would have to manually chart chart it, which is not a pleasant experience.

So you had, you you you kinda had to do it.

But fast forward to today, the harbor, advanced a lot. So your phone has the same capacity that a server had back then, but a Vercel, you can go on AWS today and and rent a behemoth of of machine Mhmm. On with a click.

And and that's fine. So you don't have to necessarily scale out anymore.

Also the SQL databases got better. SQLite itself got a lot better. SQLite, the software in SQLite today is not the same as it Wes, 5 years ago. It has the the infamous journal mode before, which Wes, forced you to have a single writer.

Today, Sanity does not back then, SQLite, every write that you do in the database took a full lock. You could do nothing else when you're writing.

Today you still have the single writer model, but you can still read why while you you are writing, which means, okay, that's that's good enough for the for the vast majority of of read mostly workloads. So the software work better, improved.

Postgres improved as well. MySQL improved as Wes, and the hardware improved. So you don't have to do NoSQL anymore. And I think what happens that NoSQL ceased to be the default, and became this specialized thing that some use cases for some things will need.

And I was just more interested in in, you know, in going forward the core, and and that's why I landed here.

Yeah. Cool.

I wanna talk about the the whole multi tenancy or per tenant database because this is something I've been hearing a lot about recently where people say one of the benefits of SQLite is instead of having 1 big database, you can literally give every single user their own database, which Mhmm. Seems cool, but then also I'm like, well, then that seems like a bit of a maintenance headache and whatnot. You wanna talk about

that? Awesome. So so look. This this is great because when we, going back to our history and and and how we forked the this database, we came up with the managed platform almost immediately after, which means it wasn't as good as it is today. It was essentially like, okay. Now we have this this this thing that's doing well. Our fork of SQLite got 1500 GitHub stars in a week, which which is we decide, okay. So we're gonna go all into this. And we had the platform right after it. And and one of the advantages that we knew we were going to have is that SQLite can be a lot more affordable than any other SQL SQL database. So this this is still something that underpins what we do.

You can have a lot more for your dollar with Sanity. And this is true for every SQLite offering than you would have for other databases.

So we offered we offered people at the time 3 databases for free so you could come, and and you could just sign up on on, get an account, and you would get 3 databases.

But our users who Yarn at the time on the private beta kept asking us, why 3? Like, why not 10,000? Just the why why like, I have this pattern that so the users themselves as they came, but kept ESLint us that this pattern was a pattern that they saw done on SQLite.

Not necessarily those users were adopting this pattern, but they were saying, I like the fact that this exist. I would love to experiment with that pattern.

So why can't you if it's just a Sanity file, why why can't you give me 10,000 databases? And the technical answer at the time was that we'd because the way we architectured the service ESLint, again, in 2 months so we could release it.

Every database was in a VM. So we didn't have any multi tenant capabilities.

And what we've done later JS okay. Now we build the the level of isolation. CColite itself is really good with that. So c cola, you can you can determine how much memory maximum you're going to use.

By the way, our vector search implementation, we don't use the fastest algorithms. We use the most memory predictable algorithms exact exactly to be able to keep offering this multitenancy thing with vector.

And and and SQLite already isolates fairly Wes. Then we change your architecture so that now we have a big VM with a lot of files on it.

Now and now we can offer you like those 10,000 databases, which is what we offer for $29 a month. So for $29 a month, you have access to 10,000 databases.

For free, you have access to 500 databases, which you can use for site projects if you have 500 site projects. Some people do, in fact. You could you know just Wes and I. Yeah. And and that's fine. But you can also use for multi tenancy.

But once we release multi tenancy and look, this is what you should do as a startup. You you're always listening and and iterating. So once we released multi tenancy, now we can have 10,000 databases.

People kept telling us, okay, that's great. But if I have to change the scheme of those databases, I'm kinda screwed because that sounds like a tremendous maintenance burger.

And then we add the capabilities to our server to do, transactional schema changes in what we have today we have a concept in terms of schema databases and child databases. So when you create a database, you can if you'd say nothing, it's just a database.

But you can create a schema database and then you can create child databases that are connected to the schema databases.

When you change the schema of the schema database, it propagates atomically to the child databases.

Right. So you can essentially do that schema change in a single ESLint, and it it gets propagated to the other databases. So part of what we do is is making that model, making the model feasible, which starts with here are the 10,000 databases for $29 to begin with. Yeah. So so it's and building the machinery on top to make the model work and and and make it seeing where the pinpoints are, and and building infrastructure to to go around it. Right? Okay. And why would why would you want 10,000 databases?

Like, why would it make sense to put every single user in its own database?

The first thing, I mean, I I wanna clarify is that this is not what Thurso is. This is one thing that Thurso allows. And and, I I'm I'm I I really wanna stress that because some people understand that, like, if I don't do multitenancy, then I Tursus is not for me, and it's not true.

If you can't like, it's not because you can create that a 100,000 databases, doesn't mean you have to. So if you prefer to put everything on a single database, do it by all means.

We don't we don't mind. I mean, do do you have all of those databases. I'm not gonna take offense if you use only 1. That's fine.

But but but but you can. But you can. And the what I see is that some architectures are made simpler by doing multitenancy.

And those are the architectures that I believe should do multitenancy. I don't believe, or I I I as I've been calling recently, hyper tenancy because you don't even you don't need it you don't even need those databases to be long lived. You can create a database that lasts for an hour to host the session.

And then when that is when when that is done, you shut it down. So you you really can can go take multi tenancy because those databases also come online in 500 milliseconds. Right? So in 500 milliseconds, your database is there. Wes for an hour, shut it down. That's fine.

But some architectures are made simpler by this. Imagine, for example, that you are a company in the financial space.

You can have multiple you can have a database per client of yours.

You can encrypt that database of different keys.

And lots of things in your back end become a lot simpler because the isolation is now guaranteed at the database level.

If you if if you are, GDPR compliant and a user requests a copy of their data, it's trivial now to extract the copy of the database.

You can be more aggressive with your back end optimizations because you know that nothing that you do will cause by accident you to mix data from different customers.

Right? So a lot of architectures and and what we see a lot is people in the health care and finance space being heavily interested by this because those are heavy compliance based, industries in in which data segregation is very important.

And then they say, look. I mean, if I can guarantee this at the database level already, a lot of my back end become a lot simpler.

Yeah. So offhand, you had mentioned that, SQLite being cheaper was, you know, a big factor in some of the direction you went.

What what exactly makes SQLite, like, a cheaper to run than a traditional MySQL or Postgres?

First of all, the 20 years of optimizations in making it runnable in toasters. I mean, the toaster thing is mostly a joke, but at the same time, it isn't because yeah. I bet I bet some toaster out there runs Sanity. I mean, just, we've we're now we're now on the AI craze. You might remember a couple of years ago, you had the IoT craze in which you had to have smart everything. And I'm sure at that point, somebody, came up with a toaster that was a smart toaster and runs.

I am pretty much, on the, camp of I don't have smart stuff in my Node, and and the very few things there JS smart devices on my home. I keep a gun around. If they do if they make a noise that I don't recognize, I'll shoot it.

Don't don't trust that don't trust that too much. But look, it runs everywhere Wes it's it has, like, 20 years of optimizations to make it runnable in in in those environments, which are low memory. So at the end of the day, a database costs as much as the resources needed to fulfill the queries. Mhmm.

So if the if you use fewer resources to, to fulfill that query, it costs you less money.

So there is there is a general sense in in in which Sanity is just cheaper. Because again, it's simpler. It it will do fewer things, but those things they will do, it will do at the same speed or faster, but using a lot fewer resources. So that's number 1.

And number 2, because it allows for for those architectures being just a file.

At the end of the day, SQLite is just a file.

Allow for those architectures where where, for example, if I had to give every user a Postgres database, it will cost me, like, being charitable a dollar a month just to keep the database online in in a container running.

Even serverless Postgres that you see today, what they do is that they bring compute up.

They that compute stay up for 5 minutes to handle 1 request and then it shuts down.

SQLite being just the file, the file is always there. If you send an HTTP request that that gets served, there's no buffering. There there there's no extra anything. You send this HTTP Wes. You serve this one request into that file and that's it. You didn't have to bring compute. You don't need Kubernetes. You don't need any of that. So you end up having a cheaper infrastructure to run. You you end up having a a a query that uses fewer resources to be fulfilled, and that translates into essentially a cheaper offering.

It's not all about being cheap as well. We also have a couple of features that I think are very unique to and are are enabled by the SQLite model. One of them, which is my favorite, in fact, is is what we call embedded replicas. So embedded replicas allow you have your database accessible over HTTP, but I can now create a replica of that database in a file that sits in your own server.

And that file is kept up to date by the server. So now you can read from a file locally in 200 microseconds, instead of going into a server, that is 10 milliseconds away.

So now you you are faster, right, as well as as JS Wes as cheaper. So I think that the the the fire Bos model of SQLite and and how resource efficient and memory bounded it is to serve your queries allow us to build a service that has all of those properties.

But if you if you need none of those properties, you don't need the this level of replication and microsecond level latency. You don't need multitenancy.

You end up with a service that allows you to do a 1,000,000 reads per month for for free.

And because it doesn't cost me too much to to to serve those reads. Right?

Wow.

So when you have something that is, like, seemingly so, like, cheap to run, is it the strategy for building a business off of that, is that all ESLint, like, what kind of novel things can we do that other folks can't do? Is that the approach to to take something that is, like, you know,

trivial to run? Yeah? Yeah. Yes. Because at the end of the day, Wes we're gonna make a little bit, we're only gonna make a little bit of money because we charge you only a little bit. It's not something that gets a lot of people in the venture capital industry excited.

So so you have to you have to build to to build a really big business, you need to transform that into features that are themselves very valuable.

And they're only possible, because you drop the baseline cost. Alright? So so there are many examples in the industry across time in in which, something that was very expensive now becomes very cheap and then allows you to build things in a different way and allows you to build things in ways that you wanted to do before but wasn't possible. One example that comes to mind is Git.

I mean, I am a a little bit of a dinosaur, as as I mentioned, in the beginning. I was in the Linux kernel as one of the core contributors as my cofounder was from, essentially, 2000 to 2,000, 2002 to 2012.

And in in the middle of that period, there was this thing called Git, that was written by Linus.

Mhmm. And and the whole story of how it came to be was actually quite interesting. But, one of the thing if if if you're not familiar with the landscape at the time, the dominant version control system in the open source was subversion and c v s. They were like people some people use c v s, some people use subversion.

And creating branches and merging branches in those tools were really complicated and humanly expensive procedures. Like, you had to stop what you're doing.

You you had to go and and sync with your team. Oh, I'm gonna merge something now. Like, everybody right? Everybody beware.

I need your branch. So you need you need to think about how I'm gonna branch this. And one of the things that Linus said a lot about git at the time JS, like, hey, look. Just creating a branch now takes 200 milliseconds and you just do this. And that's just a hash, and that's a branch. And and, like, once this becomes trivially cheaper, you can have a completely different workflow that would have been impossible with this other tool. And I think of databases in the same way. What we wanna do is, you can the the same way you can still use git the in the exact same way you would have used subversion or CSS back then, new workflows were enabled.

And in a in a sense, we wanna do the same with Terso. You can still use Terso as a SQL database as you used before. That's fine. And the baseline works.

But because the cost of a database essentially drops to 0, the cost of creating a new database and and maintaining a new database drops to 0. There are new things that you can do that you just couldn't do before and that allows you to have, new architectures those new architectures based on hypertenancy in in again in which Bos on on on, like, very capillary replication in inside your mobile device, inside your own server, that Yeah. That allows for architectures that just were not possible before.

Right? So one thing we really like about, we use on the syntax website, we MySQL, and we use PlanetScale to host it, which they just had a bit of a rug pull or the other, 6 months ago or so. Like, they took away the free plan. They jacked up the price on everything. So I'm here to tell you to please don't do that to us as well on.

But Wes despite that, I still really do like Planet Scale because of their, like, branching feature where it's like Yeah. Oh, yeah. I wanna work on a I wanna work on a feature. I'll just, like, take a fork of the database so that I can work with the most up to date data and then sort of submit, like, a pull request for the the schema updates as well. It's is there anything like that? Like, what's the what's the flow for that when you're working with SQLite?

The the pull the the pull request thing, we don't, which is something that we wanna get you in the future. But the branching, we do. And, this is one of the things that look. Our company is 14 people.

And this is probably one of the smallest heavy hitter database players that you've ever seen. Like any other database company out there is is at least 5 times our size.

And again, SQLite SQLite is part of the reason why we can do this because do you know how our branching feature works? We copy the file.

Yeah. Yeah. Yeah. So That rule is Yeah. That's a so simple. Yeah.

There is a lot that you can do when you have a fundamentally and and intrinsically simpler architecture that allows you to do like, any other database, you have pages and then and then you have LSM trees and and b trees. There are very directly.

And if you wanna branch based on that, you have to be aware of this whole structure.

And we wanted to implement Scott.

Like, you can just copy the file. And does that take long and that that uses up a lot of space? No. Because the XFS file system has copy and write capabilities it's itself. So you can you just back this up by Wes, and then you copy the file. And and that's it.

It takes the same right? It JS just the file. So this architecture is very beautiful. Mhmm. But but you you can at any time you can at any time in Turso, create a new database based on another database, which is a branch.

But the reason we even say, like, create a database based on another database is that you can create that based on any point in time that these are the database well. So you can say give me give me a copy of this database as it was, like yesterday. Right? Just then and then you'd at 3 PM Pacific, and and there you go. You've got a copy of the database at that point in time. Right, Justin?

JS that work for, like, rollbacks as well? Because I noticed, I was in Cloudflare, and they they literally allow you to scroll scrub back in time to, like, literally any point in time, which is wild to me, and it's all instant.

We can we can do that too. However, we have a product decision on top of that that I haven't I I still believe it it's it was the right choice for us. In fact, I would love to know your opinion, since you're as as users. But we don't roll back to the same URL because the the the the problem there is that, like, some people just shoot themselves in the foot. So you you we give you a new URL with that database in that point in time, and then you can go verify if this is really what you wanted, and and then you you go you switch that URL. So you can do that, but yeah.

It's kind of the fix forward instead of roll back idea. A lot of people in Git are like that as well as don't roll it back. Fix forward. Yeah. So that JS the only difference. Like, I can give you a I can give you a snapshot of your database at any point in time in the last day if you're a free user, in the last 10 days if you are a $9 a month user, in the last month if you are a $29 a month, user, and more if you are on pro and enterprise.

But I give you a new URL with that with that database. And now you can go verify and see if if things are the way you want and and switch to that URL if if if yet if if not, you just discard it. Right?

Yeah. And I'm I'm just thinking, like, how awesome it would be to how much more you can store given the prices of this right now. You know? Because, like, I think right now Wes have we our database is 1.7 gigs, which is it's relatively small.

Mhmm. And, like, that that is a pretty small database in terms of the pricing for.

Yes.

So you could we could store a lot more stuff. Like, sometimes you think, we're not we're not gonna store absolutely everything because that's that takes up too much space. You know? But Mhmm. As the cost of storing things goes down, you think, oh, what what more can we store?

Go go goes back to goes back to that conversation we're having about architecture. The thing with multitenancy is exactly the same thing.

We're not just after the people who are doing multitenancy.

Let me give you a couple of examples on how driving the cost of a database to Deno allows you to rethink architecture. You just Vercel came up with 1. I would store things that I would not otherwise store. I'll give you 2 more. The first one is just a hypertenancy story.

You know what? I can get a database per session of my user in in the web. I can get a database per user of, every every episode of my podcast gets its own database. That's fine. If it simplify things in architecture, you can do this. And the third Node is every time you were thinking, you know what? I need to be very careful how I I I construct my tables and how I query my tables because I don't wanna have an n plus one problem in which I'm querying the database all the time. This is gonna end up taking like many seconds and my API is now busted.

Look. If you replicate that into a server that and you're Node reading from a local file, query away. Like, query away. Your your your queries are now free. First of all, because we don't charge you for that queries. We charge you for the sync. So we charge you to keep the data in sync, but you you now can do as many queries as your heart desire completely for free, both in terms of cost and and pretty much time JS as Wes. Because, you know, fire 10,000 queries and each query takes a 100 microseconds, that's fine. You know? Yeah. Yeah.

I I've been I've been doing a lot with local data on device, and it is. It changes the way you think about saving data.

And I I think that's such a powerful way to be. It empowers you to to to do more with the data. You don't have to worry about the cost associated.

Does this also in like, inform the the edge functionality as well? Is that JS the edge functionality of Churso, is that related at all to the the way that we're talking about, SQLite here as being a file?

Yes.

I just wanna clarify that I regret using that word, that we used very heavily in in the beginning.

And in in our defense, I mean, Pnpm and I come from a very different background.

And edge for us was just the Scott that we wanted to tell by positioning ourselves as an edge database is the story in a sense that we are achieving today with embedded replication.

Push your data as close as possible to you, make those latencies very small to nonexistent, and that can change the way you think. However, what happened is that in especially in the JavaScript Sanity, in particular, but not only, but but but in a special way, That word got a very unfortunate meaning because what people people came to understand edge as being something that is not compatible with Node. Js, which was mind boggling too. Because it edge edge in my mind is a concept that is geography. Responsible for that slightly.

Okay. And and and we all this happens, you know, just people once you put a term out there, like, people understand the way they understand. Like, I'm I'm not I'm not assigning blame to any specific individual or entity, or anything. But the fact of the matter, is that we had many many potential customers before when we talked. They would say, I love the idea what you're doing, but I'm not using the edge at all. I like Node. JS, so I'm not gonna use it. So, so it was a word that is was very confusing.

But what we wanted to do with our initial edge positioning is this idea that, look, you can have a database that is incredibly fast with 0 to low either 40 0 or very low latency by replicating Sanity massively.

And the reason we knew we could replicate SQLite massively is exactly because this file based, low cost nature.

Because if you're replicating any other database, you now have to keep servers running, to serve those requests from those other locations, which may or may not have a lot of requests. So imagine, for example, you you have a couple of users in the United States and some users in Australia. Those users in Australia, they show up, for 8 hours a day.

Sometimes because maybe your business sells to Australia, but not that much. You're not gonna put a replica there because it's gonna be very expensive to keep that running all the time and up to date and sync, etcetera.

But if I keep the if I can keep the file in sync, so there's some bandwidth cost in in there, which is fine. It it's not the most expensive part. But if you're not querying that replica, you're not paying anything, then that's fine. Right? Just it it allows you to have this architecture. And that's what we wanted to do with the edge. I think we we realized this vision even better with far embedded replication. And the way we think about this today JS, like, look. If you if you are using a serverless platform, then use use our edge replicas.

Because you now have an HTTP accessible thing that you can access, with the same latency from Australia, United States, Brazil, Canada, Europe, whatever you are. If you have a server, a VPS, an AWS server, anything like that, you're actually better off don't use our edge replicas. I can offer you Node better. Just replicate that inside your own server.

Put in put in your user's mobile phone so you have 10,000 users. Keep a database per user and replicate the database individually to each of their phones and encrypted with different keys.

So this is even better than just having 5 or 6 locations that we offer that you can query from, with low latency, which is still how you use it on serverless. Because on serverless, you don't have a file system where to replicate that file. But if you do, that's what we suggest. Oh, man.

Okay. I got I got 2 questions after that. So, like, the first one JS, if I have a database in New Jersey and I have, an edge function which could run anywhere over the world, my serverless function or or edge function or whatever is gonna connect to the closest version of that database. Right? So if the serverless function is running in in Australia, it's gonna try to find the Australia one. And if I update any of those any of those databases, they all sync with each other. Is that right? Yeah.

Yeah. Well, the the the the Vercel Node is, we we wanna keep things very simple for a couple of reasons. One of them is so that we can keep fighting the big guys with a 14 people team.

And but also but also because we believe in simplicity.

We truly believe in simplicity as a goal. We it it's easier to cognitively understand what's going on. So we have a model in which there JS a single primary.

So all your rights are gonna go to your main region, which is likely like New Jersey in your case. But your reads can be served from any other database.

And then we have something special on top, which I think makes it usable, which is read your write semantics. So in the same client, if you write something, you Yarn guaranteed to read that right. Because, what a lot of databases that go for massive replication they do, like our previous NoSQL databases, that you don't really have any guarantees. You you read, you write, and you read, and you may read what you wrote or Scott. And eventual consistency in the future, those things will coalesce.

But with Thurso, in in the same client that you created, if you write, you're gonna read that data back. But a new client that is that is just getting started now, that client can read data in the past because the the the data hasn't reached that replica yet. And the reason that works well is that this is the same consistency model that a transaction offers you. If you start a transaction now, other transactions will come in in the middle and update the database, but you will still read the data as if you were when when you started the transaction. Okay. So so you have that. But your rights are always gonna be slower in Australia than they are in New Jersey, assuming, of course, your main database is in New Jersey. And the other way around if if

yeah. That makes sense. And because a lot of people always say, like, edge functions are not a great idea because your your database is always gonna be your your bottleneck. Right? If you have database.

Yeah. There still changes that. There there still changes that radically. You can just Yeah.

And the other question I had was you just said, well, if you have a server or if you have, like, a, like, somebody's phone or whatever, instead of connecting to the database, just put the put the the file on the server themselves. Mhmm.

And that Tershow does that. Like, I'm able to just like, if I have a a 1 gig database and I'm running a DigitalOcean droplet, I could just download this SQLite database, run it on my droplet, and it will sync with the rest?

That's right.

Wow. And in that same regard, you had mentioned embedded replicas

that Yeah. That's the name of this feature. Yeah. Yeah. Embedded replica.

So you had mentioned it could be stored on on device and that will sync like we just said.

Will it will this, like, conflict resolution happen if the user's offline comes back online? Will it sync correctly? I wanna clear I wanna clarify something.

At the moment, we're actually planning to change that. Then I'm happy to tell you the future road map in here, which will answer your question fully. But at the moment, they are called embedded replicas.

So it it is you you don't write to them. You you you write to the main database. So you still need, what you can do with this model is that you can do offline reads.

You you can you you have the copy of the database in your device.

Then let's say you have something, that's a use case that that we have here like a farmer that has a tablet that has like a schedule, and they have to go in the fields. There's no Internet connection in the fields.

So you can still see all the data in your database there.

But but if you make changes like those changes, you have to be off online to make the changes.

But you can still see the data you can still see the data on ESLint.

So which is why, like, we wanna change that and we we in fact have it on our Node map by October this year to change it to allow rights in the replica as well. And now you need some form of conflict resolution. Right? But the the the one of the reasons we started with just read replicas is that it it is the simple thing to do. You just get a copy, and now you can read those things from your, DigitalOcean droplet in microseconds.

Well yeah. Give me that, that,

offline sync, and that's my, dream service right there. So, yeah, sick. Can you do it from, like we've been doing a whole bunch of shows on, like, offline first as well. And, like, what does the story look like if this if your server's offline, if your digital ocean is offline, can you put the database in the client or or or put it in Wasm or something like that? No. If your if your digital ocean,

droplet is offline, it's the same story. You you can read from it. You will not be able to write.

Because I wanna I wanna clarify something as well. The all of this is transparent in. So you the the way it looks in code is they create a ESLint, you define a period for the sync. Let's say 5 seconds, 1 second, 1 minute. And then what happens is that if you sync more often, that's more expensive.

Right? Because you're transferring frames more often and you have less coalescing opportunities.

And if you're syncing less often, this is cheaper, but the data is less fresh. That's a trade off that you make.

And on the client, you you call, let's use Drizzle as an example, like d b dot execute and then you put a insert query. That works.

The only thing is that their write will now go through the network. So you don't have to do anything special in the client.

The write is just slow.

But then when you do d b dot execute Vercel, that comes back in a 100 microseconds. So that's the only like, it's all transparent, but but you one of them is slower and and they want the the rights are faster. The the rights are slower and the reason faster.

When you go into conflict resolution, when you go into, like, okay. Now I need to allow rights to happen in the in in the local endpoint, then the only way to do this is to make things less transparent. Because, SQL has the the asset model, has the transaction transaction model. It's not like a NoSQL, eventual consistent database that you always write like idempotent mutations.

That's not how SQL works.

So you need to account for the fact that somebody else may have written something before you did that invalidates the transaction that you had.

And our approach to that will be essentially like you as the user defines what happens in this case. So if we detect the conflict, we call a callback that you defined that in which you decide what to do. Like, so do you ignore this data or you try do you try to write it again? Making sure that the values are still make sense. Like, whatever you decide to do, you do it. But it has to be the trade off there is that it has to be less transparent. You have to come and decide what to do if if your if your rights fail. Right? Yeah. I I I think that's alright. You Node, I'm I'm doing a lot of this stuff with Replicash right now, and it they're you Scott handle everything. You know? It's just sending you patch messages and all that stuff. And so it's not like

a a huge problem to understand what's going on. It's funny when we were looking through the Replicash docs, and it's like, there's a whole massive, like, 4 pages on, like, conflict merging strategies Yeah. Whether you're versioning tables, whether you're just overwriting, whatever. So it it's all very, situational dependent and and complex.

Is there anything about Terso specifically that we didn't get to before? Maybe we take some some questions that are are not Terso related.

Look. I I think we cover, for me, if if you ask me, like, what what is the the thing that really shines about Truso and what is the thing that people should come to Truso for? There are 3 things.

First of them, just how incredibly easy it is. In in fact, that is the one thing the the one thing that we did not talk too much about.

Yeah. The way things work with Trussell is that you create a client, and let's keep I understand most of your audience will be a JavaScript audience by and large. Yeah. So you create a client using Drizzle or using Prisma or just using the raw client, Libsico ESLint, TypeScript.

And in that client, you're gonna pass a bunch of options. One of the options is the URL.

And then you have a token, you have a sync period, you have sync URL. You have, you know, for for you you have extra things, but the main thing is the URL.

And if you pass a file, that is just local SQLite. I mean, it technically speaking, that's SQL, the fork. So you can do vector search. You can do all of those things.

But when you pass a file, that's just that it's just this. That's a file that and one of the things that people love about Turso, and if you go in our community, our Discord community JS discord.gg/turso.

That is how a lot of people develop. You can develop your application in a file. You do not have to connect to the service to develop.

You don't have to spin a container to develop. You don't have to, like, keep bringing databases up and down on CI to develop. You can just put the flat file there.

So the developer experience of developing with SQLite is absurdly good.

And we keep this.

And when you wanna connect to the Thurso service, then you can replace that file URL with the Thurso URL.

Or you don't or you keep the file URL and add a sync URL and now you're reading from the local file. Right? So one of the things that is very good, about what we offer is just like, hey. Start with a file.

It it's it's all you Node. Like, it's the simplest getting started thing possible.

But then there is a very smooth on ramp to use this in production, which is why like our messaging today is not even though we do replication very well, even though we do, hypertenancy very well, the messaging is c colite for production. JS you you start with with this c colite idea. We start with with the file, and then you get your on ramp to run this thing in production with with with replication, with, with backups, with point in time recovery, with schema changes, automated, in in a single point. So what we're doing essentially is adding all of those features and functionality that allows you to get the good stuff about SQLite.

The stuff that people love like, which is again, the file based DX, and local reads and multi tenancy, hyper tenancy JS we've been calling it, and put JS in production. So

Beautiful.

Yeah. That's awesome.

I I I had a quick question. You know, you had mentioned that you had done a lot of work on the Linux kernel.

I'm just curious about that. What was what was that like?

So first of all, that was a long time ago. I I started as I I suspect I Node.

Node one of the things you wanna know, is, correct me if I'm wrong, is people really that harsh and and are people really that harsh in The Little Sparrow? So let let me start with a story. Let me start with a story. That was around 2003.

I was a college student, and I was very interested in open source, and I was very interested in low level systems. And Linux was just the perfect intersection of this.

And at the time, by and large, people do not believe in open source. Right? Open source was the thing that you if you do this, your kids are gonna starve because, you know, it's free and blah blah blah. So that was and and I loved it. I didn't care about that because I didn't have kids. So it's like if they starve, they don't even exist to begin with. It's just me.

So so there was and I very proudly found a what I consider to be it wasn't even a bug. It was like, hey. We could reshuffle this code and and make things a little bit better.

And I sent my 1st contribution to the Linux kernel. This again, it was either late 20 2003 or early 2004. I don't remember. I actually tried to find this message on on in archives, and they only go as back as far back as 2005. So it's it's not there. But then this guy that was very high profile, very high profile in the Linux Vercel, he was the maintainer of all the virtual file system interface all viral.

Wes interacted later in more pleasant ways.

He responds to to my first contribution as a 20 year 21 year old guy, like, just trying to understand software.

Learning c by reading the Linux kernel source code, and here's my contribution.

And his response Wes, this is the worst piece of code I've ever seen in my life. You managed you managed to introduce 3 bugs in a single line of Node, and people like you should never be allowed to get close to a keyboard again. So that was my welcome to the Linux kernel.

And in a sense, I was hap in in a sense, I was blessed because it did not get, you know, it doesn't get much worse than that. And that was already like the my first impression. So things actually got better after that. Right? Just but but that was, that was that was my entry point. And, look, I think, I I don't follow Linux that much these days, the the development, and and maybe things got better. I think I have, you know, when I was leaving, I think people things were already getting better.

But the thing is that I, at the time, I actually appreciated that. I I, you you may find, like, a Stockholm syndrome or maybe I'm a masochist or whatnot. But the way I saw the tech industry back then was, like, this very fake place in in which, like, nobody's telling the truth at any time. And, like, I was starting to see, like, the folks that were later in in in college, like, 4th year, 5th year starting, like, their internships.

Like, they were all, wearing, like, nice clothes and and speaking on a very corporate fake way.

And I just consider that attitude at the time should be just a breath of fresh air. I mean, like, this person is telling me that I should never get close to a keyboard again because he thinks I should never get close to a keyboard again.

And in in a sense, I I did not like hearing that.

But I like the fact that he was not telling me, you know what? I I wished, in in future contributions, we should strive for a more synergetic approach to get our OKRs in in a better place to, so I I I found that refreshing in in the beginning, if that makes sense. And and later I got tired of it, but but I found refreshing. It just

Do you use Linux as a a desktop

right now or are you? No. I just Wes when I joined, when I joined Datadog, I I got a Mac and then I just got used to the Mac. And and, my cofounder, was actually the maintainer of the memory management subsystem for Linux at the time.

Mhmm. We've worked together in the Linux kernel for around 5 years, and he also uses a Mac. So we're always joking that, you know, we we made the sausage, which is why we don't eat the sausage.

It's Dallas. Yeah.

But but I I still I still have a bunch of Linux servers. I use Android for phones.

It's just that, like, a Scott of things got simpler, on my especially now that I don't Node that much anymore. Like, I'm just managing the company.

A lot of things got simpler for me on a Mac, so I just keep using the Just Yeah. I Node that.

Cool. Let's move in the last section of, the podcast here, which is, 2 sections. We have our sick pick and our shameless plug. Did you come prepared with those?

I am terrible of being prepared for anything. I just Okay. So I definitely think I did not Anything in your life that you're a big fan of It's giving you joy. Coding related or Scott? Man, what in my life gives me joy? That I I'm I'm happy that my wife is not listening to the podcast because, like, she's always complaining. Like, nothing brings you joy. You are probably the worst person.

So she it's just if she if she's watching this, which I'm sure she's not because, you know, she's not a techie, thanks be to Node, she she will be, essentially saying, like, nothing brings you joy. You just but you know you know what? This is not a sick thing, but I wanna share something which which which again is related if if I may.

The other day the other day I found out that, I suffer from a condition called aphantasia. Have you heard of this before? No. I I feel like I've heard of it, but I'd have to look up. Yeah. So all my life, peep people would tell me things like, oh, visualize this and picture this in your mind and make it like, if you're trying to do meditation, for example, which maybe will be my sick pick. I will I I was trying to the other day to do meditation, etcetera, and I completely failed. And I think because of this, people try to get you to visualize those things very vividly so you can be transported into that environment.

Mhmm. I I don't I cannot visualize anything.

And for all my life, for all my life, I was always thinking maybe people Wes they say visualize, they mean something else. Node real visualize being a metaphor. And then I found out that actually apparently 4% of the population are like me.

We can't visualize s. Right? Nothing. Like, my mind is I does not have pictures. If I close my eyes and and I'm asked to imagine an apple, I don't see an apple. I can describe an apple to you. I can remember what an apple is, but I don't see an a apple.

And then my left 1 hour yeah. And and apparently, I learned recently, like, 2 months ago that no. It's not a metaphor. Apparently, normal human beings can visualize the damn apple, and I'm the one who can't. And and there's something wrong with me. So, my joy recently have been trying to figure out more about this condition.

Yeah. And and if there is any if there is any way that I can still, and and things that will require visualization despite the fact that that I suffer from this, which apparently there's no fix. It's just by it's just like being left handed. It's it's what it is. It it just born like this and there you go. Not that sick. In factually, if this is a condition, this is technically a sick, pick, except that there JS nothing that I can do to pick sickness pick, except there's nothing that I can do to to cure this apparently, but I've been reading a lot about it. And I've been, finding, a lot of fun in trying to understand, how the mind works, how the brain works, and and why is it that I I cannot see a apple at all while, apparently, if you guys close your eyes now, statistically speaking, you will see an apple if if I tell you I and I don't.

Yeah. I I think that is a sick pick though because, you know, what? You know, my wife's a doctor of psychology.

And one of the things that she talks to even with parents is it's it's not about the label that you get.

It's about the empowerment you get to understand yourself Yep. Because you have that that that bigger picture now of why you're like this and that. So it JS funny because, you know, parents sometimes they because she she's mostly testing children. They get wrapped up and, like, I don't wanna put these labels on my kid. It's it's not about the label. It's about being able to understand,

you know, what's going on. So Oh, my god. That Wes liberating. That was liberating for me because, again, all all my life, I thought I thought that this was just a metaphor. When people say visualize a visualize yourself on the beach.

Mhmm. I say, oh, that must be a metaphor because if I close my eyes, I see nothing. And this understanding that other human beings are different and and how they're different and and what is the baseline allows me, for example, to try because I I was always fascinated attaining any meditative state.

And I don't know if I ever will, but at least now I know that if I am to, I have to try a different way. I can't just try the same way I've always been told because it always involves some level of visualization,

and I just know I'm physically incapable of doing it. Right? So just I have to find another way, which is Yeah. That resonates a lot with me. I mean, I I for a long time I'm I'm a really bad reader. I have dyslexia.

And for a long time, I was trying to force it. Like Mhmm. I feel everyone says you should be reading. I feel like I should be reading.

And then at some point, once I realized it's not good for me to try to do that, I just embraced audiobooks completely, and I I I don't even try to read books anymore. And it's made my life so much better. Yeah.

So just understanding yourself. Is there anything you would like to plug? Any events coming up, anything specifically, or, just anything you'd like to plug at all?

I I will be in August in, Park City, Utah for React Rally. So again, I don't, I please be gentle with me if you're there.

I'm not a front end developer. Again, I'm a Linux guy turned database engineer, now turned database company CEO. So I do not understand React.

But I always like to be, you know, those are, we we have a lot of users of coming from those communities. I think Turso is a tool that empowers, developers to use, to use their own databases. Right? Just that without having imagine that which is by the way, one of the things that we do not talk too much about.

Imagine you're but you you hinted at this, Wes. Like, you you you are not you are a front end developer. You don't understand databases very Wes, at least not as much as, you know, the core database people in the company.

Durstree will allow you to create a replica of the database and operate in replica without disrupting the main service. Right? So one of the things that we do wanna allow people to do is, like, have this flexibility and and and go wild. Don't fear your database anymore.

So I'm always trying to get close to those communities, understand what matters to folks like you so we can provide those things to you through. So so I'm gonna be on rally in August.

And the end of August, I am attending the Laravel conference.

I go I hope you guys are Scott, mad at me, for playing both ways. I do No. We love we love Laravel. Yeah. I've spoken at that conference many times, and,

a bunch of Century folks are going to be there as well. Awesome.

And Lambos Lambos for for for them and the West Coast for the JavaScript community. I'm happy to put it that way. But, but but, I'm gonna be there. And then in September, Kent Dodds, is is organizing a very select conference, which is invitation only.

There's gonna be a 2 day conference, in in in in a hotel in Provo, Utah, and I'm gonna be at that conference as well. We at Thirse who are sponsoring that conference. So, like, if you wanna chat and if you are in any of those 3 places, feel free to, come chat. It's always a pleasure for me to to chat about anything really, from Fantasia to databases to in the Linux kernel. I would love to connect with with folks, there Yarn there.

I love it. Well, thank you so much. This has been awesome. You know, I really illuminated to me, like, what makes Terso different because it's easy to look at things and say, oh, it's another, cloud database. But it's it's been really awesome to dive into what makes it all so special. So thank you so much for all that. It was a really great game to chat.

Absolutely. My pleasure. Thank you so much for having me, folks.

Thanks, Adam. Peace.

Peace.