OT: What happened in the database space and why? In 2000 you would have had a hard time naming enough database systems to run out of fingers for counting. But in recent years database system seem to pop out of the ground like JavaScript frameworks.
So what changed? Did the idea of a general purpose database system break because the usage scenarios became so diverse that we can no longer pretend that one size fits almost all? Is this just a perception thing because everybody started uploading his weekend hack toy database to GitHub and blogs about it?
And what really irritates me the most, database systems are really complex pieces of software requiring years and years of development time, where does this time come from and who pays for it? Especially having more and more different databases system to choose from does not really promise that developing yet another one will be an easy commercial success.
Realm has been available for many years and is actually really different from what else is out there by the fact that it is a real object database. It has just been flying under the radar a bit given that it has only been available for mobile devices.
I for one think that it is awesome that it is now available for node.js as well. Especially with the new replication functionality that allows you to share live data between node instances running on different machines.
"Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases.
Those databases failed not necessarily because the technology was wrong, although they certainly made some choices that didn't help; those databases were often tightly coupled with opaque binary object serialization formats (with no way of looking into a database without going the binary route) that were in turn tightly coupled with the app's class hiearchy. For example, several of these actually post-processed your code (e.g. your Java bytecode) to insert serialization code, so that when you followed a relationship (e.g. "library.books[0].author"), it would automatically query under the hood and return the author as an object. They ended up being clunky to work with as a result; for example, in the beginning, very few of them had a query language, so the only way to query the data was by following relationships through your graph of objects.
Realm looks interesting as a new approach to the object database design, but at the same time seems a bit limited. It doesn't seem to have schemas or any kind of role-based permissions, and seems to make the same mistake of not including a query language (so you can't build a REPL for it, and every language binding needs its own query builder). The transaction semantics also seem to be entirely undefined (is it read-committed?). From what I can tell, it's also "offline first", in the sense that you have a local database that can sync to/from a server, but you're technically always working on an offline copy of the master data; while nice to have for a mobile app, it does limit what you can use it for. I also don't see any support for fine-grained updates. Every write you do seems to overwrite, which I assume means the last write always wins; how do you deal with conflicts, or implement things like counters, maps or arrays that need to be incrementally updated?
Realm certainly has a schema - anything that made you think it doesn't?
It's ACID and Serializable.
It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well.
It's absolutely designed to be offline first. I'm curious as to which limitations you see with that?
Operational Transform is used to resolve conflicts at the property level, so I would say it's much more fine-grained than most other systems that would overwrite entire objects. The specific resolution rules are very simple and intuitive and how to deal with custom needs are described here: https://realm.io/docs/realm-object-server/#conflict-resoluti....
There is ordered lists already and explicit counters are being exposed in the SDKs at the moment.
I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized.
(By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language, written as text, which is portable across clients.)
OT is great, but it doesn't have that until it's actually exposed. My pet use case is string edits, which absolutely need to be cleanly conflict-resolvable.
The inconvenience with offline-first is that by definition you're never operating on current data: You have to sync to a central server before the data is visible to others, and you're editing potentially stale data. The latency of inbound syncing is particularly dependent on the volume of changes being sent from upstream, which might require that a client, so as not to be overwhelmed with updates, subscribe to a very small subset of the entire dataset. For example, if you open up a UI that edits a single document, you might want to subscribe to changes to only that document, in order to update the UI in real time. I don't know if that's even possible with Realm, or if you can in fact run it in "classic client/server" (i.e. "online first") mode.
Why would you want a text-based query language if you are an idiomatic C# programmer? Every database product on that platform is queried through the standard LINQ syntax - that's why we implemented it that way.
NoSQL databases arose in response to the growth and change in data use, driven largely by the web. Typical needs cited include scalability, availability, performance, and flexibility. Development has been going on for a long time.
I think most would agree there is no longer a one size fits all (or even most).
Realm (and I know someone will correct me if I'm wrong) started out as an on-device only replacement for relational-style dbs focused on ease-of-use and performance.
It's not the only one. Couchbase Mobile is a NoSQL stack that has an on-device db that's also easier to use, and has focused for many years on getting sync right (which is a hard problem).
I've had some nasty bugs happen in production with Realm, as is expected with any new database store. Newer version seem better in this regard, but if you read the changelogs you might be shocked at how fundamental the fixes can be.
It also times some time getting use to -- my biggest gripe is that the notification API won't tell you what actually has been modified, just that an object changed _somehow_. That makes a pure reactive design quite hard if those changes have non-trivial consequences, and not just a quick re-rendering of some element on the screen.
Looking forward to finding time to go back to SQLite where I can actually do complicated queries, relational joins, triggers, etc on a fast and mature code base.
I've also experienced some production issues with Realm. I've been using it for an App Store mac app since before 1.0 (Quite risky, yes, but it's for a simple use case). In the worst cases, it caused data corruption for end users.
Over the past year, it's become a lot more stable and I've seen far fewer user reported issues with it. As others have pointed out, it's also a very nice API and very simple to get set up.
The Realm team has also been very helpful in my experience and usually have responded to my support requests almost immediately, despite the fact I'm using their free product and not the DB-as-a-service they recently shipped.
I wouldn't take exception to it's stability if they didn't call it ACID compliant. I was about to use it in a setting where failure to be ACID complaint would be fairly expensive (but not safety critical). In just 24hrs of testing I managed to end up with a corrupt DB. In the same tests SQLite's been running for almost a year without corruption...
I started using sqlbrite (in combination with sqldelight) and stopped treating the database as an object store. Instead I create an interface for the data the customer needs (e.g. an item interface for elements in a listview), write a select query whose result fulfills this interface, and let the rest happen via databinding.
This seems to work fine up to a couple of thousand of elements for me.
I am assuming this is coming from an Android point of view? If not, we do support passing what changed in the notifications for all the other platforms now. We call it "collection notifications" for example with JS: https://realm.io/docs/javascript/latest/#collection-notifica...
Android support is coming in a matter of weeks, so stay tuned.
Property changes are only supported with individual object notifications that were introduced with Realm 2.4. We haven't planned to add property changes in the collection notifications yet, but will definitely note the request!
Except that's on an individual object -- existing KVO functionality re-invented. I want it on an entire collection... basically their existing API except give me more than an index.
Is there any technical information on how the database works? For a supposed Sqlite replacement, there is a surprising lack of the information. The only reference to concurrency mode I have found is this:
> All changes to an object (addition, modification and deletion) must be done within a write transaction.
> Write transactions incur non-negligible overhead - you should architect your code to minimize the number of write transactions.
So is there a single write lock? If you try to start two transactions, will the second one be delayed or retried? Can reads proceed while the transaction is processing?
> Results instances are live, auto-updating views into the underlying data
So if I am doing x.first, x.second while the object is being updated in a second thread, will I get mixed state?
Is database crash proof? If the phone runs out of battery while in write transaction, will the database get damaged?
It's ACID & Serializable. A write will block all other writes, but multiple reads can occur without overhead or blocking due to MVCC.
Reads are consistent within the runloop, but updates to your objects occurs when you start a write transaction, so you know you always work on the latest version of the data. Basically, you are in an implicit read transaction all the time, except when you are in a write transaction :-).
> So if I am doing x.first, x.second while the object is being updated in a second thread, will I get mixed state?
So no you will not get mixed state.
> Is database crash proof? If the phone runs out of battery while in write transaction, will the database get damaged?
I have dabbled a bit with it quite a few times and was quite happy about it, but my main issue is that AFAIK there is not "pure" Java API for Realm.
It seems there have been attempts at removing Android dependencies but it seems they were never finished, meaning you can't have it running on desktop apps.
Adam from Realm, thanks for the feedback. We do plan to offer a pure Java API, but it hasn't been a top priority. However, with the launch of Realm Mobile Platform we have been getting more requests for this because teams want to use Java vs. Node on the server-side to build integrations. My best guess would be Q3 or Q4 of this year, since we still have higher priority features to work on.
It would be really cool to see this working in Ionic/Cordova. I've played with Firebase and CouchDB (and also SQLite) in Ionic, so would be good to see how Realm could work in the same context.
Sometimes I just want a simple persistent on board storage mechanism for my mobile apps without all the problems that localStorage exhibits. I wonder if Realm could be a lightweight alternative?
There are two products - Realm Mobile Database[0] and Realm Mobile Platform[1].
The former is, as the HN post suggests, basically a database for mobile applications which you can use in place of SQLite or as a key value store.
The latter combines the database software which runs on the mobile devices with the Realm Object Server[2] which runs server side to provide something like Firebase (disclaimer: I've used the Mobile Database but haven't used the Mobile Platform (and Object Server) yet).
The biggest would be the Realm Mobile Platform is not a service, but a server-side application, Realm Object Server, that you can run on your own alongside the mobile client SDKs embedded in your apps. Another key difference, is that with our Node.js SDK you can create a level of integration with any existing backend infrastructure that would be harder with Firebase.
No, it's (at least) as easy to use as Firebase, but it's much more powerful. It uses actual objects, not json. It's very close to manipulating normal objects in the language, extremely nice to work with. There are limitations, especially with the syncing, but it's by far the nicest database solution I know of.
I don't think of Realm as being at all like Firebase, except from a very high level. Realm isn't hosted, it's an object database, until recently didn't do sync, and more.
So what changed? Did the idea of a general purpose database system break because the usage scenarios became so diverse that we can no longer pretend that one size fits almost all? Is this just a perception thing because everybody started uploading his weekend hack toy database to GitHub and blogs about it?
And what really irritates me the most, database systems are really complex pieces of software requiring years and years of development time, where does this time come from and who pays for it? Especially having more and more different databases system to choose from does not really promise that developing yet another one will be an easy commercial success.