SURREALDB Posts
I've been spending my spare time working on Node.js tools for SurrealDB. The 2 that are showing some promise are named Surrealigrate (a migration tool) and Bole ORM. I may end up changing their names, but for now that's what I'm calling them.
I've talked about both of them, so I'll try not to repeat myself. I've worked on expanding Surrealigrate some, but I need to work on some more advanced schemas and namespace setups to really push it. This week I've mostly focused on Bole ORM. The word "bole" is a kind of clay or the trunk of a tree, which both are kinda fitting for an ORM, in my opinion. My approach with all of my SurrealDB tools has been to take advantage of the built-in info queries and generate database specific code. A lot of tools, especially ORMs, are very standardized and strict with their implementation. I don't want that. I want to be able to generate query tools that are specific to the database and then be able to adapt them however I want, without the ORM getting in my way.
Bole ORM is a simple CLI tool and it generates a query class built for that specific database, down to table names and specific fields' validation. It doesn't require any kind of schema files or anything that manually needs to be done by the developer, it's all generated by inspecting the database. While it currently generates a class, to be used as object-oriented, I also plan to make it configurable, so if you prefer chaining, it'll do chaining, instead of objects. It resembles Prisma in the way the queries look, when using the class output, but it's simplier and I'm pretty sure the where and fields (Prisma's select object) objects may support more options than most Prisma methods support.
All the Flavors uses Prisma, so I'm very familiar with it and it's actually powerful when combined with a GraphQL parser. That is why I'm mimicking Prisma, to some extent, but also adapting with some things I feel Prisma could have done better. Once the feature set is, well, set, then I'll go into more details for the actual capabilities...but there's also a catch. I plan to make the generator modular, so you can plugin whatever you want, leave out whatever you don't, or add custom modules. This means you can basically design your own ORM with modules and just let Bole generate it.
I want to add some additional utilities to each. For Surrealigrate, I want it to be able to identify potential missing indexes and suggest changes. I also want it to be able to use either schema within SurrealDB or from a library, such as Zod, to generate migrations. The other side of that, I want Surrealigrate to be able to generate a Zod schema and validations for use outside of Surrealigrate, in Bole ORM or just in your application in general. As far as Bole ORM additions, I want to generate documentation for the specific database, and Bole ORM itself, and I want to do automated tests, primarily on data integrity, but also for the application itself.
The idea behind these tools is to simplify development and have something you use when you need them, but they get out of your way when you don't. There's no point in making someone run a bunch of CLI commands, when you can use Surrealist to design your database, then just run a simple command that does everything else. That's the way I see it anyway. If extra commands are involved, they are optional and they add value, like a command to generate a Bole query that you can paste into your application. That could just be a plugin though, and then it just updates a generated export and you never have to touch it again. Working for you, not against you.
In case you haven't noticed, I've been building a lot of SurrealDB tools lately. I'm a big proponent of having a deep toolbox that works for you. The old PHP frameworks had basically everything you needed and you could build pretty much anything with them. Node.js is slowly getting there will the SSR frameworks, but if you are using something new, like SurrealDB, there aren't a lot of options.
Everything I've been building isn't just for SurrealDB. I've also been building more generic Node.js tools too. My approach is to build some libraries that aren't NPM packages and don't have a lot of NPM dependencies. The idea is you can choose the libraries you want to use, put them in your project as a submodule, and either follow the projects' progress or just freeze it and use as is, or even make it into something more tailored to your project's needs. Here are some of the more generic ones, before I get into the ones I'm really excited about.
Node.js Process Manager
A simple library that can be used to manage your Node.js processes, like starting, stopping, restarting, or auto-starting a script. There's not a lot to it and it has very few NPM dependencies. If it expands much, it will likely be to add dependency management, so if your script requires another process, it will ensure that one is running too.
Quality Control Manager
This is a different approach than what I normally do and has the opportunity to really make managing a project easier. This is a bit broader than the process manager and wears several hats, but I think it's a needed tool. At its base, it's a documentation generator, but it's also much more. It uses AST to outline all of your code, then uses that to generate documentation, a code trace tree for each function, and usage examples. It has a built in server or it can output the files for use elsewhere. I also want to add tests generation, error checking, and quality analysis.
Others
I have some others, but I don't want to discuss those yet, as I don't have a clear vision for their potential implementations. Let's just say, I'm working on automating pretty much everything.
SurrealDB Tools
So this what I'm getting excited about.
Database Migrations
I've built a migration tool for SurrealDB. Not a big deal, right? Well it's pretty cool. It does the normal migration handling stuff, so you have a development server and a production server, and it allows you to sync your database schema, by using migration files. It also allows you to roll back migrations, either back to a specific migration, back to the beginning, or forward to a specific migration. The cool thing is you can allow it to inspect your database, you can then make whatever changes you want, then allow it to re-inspect your database and generate your new migrations for you. It compares the inspected database to the new, modified database, and builds the script to go from old to new on another server.
Database ORM
I've built a tool that can generate an ORM, based on your database schema. It's built on top of surrealdb.js, so it doesn't limit anything you can currently do, but then it provides some really cool shortcut features to make complex queries as easy as possible. It currently looks a lot like how you use Prisma, but without a lot of the corners Prisma tends to push you into. This thing really adds a lot of features and I plan to retool the generator to let you customize how the ORM works. So if you prefer chained queries, then it'll do chaining, if you prefer object queries, it'll do objects. It also uses your current database to generate the ORM, there's no schema file to deal with, like Prisma has.
SurrealDB Crucible
Finally, this isn't so much a tool, as a way to push the limits of the tools. This is where I put the tools together and make sure they all work together. It's already been useful to find limitations and prove the loose projects approach is capable. It's basically a proving ground, but I'll likely use it as a way to document all of the tools and demonstrate how they work together. It could also become a starter template, or at least the basis for one, in the future.
So that is some of the stuff I've been working on in my spare time. It's been a lot of work, but I feel like each tool is building onto the previous tool, so eventually it'll add up to saving time, I believe. The migrations and ORM alone have a lot of time saving opportunities, for myself anyway. I plan to add branching and more complex version control to the migrations. I also want to add the ability to export specific tables to create a new migration tree, so then it becomes a tool than can be used to spawn new projects. Learn from the best, then make things better :)
I've been building a new ORM for SurrealDB, which is designed to work with my SurrealDB migration tool. It is a generative ORM, so it will be tailored to the database and you'll be able to generate the ORM methods in either Typescript or ESM Javascript. It will have built in types validation and a ton of query options. It will also, eventually, have custom specs, so you can override the build instructions to result in the structure you desire. This means if you prefer one ORM style over another, then you'll be able to use that style, but have the same features. Whether you want chained methods or a simple object parameter, it'll all be configurable. You'll also be able to create or utilize multiple styles, using a namespace, or import additional table classes from within packages, because all of the methods will be compatible. Pretty cool I think :)
Among my SurrealDB tools I've been working on, SurrealMaestro is a CLI startup and shutdown manager for SurrealDB written in Node.js. It can manage multiple databases, comes with a status tool, PID tracking and checks, and can be used in automation tools. This should be beneficial to Node.js projects running SurrealDB, as it can be used with start up commands and used in a development environment, as it doesn't attempt to start a server that is already running. I plan to also build a library within SurrealMaestro that lets you build the database control into a UI application.
My interest in SurrealDB has really grown the past few days. I really feel like it can replace ArangoDB in my stack. The only issue is the same issue I had when I adopted ArangoDB: there aren't a lot of tools for Node.js. Like ArangoDB, I'll maybe have to build some myself. The first such tool is a CLI migration tool.
First of all, the Surrealist database manager application is awesome. It's great, it really is and, while ArangoDB has it's web client, it has functionality I missed using ArangoDB that I used to have for relational databases. That is a great tool, but it's more of a development and management tool. I need a migration tool for automation and to make my applications more portable.
Today I made a big push toward creating this tool and it's named Surrealigrate. I've built tools like it for other databases, such as the one we use for All The Flavors, Arangrate for ArangoDB, and I had initially built a multi-database tool for the CMS behind this site (now it uses Prisma, well, for now). Surrealgrate isn't quite ready, but like I said, it's getting very close. It's the first thing I've built from the ground up with AI assistance, which has sped up development considerably. Unfortunately, AI's aren't perfect, so there were a few issues and I had to rewrite portions of it, but most of the changes were just oversights that occurred as I asked the AI to add specific features in steps.
I need to build Surrealigrate up fairly quickly, so I can move some of my current projects to it and get them moving again. While it is already more advanced than my other tools have been, there will be more features coming later. Using AI allowed me to put in a lot of things initially that I wouldn't have had time to do right now. One different approach I'm taking with this tool is it's not intended to be an NPM package. It's intended to be included in a project as a component, potentially using git submodules. I'm hoping this approach makes it more useful per-project and will lead to others contributing.
I've been learning how to use SurrealDB and, honestly, I'm kinda loving it. Still learning, but I haven't found anything I don't like. It actually seems to solve my two biggest issues with ArangoDB too...it has very low resource usage at smaller scale and it's incredibly easy to deploy a simple server. Like I said, still quite a bit to learn, but I'm thinking SurrealDB would be a viable primary option for Grazie! 2.0. ArangoDB never was, just because of the issues above. I wish I'd found it sooner.
ArangoDB's new licensing has made me not want to use it anymore. Some of my ideas lean very heavily toward being a SaaS and ArangoDB explicitly disallows it's use as that. I know it states "source code", but I would be using it from the openSUSE repository, which would mean it would be compiled from source and not the binary from ArangoDB. I don't like it. It just feels like it would be too easy for them to make an accusation with the way they have it worded. I also don't want to grow to their 100gb limit and then have to find an alternative or agree some to Enterprise version price that they won't even advertise on their web site. I tried to contact them through the web chat and email [for a quote of the Enterprise version's price] and didn't get a response yet from either. It was really for confirmation (that I can't afford it) and not a last ditch effort to keep using it.
So...I started looking for a new database. Initially, I wanted something with an OSI-approved license. Unfortunately, the one I like the most also uses the BSL 1.1 license, but without the extra conditions that ArangoDB has instated. To me, that's fine. It's the MariaDB license. They have to protect their product. I understand all of this. The ArangoDB's BSL wasn't what I had a problem with, it was their extra conditions; I don't want to run a DBaaS or offer managed database hosting, that's not what I'm interested in doing.
And...it looks like I'll be moving on from ArangoDB to SurrealDB. I considered all of the ways I could get around a change; I don't really like change and I've loved everything about using ArangoDB. I considered staying on a version before the license change and just updating every 4 years when the BSL rolls over to become an Apache license. I just don't want to deal with all of that. I don't want to use 4 year old, unmaintained software. SurrealDB doesn't have an Enterprise version. They've held their BSL license for quite some time and it at least appears that a lot more people use SurrealDB than I've ever found to use ArangoDB. Plus, I may be able to actually find someone else who uses it...who knows, I may want to even hire them.
Yeah, it's becoming a thing. Now I just need to find or create the tooling I need to use it and...start using it.