PROJECTS Posts
This past week we released Recipe History, which allows you to track recipe revisions, revert to older versions of a recipe, or save an older version as a new recipe. We also released an update that allows you to copy a new recipe from a Batch of another recipe or update a recipe from a batch that was modified in the calculator before saving it. The third update added the flavor information from the Recipe editor to the calculator freeform and modify features. Finally, we added user cards that can be clicked to activate on Recipes and Formulations.
It's hard to believe we only have 12 weeks left this year. We still have a lot we want to do. This past week I developed the user cards and did some finishing touches on the Recipe History before launch. I also worked on the ability to mix a recipe as a flavor shot, which is mostly functional now, and the ability to select ingredients in a formulation to be part of a premix. These are all built on the branch that adds the ability to mix a formulation without a recipe, so the part that creates a premix.
We are getting close to the calculator state we want, which will allow us to add more functionality to it. We also want to break up the calculator and make it more modular, but that should preserve functionality and just allow us to do more things more easily in the future. I should be getting my co-worker back soon too, so either the pace will pick up or we'll be developing a wider range of features.
This week I opened pull requests (reviews) for updates that add a revision system for Recipes. This displays every change to a recipe, including if it was created from a copy of another recipe or from a batch of another recipe, or updated from a batch from the same recipe.
I also opened a pull request that adds the flavor info to the calculator modify function and to the free form calculator.
After that I've worked on adding the ability to mix a recipe as a flavor shot, at a specific percentage, the ability to mix formulations as premixed bases, and the ability to designed premixes in the calculator. Those aren't complete and, due to the overall complexity of the calculator, are actually a little more difficult to implement than you may imagine.
There is another calculator update on tabs next, which will make it possible to select any formulation from the calculator itself, rather than having to find it on the recipe page. That should catch us up to what most people have requested and lines us up for a new type of ingredient and a new way to create formulations and use formulations. Eventually I want to add some new community features and better browsing features for batches and tags...but I'm slowly getting there.
I've been working on a Recipe versioning system for ATF. It's been...not that easy, but I ended up reverting back to the simplest way I could think of. I've probably built 10 different versions, ironically, of the version system. The system I ended up deciding was the best is mostly like openSUSE's Snapper. It creates a pre and post-change snapshot, with some pre variations, depending on the source of the revision. Currently, the only time a revision is triggered is when a flavor changes, but I am experimenting with other things, like recipe descriptions.
Additionally, I've also been working on the ability to update or create a recipe from a batch. This ties into the versioning system, but is change to the way we normally update recipes. The idea is, instead of simply editing a recipe, you create a new mix and change the flavors within the calculator. You then mix it and try it, if it's good, then you can update the recipe directly from the batch. This creates a new revision of the recipe and ties that revision to the batch and recipe. It's a change to the normal recipe development workflow, but it should make it simpler and reduce the number of clones and things like that.
Finally, I haven't forgotten about clones. If you clone a public recipe, that recipe will be tied to revision 0, which is the initial version. If it wasn't a public recipe, then it acts like the current cloning system and ignores the original version. This allows you to create original recipes, even if they are actually based on a private recipe base. Also, sometimes you may create a new batch that is good, but should be it's own recipe. The new version system will allow you to create a new recipe from that batch, like a clone, except it's from a batch. This means you can start out just creating batches, and when you get to the point where it's a legitimate recipe, then you save it as a recipe from the batch.
So the next features will be recipe versions and recipes (updates or new) from batches. After that we have some calculator updates and some new recipe features, plus some community features. I'm not sure which order those will come, but both are a priority. Among the calculator updates are some new features, mostly for our flavorists and vendors, but they could be exciting for others in our DIY community, who aren't necessarily happy just using regular flavors ;) More to come on those. That's it for the week 38 update. Sorry for the gap in weeks, I'll try to do better with the updates.
I've been spending my spare time working on Node.js tools for SurrealDB. The 2 that are showing some promise are named Surrealigrate (a migration tool) and Bole ORM. I may end up changing their names, but for now that's what I'm calling them.
I've talked about both of them, so I'll try not to repeat myself. I've worked on expanding Surrealigrate some, but I need to work on some more advanced schemas and namespace setups to really push it. This week I've mostly focused on Bole ORM. The word "bole" is a kind of clay or the trunk of a tree, which both are kinda fitting for an ORM, in my opinion. My approach with all of my SurrealDB tools has been to take advantage of the built-in info queries and generate database specific code. A lot of tools, especially ORMs, are very standardized and strict with their implementation. I don't want that. I want to be able to generate query tools that are specific to the database and then be able to adapt them however I want, without the ORM getting in my way.
Bole ORM is a simple CLI tool and it generates a query class built for that specific database, down to table names and specific fields' validation. It doesn't require any kind of schema files or anything that manually needs to be done by the developer, it's all generated by inspecting the database. While it currently generates a class, to be used as object-oriented, I also plan to make it configurable, so if you prefer chaining, it'll do chaining, instead of objects. It resembles Prisma in the way the queries look, when using the class output, but it's simplier and I'm pretty sure the where and fields (Prisma's select object) objects may support more options than most Prisma methods support.
All the Flavors uses Prisma, so I'm very familiar with it and it's actually powerful when combined with a GraphQL parser. That is why I'm mimicking Prisma, to some extent, but also adapting with some things I feel Prisma could have done better. Once the feature set is, well, set, then I'll go into more details for the actual capabilities...but there's also a catch. I plan to make the generator modular, so you can plugin whatever you want, leave out whatever you don't, or add custom modules. This means you can basically design your own ORM with modules and just let Bole generate it.
I want to add some additional utilities to each. For Surrealigrate, I want it to be able to identify potential missing indexes and suggest changes. I also want it to be able to use either schema within SurrealDB or from a library, such as Zod, to generate migrations. The other side of that, I want Surrealigrate to be able to generate a Zod schema and validations for use outside of Surrealigrate, in Bole ORM or just in your application in general. As far as Bole ORM additions, I want to generate documentation for the specific database, and Bole ORM itself, and I want to do automated tests, primarily on data integrity, but also for the application itself.
The idea behind these tools is to simplify development and have something you use when you need them, but they get out of your way when you don't. There's no point in making someone run a bunch of CLI commands, when you can use Surrealist to design your database, then just run a simple command that does everything else. That's the way I see it anyway. If extra commands are involved, they are optional and they add value, like a command to generate a Bole query that you can paste into your application. That could just be a plugin though, and then it just updates a generated export and you never have to touch it again. Working for you, not against you.
Every week I'm going to do a summary of what I've been working on for All the Flavors (ATF) and some of the things we are planning. I'll write it throughout the week and try to publish it by the following Monday. This is the first time, so I'll start with
Week 31:
We released ATF v.2.3.13. It featured administrative tools for creating and managing badges for Users, Recipes, and Vendors. It also fix a bug in the messages application that prevented replies to reports sometimes. Finally, it introduced improved email queuing and sending, without which could make the API seem sluggish at times.
I also submitted the Pull Request for Remixing Batches for Recipe + Formulation batches.
Week 32:
Monday I submitted Pull Requests for improving the button layout in the Recipe editor and permissions system improvements. Finer-grained permissions will allow us to delegate some tasks to specific users, such as a Vendor being able to assign their badge to users or a streamer having a badge for their followers.
I also worked on a new feature earlier in the day, which I wasn't able to complete. It involves a new calculator mode, but I'm not sure when I'll be able to finish it, so I wont say what it is just yet.
Tuesday I worked on ways to make the mix calculator more modular and the new mix mode. The calculator is very complex, so I want to refactor it to be more data driven, but found it was too much work for this time allotment. Then I planned out some ways to utilize public batches.
Wednesday I submitted a Pull Request that adds a Batches list to Mixers' profiles to display public batches. I also restyled the way batches display to make some slight improvements and added a Mix button to both the My Stuff batches and profile batches, which allows you to mix that batch [again].
Thursday & Friday I worked on a data expansion project we are integrating into ATF
Batches on All the Flavors are the result of mixing a recipe. They are an optional last step (Create Batch button), but they are useful snapshots of a recipe's history, and they provide the utility of updating Stash amounts. One of my goals is to make the batches more important and have more utility. The first step in that was to allow batches to be public, which has partially been done, but there isn't a public directory of batches, you have to have a link to find it. Now I am working on allowing you to remix a batch. I'm still working out the exact details, but the goal is to allow you to just go to a batch, click Remix, and do it over again. It's currently functional, but it's not using all of the batch data explicitly, as I'm still working out option for formulations.
This is a pretty big change in the normal workflow and isn't even the end goal. I wont share all of the plans, just yet, but this new feature, combined with public batches, means you can easily share different iterations of a recipe and other users can actually mix them. It also allows you to move recipe development out of the main recipe, because you can edit in the calculator and keep remixing, editing, etc before updating a recipe or needing to make copies of a recipe. It keeps the workflow for a recipe contained to that recipe, without requiring changes that you don't know if you want to keep or not. It also means if you've mixed a recipe and it was later changed, you can still mix the original recipe without cloning it, just remix the batch you mixed before.
ATF v1 had the concept of Badges, which are like awards that users received for doing something, like having a recipe that had 500 views or submitted bug reports, I even have a developer badge on my profile. Badges were just ways to reward users for contributions and a thanks for being a part of the community. Now, badges are coming back in ATF v2.
Yesterday, I submitted a pull request for All the Flavors to reintroduce Badges. They've been present on user profiles for ATF v1 users, but we haven't issued them since moving to ATF v2. Once the pull request is reviewed and approved, we'll be able to introduce new badges for Users, Recipes, and Vendors. ATF v1 only support them for Users.
We have a lot of exciting new things coming with badges, they are just another step and help restore one more feature we didn't fully implement in v2. There aren't many of those v1 features left, so the new stuff coming is pushing us more toward v3 than adding v1 parity.
I hadn't been talking a lot about All the Flavors here, so I think I'll start. This new stuff is maybe going to change our part of the DIY world.
Fixed some bugs:
Pages now display titles as set in site settings
Improved SEO support
Posts category pages no longer show 404 when not logged in
Home page now shows even number of posts and columns
Error pages have better theme/color scheme support
My next focus will be on finishing Notes and then I'll probably be expanding Pages to support Categories and some more advanced options.
In case you haven't noticed, I've been building a lot of SurrealDB tools lately. I'm a big proponent of having a deep toolbox that works for you. The old PHP frameworks had basically everything you needed and you could build pretty much anything with them. Node.js is slowly getting there will the SSR frameworks, but if you are using something new, like SurrealDB, there aren't a lot of options.
Everything I've been building isn't just for SurrealDB. I've also been building more generic Node.js tools too. My approach is to build some libraries that aren't NPM packages and don't have a lot of NPM dependencies. The idea is you can choose the libraries you want to use, put them in your project as a submodule, and either follow the projects' progress or just freeze it and use as is, or even make it into something more tailored to your project's needs. Here are some of the more generic ones, before I get into the ones I'm really excited about.
Node.js Process Manager
A simple library that can be used to manage your Node.js processes, like starting, stopping, restarting, or auto-starting a script. There's not a lot to it and it has very few NPM dependencies. If it expands much, it will likely be to add dependency management, so if your script requires another process, it will ensure that one is running too.
Quality Control Manager
This is a different approach than what I normally do and has the opportunity to really make managing a project easier. This is a bit broader than the process manager and wears several hats, but I think it's a needed tool. At its base, it's a documentation generator, but it's also much more. It uses AST to outline all of your code, then uses that to generate documentation, a code trace tree for each function, and usage examples. It has a built in server or it can output the files for use elsewhere. I also want to add tests generation, error checking, and quality analysis.
Others
I have some others, but I don't want to discuss those yet, as I don't have a clear vision for their potential implementations. Let's just say, I'm working on automating pretty much everything.
SurrealDB Tools
So this what I'm getting excited about.
Database Migrations
I've built a migration tool for SurrealDB. Not a big deal, right? Well it's pretty cool. It does the normal migration handling stuff, so you have a development server and a production server, and it allows you to sync your database schema, by using migration files. It also allows you to roll back migrations, either back to a specific migration, back to the beginning, or forward to a specific migration. The cool thing is you can allow it to inspect your database, you can then make whatever changes you want, then allow it to re-inspect your database and generate your new migrations for you. It compares the inspected database to the new, modified database, and builds the script to go from old to new on another server.
Database ORM
I've built a tool that can generate an ORM, based on your database schema. It's built on top of surrealdb.js, so it doesn't limit anything you can currently do, but then it provides some really cool shortcut features to make complex queries as easy as possible. It currently looks a lot like how you use Prisma, but without a lot of the corners Prisma tends to push you into. This thing really adds a lot of features and I plan to retool the generator to let you customize how the ORM works. So if you prefer chained queries, then it'll do chaining, if you prefer object queries, it'll do objects. It also uses your current database to generate the ORM, there's no schema file to deal with, like Prisma has.
SurrealDB Crucible
Finally, this isn't so much a tool, as a way to push the limits of the tools. This is where I put the tools together and make sure they all work together. It's already been useful to find limitations and prove the loose projects approach is capable. It's basically a proving ground, but I'll likely use it as a way to document all of the tools and demonstrate how they work together. It could also become a starter template, or at least the basis for one, in the future.
So that is some of the stuff I've been working on in my spare time. It's been a lot of work, but I feel like each tool is building onto the previous tool, so eventually it'll add up to saving time, I believe. The migrations and ORM alone have a lot of time saving opportunities, for myself anyway. I plan to add branching and more complex version control to the migrations. I also want to add the ability to export specific tables to create a new migration tree, so then it becomes a tool than can be used to spawn new projects. Learn from the best, then make things better :)
I've been building a new ORM for SurrealDB, which is designed to work with my SurrealDB migration tool. It is a generative ORM, so it will be tailored to the database and you'll be able to generate the ORM methods in either Typescript or ESM Javascript. It will have built in types validation and a ton of query options. It will also, eventually, have custom specs, so you can override the build instructions to result in the structure you desire. This means if you prefer one ORM style over another, then you'll be able to use that style, but have the same features. Whether you want chained methods or a simple object parameter, it'll all be configurable. You'll also be able to create or utilize multiple styles, using a namespace, or import additional table classes from within packages, because all of the methods will be compatible. Pretty cool I think :)
Update: I disabled the color scheme change transition, due to a bug. It was kind of a novelty anyway, I'll try to fix it and bring it back later.
I decided to go ahead and update to the latest development version of Grazie! CMS. There are lots of theme fixes/updates and little enhancements here and there. I decided to go with a darker dark color scheme and I've tried to make title colors/sizes more consistent across the site. There is also a new color scheme toggle on the bottom right corner of the page and I'll have some social links there (once I update the settings). The light color scheme also has a lot of fixes and clean up, but it's quite where I want it. You can check out my previous Grazie blog to see the whole list and some of the things I'm working on.
I've started work on a Process Manager for Node.js projects that can be used to start, stop, and autostart Node.js scripts. The idea is to create something small that has a simple purpose and doesn't require a lot of dependencies. I'm still looking for a name. Like several of my other utility projects I've currently working on, this one is intended to be included in projects and not installed as a package. I also plan to include a library that can be used for UI control of Node.js applications.
Among my SurrealDB tools I've been working on, SurrealMaestro is a CLI startup and shutdown manager for SurrealDB written in Node.js. It can manage multiple databases, comes with a status tool, PID tracking and checks, and can be used in automation tools. This should be beneficial to Node.js projects running SurrealDB, as it can be used with start up commands and used in a development environment, as it doesn't attempt to start a server that is already running. I plan to also build a library within SurrealMaestro that lets you build the database control into a UI application.
My interest in SurrealDB has really grown the past few days. I really feel like it can replace ArangoDB in my stack. The only issue is the same issue I had when I adopted ArangoDB: there aren't a lot of tools for Node.js. Like ArangoDB, I'll maybe have to build some myself. The first such tool is a CLI migration tool.
First of all, the Surrealist database manager application is awesome. It's great, it really is and, while ArangoDB has it's web client, it has functionality I missed using ArangoDB that I used to have for relational databases. That is a great tool, but it's more of a development and management tool. I need a migration tool for automation and to make my applications more portable.
Today I made a big push toward creating this tool and it's named Surrealigrate. I've built tools like it for other databases, such as the one we use for All The Flavors, Arangrate for ArangoDB, and I had initially built a multi-database tool for the CMS behind this site (now it uses Prisma, well, for now). Surrealgrate isn't quite ready, but like I said, it's getting very close. It's the first thing I've built from the ground up with AI assistance, which has sped up development considerably. Unfortunately, AI's aren't perfect, so there were a few issues and I had to rewrite portions of it, but most of the changes were just oversights that occurred as I asked the AI to add specific features in steps.
I need to build Surrealigrate up fairly quickly, so I can move some of my current projects to it and get them moving again. While it is already more advanced than my other tools have been, there will be more features coming later. Using AI allowed me to put in a lot of things initially that I wouldn't have had time to do right now. One different approach I'm taking with this tool is it's not intended to be an NPM package. It's intended to be included in a project as a component, potentially using git submodules. I'm hoping this approach makes it more useful per-project and will lead to others contributing.