"The Open Data Protocol (OData) is a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today. OData does this by applying and building upon Web technologies such as HTTPAtom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores." - 

CDaT (Clay Data Transport) is a layer/library I am working on that supports the OData standard. It is intended to be used to provide a data services interface from any application within Clay and will accept data from other services using the standard. Each application will be able to use native privileges to determine which type of data should be exposed and accepted. CDaT will also be a prototyping tool for creating the OContext standard for data translations across service platforms.

 Update: CDat has been officially named Passage. More info to come.


A new project named OUYA has been crowd funded to $3M in just 2 days. The project's goal was to raise $950,000 in 30 days. OUYA is set to use Android for its operating system, but promises to be open and allow any of the software and hardware to be hacked. It's not open source, though. It seems like a pretty cool idea (not the first) and the kickstarter crowd has embraced it with over 25,000 backers in the first 2 days. The first day it hit the $2M mark, more than doubling its funding goal for the month.

I can see this having potential if it has support for Netflix and other premium services. I would rather see a modular console that is open and open source, but I think this is a good step in that direction. I think we need an open source console operating system, dedicated to game development. That would get my backing.

The goal for ClayDB 2 has been to expand the Data Dictionary capabilities and build a better way to create and manipulate database tables on any supported database. I've been prototyping the PDO MySQL Data Dictionary within ClayDB to reach that goal. The Data Dictionary has been expanded quite a bit and allows a range of table manipulations. It is not fully tested, so I haven't moved on the other databases' datadicts yet. I have been comparing SQLite, PostgreSQL, MSSQL, and MySQL while developing the prototype in MySQL. All of them are very different, so it has been a balancing act. It looks like PostgreSQL will miss the most functionality, mainly because there is so much there that doesn't translate to the other databases. I've been working with the dummy application to test ClayDB 2 and hope to soon make the necessary application changes across the board, followed by a merge back into the master branch.

Like I said before, ClayDB 2 has been all about improving the datadict classes, but I couldn't resist adding a feature to the adapters. ClayDB 1 only returned arrays for result sets. In ClayDB 2 I've added a getObject() method that returns either an object or an array of objects. I don't plan to change a lot more than that on the adapter side, although I have improved the adapters' abstraction and interface. I may make a few other changes, but I do not plan to mess with compatibility on the adapter side with this version change. The datadict changes will be enough work to bring everything up to date. I have thought about some ideas for ClayDB 3 though.

I think the main feature it is missing, as far as popular features go, is a model system. Now I don't believe ClayDB should be responsible for ORM or anything that complex. Its purpose is to make connections and queries to varying databases easier and it does that fairly well. Instead, I'm considering offering an abstract class that allows someone to implement models and, with some expansion, object-relational mapping. The class would allow someone to implement traditional models (at the framework level) or allow you to work with an object model from within an application API. Now within ClayCMS I have been planning to migrate much of the ClayCMS libraries to application libraries, so there's a little push and pull there. I believe the framework level libraries should be used to implement features within applications, instead of carrying the load themselves. So, as of today at least, ClayDB 2 will focus on improving the Data Dictionary, while ClayDB 3 will likely focus more on data handling and moving more toward object models.

11:59 pmCycle of Change

To every thing there is a season, and a time to every purpose under the heaven:

A time to be born, and a time to die; a time to plant, and a time to pluck up that which is planted;

A time to kill, and a time to heal; a time to break down, and a time to build up;

A time to weep, and a time to laugh; a time to mourn, and a time to dance;

A time to cast away stones, and a time to gather stones together; a time to embrace, and a time to refrain from embracing;

A time to get, and a time to lose; a time to keep, and a time to cast away;

A time to rend, and a time to sew; a time to keep silence, and a time to speak;

A time to love, and a time to hate; a time of war, and a time of peace.

Ecclesiastes 3:1-8 KJV

I haven't read the Bible as much lately as I would like, but this chapter has been on my mine a lot. I have to admit it was more of verse 8 than the rest, but all of it is equally as relevant at any given time. Everything we go through in life is a cycle of change. It's not an exact science, it's difficult to know how long you'll be in one of those times. Of course, we're not just constantly cycling through each one of those individually, they are all intertwined and tangled around one another. Lately it feels like I am somewhere between a time to gather stones together and a time to cast stones.

I've come to realize lately that if I remember each one of those, I will hopefully know which time it is. There is a lot that can be written of those verses, but I will save some of the rest for a different day.

The ClayDB 2 upgrade began because I was trying to finish the Privilege system in ClayCMS. I decided to implement an SQLite adapter for ClayDB, while working on the Privileges database schema, and realized the shortfalls of ClayDB. Here's what I'm come up with for the Privilege system (which I'm still working on):

  • Uniquely scoped privileges
    • Each privilege in the system is required to be unique, but can be assigned multiple times across it's scope to multiple Roles
  • Each application has complete control over its privileges
    • Using Privilege libraries within each application, the application determine what a privilege means and what it allows
    • Applications choose their own naming system and are not tied to a set tree of privilege types
    • Explicit and user created privileges are allowed, as long as an application provides support for determining scope for them
  • An optional base set of privileges will be provided for applications to use
    • This set of privileges will be part of the Apps application and can be extended/scoped by any application
      • Intended to be used for developmental purposes, as it will involve more processing, but can be used regardless
  • Role triggers will be provided that allow Roles for Owner, Shared, Public, and Private
  • An experimental Entities will be included later on that will be a mix between Roles and Users. 
    • Entities are for applications requiring users to be split up in to organizational groups, rather than roles within a single group

ClayDB was one of the first libraries I wrote for Clay and is the only one left that hasn't been completely rewritten. As well as it works, it's still lacking in several areas and is overall incomplete. I've been working on adding new adapters and datadicts to support more databases. What I came to realize was ClayDB could be a lot better. 

Now before I get into the details, this is only about ClayDB. I have not started Clay 2 (yet). That is quite a while away. ClayDB is a separate library that comes with Clay. ClayDB in Clay 2 will be based on Clay 2's roadmap requirements and could still be ClayDB 2 or ClayDB 5, it's hard to tell.

ClayDB 2 is going to be geared more toward flexibility for different databases and work a lot better on the datadict side (table manipulation). MySQL is the most popular database Clay caters to, like now, but will allow verbage for developers working with other supported databases. The goal with ClayDB 2 is to allow a developer to write an application for PostgreSQL and ClayDB can translate it for someone using SQLite or MySQL. Now obviously some features do not translate, which is a shame, but for most standard SQL requirements all of the databases should be able to work seamlessly. 

My hope is to make ClayDB much more complete than is it now and easier to implement the adapters and datadicts. I also want to make it as usable for someone learning database schema as it is to the expert database designer. That part is harder, especially when you have to meet features in the datadicts with features in the adapters. 

I've created a new branch in the Clay repo in the Clay Project's Github named claydb2. It's a clone of the master branch. I'm hoping to get this pushed out fast and merged back into master soon.

3:31 am

I didn't make TSgt this year...but at least some of those other SSgts I was hoping to make it did. All of my scores improved and I would have made it last year with this year's scores. I'm still ahead of the curve. I should make it easily next year. (I hope)

3:57 am

Yeah. I can't sleep, because TSgt result come our in 4 hours and 5 minutes. The list will be on at 0800 today. I hope you see me on the list (and a few other SSgts I know)!

I've been in the USAF for almost 10 years (since October 2002). I've heard some pretty crazy things about us. Here's the deal.

Everyone in the Air Force isn't a pilot. Surprised? Well, we're not. It's a good thing we all aren't, otherwise: there wouldn't be any working jets, no bombs ready to fall or missiles ready to fly, no fuel in the not working jets, no one would get payed, there would be no buildings or even a runway, there would be no one to provide security, no nuclear stockpile waiting for someone really dumb to force us to use them...there'd just be a bunch of "pilots" sitting around with nothing to do. We do a lot more than fly and it takes hundreds of people to fly a plane, even if most of us never leave the ground.

It's a job. It's a really cool job and a lot of times really dangerous, but it's still a job. For some of us it's not just a job though, it's something bigger than any other profession can understand. It's a family, it's a machine, it's living breathing monster waiting for the wrong person to piss it off. What other people in other professions can probably understand is eventually, maybe, it isn't a job anymore. At one point or another something clicks, either on or off, and your view changes. You either click on and suddenly it's not a job anymore, it's what you are or you click off and realize you aren't in the right place anymore. It's the best (or worst--depending on your outlook) job in the world, because few other jobs are also a service to millions of people. 

Most of us hate war. We didn't join the military to fight, we joined to keep someone else from winning (or to go to college, or to grow up, or to get away from someone/something/some place, or...whatever). I joined to see the world and, well, that hasn't worked out so well for me. I also joined to grow up and get away from what I thought was going to be a bunch of really poor decisions. We all join for different, but similar reasons. Everyone has battles in their lives, we just chose to be ready for our battles.

We aren't here because we couldn't make it somewhere else. We aren't dumb (well, most of us aren't). A lot of us are here because we didn't make it somewhere else, but that just means we didn't try as hard as we should have. We failed at something easy so we decided to right it by doing something way harder. My mom doesn't even know exactly what I do and she would probably tie me down and not let me come back if she did. 

Well, that's a few, I'm sure I'll think of a few more for another day.

2:56 am

I've decided to start blogging more and not just about my projects. We'll see how long it lasts and yes I realize I need to hurry up with the comments app...

Just read this informative blog about PHP memory usage in some frameworks and CMS'. As of writing this blog, I have an HTML comment at the bottom of the page's HTML source that tells me the execution time and peak PHP memory usage to generate any page on this web site using ClayCMS. Just right-click and click View Source (or equivalent) to check it out (last thing on the page). 

I just checked the peak PHP memory usage for the home page of this site and it says Peak PHP Memory Usage 605660 bytes. We'll round that up to 606 KB. According to the blog post I linked to, PHP uses 256 KB just to render "Hello World" on a page. I'm sure that is dependent on the machine and how it is configured, but that sounds pretty accurate. I'm not trying to brag, I just want to point out that taking the time to optimize and focusing on performance does make a difference. Granted, there are a lot of features missing right now, like blocks and things like that. Check out the blog if you haven't yet. Those memory numbers were for a basic page, nothing extra. If I go to a page like that, maybe this page, it says Peak PHP Memory Usage 502292 bytes. So if we assume a decent sized application (this is a blog page with a pager, HTML filtering, and quite a bit of SQL data) load added 100 KB, adding blocks may add 200 to 300 KB to the memory usage, maybe more maybe less. We'll say it will add 400 KB. That puts ClayCMS' peak memory usage around 1 MB, with a full featured page being generated. 

I just hope I can keep this performance scale, because so far I'm cutting the best memory usage in half. Drupal with a limited number of modules was pulling down about 30 MB of memory at peak. Once again, I'm not bragging. I know how fast and easily something can get bloated. Let's just hope ClayCMS scales from here and keep trying to make it even better.

I've been working on a PostgreSQL adapter for ClayDB. I've only used PostgreSQL once before now and that was just to run a Xaraya installation for a client years ago. I never thought there was really a difference between MySQL and PostgreSQL. Wow was I wrong. PostgreSQL has some really interesting data types, including UUID, ARRAY, and lots of geometric data types. I don't know how I could possibly utilize all of the features in PostgreSQL and maintain any sanity within the ClayDB Adapter interface. It is a really interesting database and I'd like to experiment with it a lot more.

Working on the adapters for SQLite and PostgreSQL have made me realize the down side to having a simple Database Abstraction Layer and the features we have to sacrifice for the sake of being flexible. In the future I may take that into consideration when developing a next generation ClayDB library. Fortunately, if someone doesn't care to maintain compatibility with more than one database, they can still use ClayDB as a connection manager and PDO wrapper. ClayDB doesn't require adapters to implement the Adapter interface class, so one could create an adapter specifically taylored to any database engine and disregard any needs of any other adapters.

Anyway, I'm working on the PostgreSQL adapter. Once it's finished I plan to work on a MSSQL adapter, which will be tested with the SQL Express server. I hadn't planned on supporting any closed source databases, but I'm trying to open Clay up to as large of a user base as possible. MSSQL is very low on my priority list, as I will likely never use it. It's still on my list though, so yeah.