Blog

A few days ago I blogged about Clay Data Transport (CDaT) and OData. After some consideration and starting the early planning stage for CDaT, I've decided to go a slightly different route. 

Passage

Passage is the new name for CDaT and is another project within the Clay Project. It will be developed as a library in its own repository on GitHub [https://github.com/clay/passage]. The README in the project repo's index directory explains it, but I'll explain it here as well. Passage is a transaction server for moving data from one source to another. The abstract of the idea is you have web sites that act as nodes that create and receive data, while another web site or server is used as a hub to connect the nodes. There can be multiple hubs, nodes that also act as hubs, and nodes grouped and connected together under groups of hubs. 

Hubs

Hubs are the router for the data to flow through the nodes and even treat other Hubs as Nodes. They track requests for data access and once data is received they pass on that data to the requestor. Data requests have to be approved by the Node that is providing the data. The Hubs use routes to transport the data based on requests from Nodes. Hubs can route data based on tiered access levels, from all data coming from a node to a single transaction reference.

Nodes

Nodes are servers that can act as clients or data sources, even both. They authenticate requests for data, whether it is a request to send or receive. Nodes can not communicate without a Hub and are not required to treat data the same way. The nodes are required to be able to place context on the data they receive and to assign a data type to data they send. 

Transactions

Each transaction between nodes is recorded by the Hub for reference. The Hub responds to sent data by sending the sender a reference transaction id and then attaches that id to data it sends down stream. Nodes can change the data and resend it, so any Node using that transaction always has the current data. Additional transactions can reference a transaction id, which places attachments to the parent transaction. The Nodes are required to provide association between the data they send or receive an any assigned transaction ids. 

Data

The data transported by Passage uses data types to provide context. The Hub has no way to identify data as anything other than its data type, it does not place context on the data. The Nodes are required to be able to identify what kind of data they are sending by assigning it a data type. The Nodes are then required to assign context to data types they receive and treat it as needed. 

A standard for identifying data within context is currently under development named OContext. The purpose of OContext is to standardize data types for transport and allowing the Node to identify what the data it receives is. The Nodes use a data dictionary to translate data types into how to use the data. Each Node can translate the data differently.

Transaction Queues

Nodes and Hubs will have the capability to queue transactions for later use. If a Hub tries unsuccessfully to push data out to a Node or if a Node is not able to send data to a Hub the data will be queued. Nodes can also use a Check-in/Check-out system for data transport. What this allows is for the Node to send data (check-in) when desired and to receive data (check-out) when able. The Hub will know which transactions have occurred and duplicate transactions are avoided. If there is no data in a queue, the Hub can route a request for a Node to check-in transactions for another Node to receive.

 

That is the basic idea, without going too much into implementation. It is a very flexible data transaction system. Passage is also going to be built so it does not have prerequisite libraries from other platforms. It will be modular. The idea is I can use Passage in Clay and someone else can use Passage in the Zend Framework or in Drupal, without causing any crossover. Many of the features will be built into modules, but it will also allow a Bring Your Own Component approach, so someone using Zend can use Zend libraries in place of a built-in component within Passage.

I've been looking for better ways to manage the development of Clay, ways that may lure more developers to us. Based on the advice from Cal Evans' blog [http://blog.calevans.com/], I've decided to give phpcloud.com a try. It's a cloud environment provided by Zend for developers and it's free, sounds good so far.

I'm setting it up now and will post an update when I've tried it out a little

Update: PHPCloud.com looks to be an excellent service, but it still has some issues that are hard for me to work around. It is currently a "Technology Preview". I'll keep the account, as a do really see a lot of promise in it, but for now I'll have to try something else.

"The Open Data Protocol (OData) is a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today. OData does this by applying and building upon Web technologies such as HTTPAtom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores." - odata.org 

CDaT (Clay Data Transport) is a layer/library I am working on that supports the OData standard. It is intended to be used to provide a data services interface from any application within Clay and will accept data from other services using the standard. Each application will be able to use native privileges to determine which type of data should be exposed and accepted. CDaT will also be a prototyping tool for creating the OContext standard for data translations across service platforms.

 Update: CDat has been officially named Passage. More info to come.

 

A new kickstarter.com project named OUYA has been crowd funded to $3M in just 2 days. The project's goal was to raise $950,000 in 30 days. OUYA is set to use Android for its operating system, but promises to be open and allow any of the software and hardware to be hacked. It's not open source, though. It seems like a pretty cool idea (not the first) and the kickstarter crowd has embraced it with over 25,000 backers in the first 2 days. The first day it hit the $2M mark, more than doubling its funding goal for the month.

I can see this having potential if it has support for Netflix and other premium services. I would rather see a modular console that is open and open source, but I think this is a good step in that direction. I think we need an open source console operating system, dedicated to game development. That would get my backing.

The goal for ClayDB 2 has been to expand the Data Dictionary capabilities and build a better way to create and manipulate database tables on any supported database. I've been prototyping the PDO MySQL Data Dictionary within ClayDB to reach that goal. The Data Dictionary has been expanded quite a bit and allows a range of table manipulations. It is not fully tested, so I haven't moved on the other databases' datadicts yet. I have been comparing SQLite, PostgreSQL, MSSQL, and MySQL while developing the prototype in MySQL. All of them are very different, so it has been a balancing act. It looks like PostgreSQL will miss the most functionality, mainly because there is so much there that doesn't translate to the other databases. I've been working with the dummy application to test ClayDB 2 and hope to soon make the necessary application changes across the board, followed by a merge back into the master branch.

Like I said before, ClayDB 2 has been all about improving the datadict classes, but I couldn't resist adding a feature to the adapters. ClayDB 1 only returned arrays for result sets. In ClayDB 2 I've added a getObject() method that returns either an object or an array of objects. I don't plan to change a lot more than that on the adapter side, although I have improved the adapters' abstraction and interface. I may make a few other changes, but I do not plan to mess with compatibility on the adapter side with this version change. The datadict changes will be enough work to bring everything up to date. I have thought about some ideas for ClayDB 3 though.

I think the main feature it is missing, as far as popular features go, is a model system. Now I don't believe ClayDB should be responsible for ORM or anything that complex. Its purpose is to make connections and queries to varying databases easier and it does that fairly well. Instead, I'm considering offering an abstract class that allows someone to implement models and, with some expansion, object-relational mapping. The class would allow someone to implement traditional models (at the framework level) or allow you to work with an object model from within an application API. Now within ClayCMS I have been planning to migrate much of the ClayCMS libraries to application libraries, so there's a little push and pull there. I believe the framework level libraries should be used to implement features within applications, instead of carrying the load themselves. So, as of today at least, ClayDB 2 will focus on improving the Data Dictionary, while ClayDB 3 will likely focus more on data handling and moving more toward object models.

11:59 pmCycle of Change

To every thing there is a season, and a time to every purpose under the heaven:

A time to be born, and a time to die; a time to plant, and a time to pluck up that which is planted;

A time to kill, and a time to heal; a time to break down, and a time to build up;

A time to weep, and a time to laugh; a time to mourn, and a time to dance;

A time to cast away stones, and a time to gather stones together; a time to embrace, and a time to refrain from embracing;

A time to get, and a time to lose; a time to keep, and a time to cast away;

A time to rend, and a time to sew; a time to keep silence, and a time to speak;

A time to love, and a time to hate; a time of war, and a time of peace.

Ecclesiastes 3:1-8 KJV

I haven't read the Bible as much lately as I would like, but this chapter has been on my mine a lot. I have to admit it was more of verse 8 than the rest, but all of it is equally as relevant at any given time. Everything we go through in life is a cycle of change. It's not an exact science, it's difficult to know how long you'll be in one of those times. Of course, we're not just constantly cycling through each one of those individually, they are all intertwined and tangled around one another. Lately it feels like I am somewhere between a time to gather stones together and a time to cast stones.

I've come to realize lately that if I remember each one of those, I will hopefully know which time it is. There is a lot that can be written of those verses, but I will save some of the rest for a different day.

The ClayDB 2 upgrade began because I was trying to finish the Privilege system in ClayCMS. I decided to implement an SQLite adapter for ClayDB, while working on the Privileges database schema, and realized the shortfalls of ClayDB. Here's what I'm come up with for the Privilege system (which I'm still working on):

  • Uniquely scoped privileges
    • Each privilege in the system is required to be unique, but can be assigned multiple times across it's scope to multiple Roles
  • Each application has complete control over its privileges
    • Using Privilege libraries within each application, the application determine what a privilege means and what it allows
    • Applications choose their own naming system and are not tied to a set tree of privilege types
    • Explicit and user created privileges are allowed, as long as an application provides support for determining scope for them
  • An optional base set of privileges will be provided for applications to use
    • This set of privileges will be part of the Apps application and can be extended/scoped by any application
      • Intended to be used for developmental purposes, as it will involve more processing, but can be used regardless
  • Role triggers will be provided that allow Roles for Owner, Shared, Public, and Private
  • An experimental Entities will be included later on that will be a mix between Roles and Users. 
    • Entities are for applications requiring users to be split up in to organizational groups, rather than roles within a single group

ClayDB was one of the first libraries I wrote for Clay and is the only one left that hasn't been completely rewritten. As well as it works, it's still lacking in several areas and is overall incomplete. I've been working on adding new adapters and datadicts to support more databases. What I came to realize was ClayDB could be a lot better. 

Now before I get into the details, this is only about ClayDB. I have not started Clay 2 (yet). That is quite a while away. ClayDB is a separate library that comes with Clay. ClayDB in Clay 2 will be based on Clay 2's roadmap requirements and could still be ClayDB 2 or ClayDB 5, it's hard to tell.

ClayDB 2 is going to be geared more toward flexibility for different databases and work a lot better on the datadict side (table manipulation). MySQL is the most popular database Clay caters to, like now, but will allow verbage for developers working with other supported databases. The goal with ClayDB 2 is to allow a developer to write an application for PostgreSQL and ClayDB can translate it for someone using SQLite or MySQL. Now obviously some features do not translate, which is a shame, but for most standard SQL requirements all of the databases should be able to work seamlessly. 

My hope is to make ClayDB much more complete than is it now and easier to implement the adapters and datadicts. I also want to make it as usable for someone learning database schema as it is to the expert database designer. That part is harder, especially when you have to meet features in the datadicts with features in the adapters. 

I've created a new branch in the Clay repo in the Clay Project's Github named claydb2. It's a clone of the master branch. I'm hoping to get this pushed out fast and merged back into master soon.

3:31 am

I didn't make TSgt this year...but at least some of those other SSgts I was hoping to make it did. All of my scores improved and I would have made it last year with this year's scores. I'm still ahead of the curve. I should make it easily next year. (I hope)

3:57 am

Yeah. I can't sleep, because TSgt result come our in 4 hours and 5 minutes. The list will be on http://www.my.af.mil at 0800 today. I hope you see me on the list (and a few other SSgts I know)!

I've been in the USAF for almost 10 years (since October 2002). I've heard some pretty crazy things about us. Here's the deal.

Everyone in the Air Force isn't a pilot. Surprised? Well, we're not. It's a good thing we all aren't, otherwise: there wouldn't be any working jets, no bombs ready to fall or missiles ready to fly, no fuel in the not working jets, no one would get payed, there would be no buildings or even a runway, there would be no one to provide security, no nuclear stockpile waiting for someone really dumb to force us to use them...there'd just be a bunch of "pilots" sitting around with nothing to do. We do a lot more than fly and it takes hundreds of people to fly a plane, even if most of us never leave the ground.

It's a job. It's a really cool job and a lot of times really dangerous, but it's still a job. For some of us it's not just a job though, it's something bigger than any other profession can understand. It's a family, it's a machine, it's living breathing monster waiting for the wrong person to piss it off. What other people in other professions can probably understand is eventually, maybe, it isn't a job anymore. At one point or another something clicks, either on or off, and your view changes. You either click on and suddenly it's not a job anymore, it's what you are or you click off and realize you aren't in the right place anymore. It's the best (or worst--depending on your outlook) job in the world, because few other jobs are also a service to millions of people. 

Most of us hate war. We didn't join the military to fight, we joined to keep someone else from winning (or to go to college, or to grow up, or to get away from someone/something/some place, or...whatever). I joined to see the world and, well, that hasn't worked out so well for me. I also joined to grow up and get away from what I thought was going to be a bunch of really poor decisions. We all join for different, but similar reasons. Everyone has battles in their lives, we just chose to be ready for our battles.

We aren't here because we couldn't make it somewhere else. We aren't dumb (well, most of us aren't). A lot of us are here because we didn't make it somewhere else, but that just means we didn't try as hard as we should have. We failed at something easy so we decided to right it by doing something way harder. My mom doesn't even know exactly what I do and she would probably tie me down and not let me come back if she did. 

Well, that's a few, I'm sure I'll think of a few more for another day.

2:56 am

I've decided to start blogging more and not just about my projects. We'll see how long it lasts and yes I realize I need to hurry up with the comments app...