Blog

Introducing the Passage Project

Posted by david on 29 July 2012 at 3:14 pm

A few days ago I blogged about Clay Data Transport (CDaT) and OData. After some consideration and starting the early planning stage for CDaT, I've decided to go a slightly different route. 

Passage

Passage is the new name for CDaT and is another project within the Clay Project. It will be developed as a library in its own repository on GitHub [https://github.com/clay/passage]. The README in the project repo's index directory explains it, but I'll explain it here as well. Passage is a transaction server for moving data from one source to another. The abstract of the idea is you have web sites that act as nodes that create and receive data, while another web site or server is used as a hub to connect the nodes. There can be multiple hubs, nodes that also act as hubs, and nodes grouped and connected together under groups of hubs. 

Hubs

Hubs are the router for the data to flow through the nodes and even treat other Hubs as Nodes. They track requests for data access and once data is received they pass on that data to the requestor. Data requests have to be approved by the Node that is providing the data. The Hubs use routes to transport the data based on requests from Nodes. Hubs can route data based on tiered access levels, from all data coming from a node to a single transaction reference.

Nodes

Nodes are servers that can act as clients or data sources, even both. They authenticate requests for data, whether it is a request to send or receive. Nodes can not communicate without a Hub and are not required to treat data the same way. The nodes are required to be able to place context on the data they receive and to assign a data type to data they send. 

Transactions

Each transaction between nodes is recorded by the Hub for reference. The Hub responds to sent data by sending the sender a reference transaction id and then attaches that id to data it sends down stream. Nodes can change the data and resend it, so any Node using that transaction always has the current data. Additional transactions can reference a transaction id, which places attachments to the parent transaction. The Nodes are required to provide association between the data they send or receive an any assigned transaction ids. 

Data

The data transported by Passage uses data types to provide context. The Hub has no way to identify data as anything other than its data type, it does not place context on the data. The Nodes are required to be able to identify what kind of data they are sending by assigning it a data type. The Nodes are then required to assign context to data types they receive and treat it as needed. 

A standard for identifying data within context is currently under development named OContext. The purpose of OContext is to standardize data types for transport and allowing the Node to identify what the data it receives is. The Nodes use a data dictionary to translate data types into how to use the data. Each Node can translate the data differently.

Transaction Queues

Nodes and Hubs will have the capability to queue transactions for later use. If a Hub tries unsuccessfully to push data out to a Node or if a Node is not able to send data to a Hub the data will be queued. Nodes can also use a Check-in/Check-out system for data transport. What this allows is for the Node to send data (check-in) when desired and to receive data (check-out) when able. The Hub will know which transactions have occurred and duplicate transactions are avoided. If there is no data in a queue, the Hub can route a request for a Node to check-in transactions for another Node to receive.

 

That is the basic idea, without going too much into implementation. It is a very flexible data transaction system. Passage is also going to be built so it does not have prerequisite libraries from other platforms. It will be modular. The idea is I can use Passage in Clay and someone else can use Passage in the Zend Framework or in Drupal, without causing any crossover. Many of the features will be built into modules, but it will also allow a Bring Your Own Component approach, so someone using Zend can use Zend libraries in place of a built-in component within Passage.

Comments

Log in to comment

No comments yet!