Caching Data Mapper

[bubblenut]

Hi, I've been working on a caching data mapper for a week or so now and think I have something pretty sweet now. It's not all implemented yet but I will add to it as and when the bits get completed (very soon hopefully).

I am going to be using this as the basis for a talk I'm hoping to give at the PHPLondon user group's next meeting. I have never done a talk for my peers before and I'm a little scared (read: absolutely terrified). I would really appreciate any comments on what I've got so far.

The classes included are
ClassLoader Unrelated to the DataMap stuff really, it's just convenient for the factory.
DataMap This is a DataMap factory, it determines if we have an appropriate datamap and returns it.
Persistable This is an interface which all classes which are to be persisted through this system must implement.
DataMap_Common This is the abstract super class which all DataMap implementations must extend. It contains the methods to actually get the data.
DataMap_Default This is a default fallback DataMap for cases where a specific DataMap is not necesarry.
DataMap_Recipe This is an example implementation for fetching instances of the Recipe class.
Recipe This is an example Persistable class which has it's own DataMap implementation.
Ingredient This is an example Persistable class which uses the default DataMap implementation.
View_Recipe This is an example Persistable which is dependent on other Persistable objects (Not Yet Implemented)

Here is an example usage

<?php
define('ROOT_PATH', '/path/to/root');
require_once(ROOT_PATH . '/ClassLoader.php');

// Set up the PDO object
$db = new PDO('mysql:host=localhost;dbname=test', 'username', 'password');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

// Set up the Memcache object
$cache = new Memcache();
$cache->addServer('localhost', 11211);

// Set up the DataMap object
$dm = new DataMap($db, $cache);

// Get a Recipe and update it
$recipe = $dm->map('Recipe')->get(1);
$recipe->name = 'New Name';
$dm->map('Recipe')->save($recipe);

// Get an Ingredient and delete it
$ingredient = $dm->map('Ingredient')->get(1);
$dm->map('Ingredient')->delete($ingredient);

[edit]
I have the slides for the talk I'm going to give on this now. I tried to upload them but there's a 100k limit. If anyone has the time to have a look over them and give me their thoughts please pm me and I'll mail it to you.
[/edit]

[Shrike]

This is pretty cool stuff. I guess the Memcache extension acts like an object registry?

A couple of things I wondered about:

Creating new instances of objects. Can I say

$recipe = new Recipe();
$recipe->name = 'Paul\'s nice recipe';
$primarykey = $dm->map( 'Recipe' )->save( $recipe );

Also it'd be nice after saving objects to the cache to be able to save them all in one fell swoop. Seems to me that there would be a nice opportunity for some optimisation with PDO's prepared statements.

Good luck with the talk 🙂

[bubblenut]

Shrike wrote:

This is pretty cool stuff. I guess the Memcache extension acts like an object registry?

A couple of things I wondered about:

Creating new instances of objects. Can I say

$recipe = new Recipe();
$recipe->name = 'Paul\'s nice recipe';
$primarykey = $dm->map( 'Recipe' )->save( $recipe );

Yes, deffintaly. If the object doesn't have an id set it will be inserted and the id field in the object you created will be updated with the last insert id before it's cached.
edit: Sorry, I missread. No, save doesn't return the id, it's set to the $recipe object, I suppose save could return the id as well.

Shrike wrote:

Also it'd be nice after saving objects to the cache to be able to save them all in one fell swoop.

Do you mean

$dm->map('Recipe')->save(array($recipe1, $recipe2, $recipe3));

or more

$dm->map('Recipe')->save($recipe1); // Only written localy
// some stuff
$dm->map('Recipe')->save($recipe2); // Only written locally
// some more stuff
$dm->map('Recipe')->save($recipe3); // Only written locally
// yet more stuff
$dm->map('Recipe')->flush();  // Flushed to memcache and db

Neither are implemented but both could be pretty cool. The first way I'd probably just add a saveAll method, the second would require quite a bit of refactoring I think. I like the idea though.

Shrike wrote:

Seems to me that there would be a nice opportunity for some optimisation with PDO's prepared statements.

I am using prepared statements, if you look in DataMap_Recipe all three queries are prepared in the constructor and in DataMap_Default the select is prepared in the constructor and then the update and insert are prepared when they're first needed (as we have to query the database for the table structure in order to build the queries). Although, I've recently discovered that preparing bad statements fails, so it must do a lookup on the database itself, for this reason I'm thinking about leaving all prepare's untill they're first needed. Any thoughts?
edit:constructors ... twice!

Shrike wrote:

Good luck with the talk 🙂

Thanks, are you going to be able to make it?

[Shrike]

I mean the latter version - flush was the method name that came to mind for me too 🙂

What I meant about prepared statements is that once the statement is built the database has an optimisation strategy for succcessive queries using that statement, and so are much faster - batch operations have a big performance increase.

I think leaving the statements until they are needed fits in with JIT mentality which is usually a good thing in web apps.

In a project I'm working on currently I've used Active Record rather than data mappers, for simplicities sake. To generate the insert/update queries I've used a bit of a hack with the visibility of properties. It looks something like this:

// ModelObject is basically a dumb data container with some accessors
abstract class ActiveRecord extends ModelObject
{
  static private $db; // ADODB
  public function save( $table, $key )
  {
    /*
    This is the magic bit, when called from a subclass get_object vars
    only returns public and protected properties of the subclass, I generate
    the SQL statements from this array
    */
    $params = get_object_vars( $this );
    if( $this->getPrimary() )
    {
      return $this->update( $table, $params, $key );
    }
    else
    {
      return $this->insert( $table, $params );
    }
  }
}
class Recipe extends ActiveRecord
{
  const TABLE = 'Recipes';
  const PRIMARY_KEY = 'id';
  protected
    $id,
    $value;
}
$recipe = new Recipe();
$recipe->save( Recipe::TABLE, Recipe::PRIMARY_KEY );
// Generated query is INSERT INTO Recipes VALUES( '$id', '$value' ));

Ideally ActiveRecord should be able to figure out what the child class is automatically but I don't think there is a way. Also it only works for 1 object = 1 table, which quickly became too restrictive so I end up overriding ActiveRecord::save() in some cases.

When is the next PHPLondon meeting?

[bubblenut]

With regard to your comment on prepared statements. I'm not quite sure I get the point, are you saying the way I'm preparing the statements isn't going to allow for the optimisations? Do I have to prepare the statement and then run the queries all at the same time to get the benifit?

Ah, your implementation's interesting. I took it from the other side and said that all the fields on the table are the one's which need to be extracted and stored from the class. This means that I don't have to worry about the visibility of my parameters but, on the other hand, in order to generalise the data map (as with DataMap_Default) I have to grab the database schema before I update/insert. Now, with the caching it's easy, I just define everything in the magic method sleep and let the memcache module serialize the object. I was thinking about using the sleep method myself for the DB side of things so that I define which parameters should be persistant in only one place. Haven't done yet though.

I'm on holiday now so I'm most of the way through a bottle of wine at the moment. This is my excuse for any incoherence in this post and also my reason for not replying to later posts.

The next PHPLondon meeting is listed here

[bubblenut]

Ahh, that was the other thing I was thinking about. How would I garuntee that things get flushed if the user forgets to call the flush method (or should I even, maybe they want them to dissapear if they don't call flush?) Would __destruct be a good place for this kind of cleanup? This would bring in the question, as mentioned above, "does the user want to do this?". I'm guessing I'd then have to have some way of allowing the user to set the way they want their writes to be flushed.

To follow PDO's lead maybe

// To avoid flushing untill desctruct (unless called explicitly)
$dm->setAttribute(DataMap::ATTR_FLUSHMODE, DataMap::FLUSHMODE_ONDESTRUCT); 
// or to automatically flush on every call
$dm->setAttribute(DataMap::ATTR_FLUSHMODE, DataMap::FLUSHMODE_ONCALL);
// or only when the flush method is called explicitly
$dm->setAttribute(DataMap::ATTR_FLUSHMODE, DataMap::FLUSHMODE_ONFLUSH);
// and, of course, the one you put in when you get fired ;)
$dm->setAttribute(DataMap::ATTR_FLUSHMODE, DataMap::FLUSHMODE_NEVER);

Your ideas are really starting to make me think that this could be a pretty cool project in itself. It would need a lot of refactoring to make it usable in the general case but I think it could be done.

[Shrike]

From the manual on PDO prepared statements:

The query only needs to be parsed (or prepared) once, but can be executed multiple times with the same or different parameters. When the query is prepared, the database will analyze, compile and optimize it's plan for executing the query. For complex queries this process can take up enough time that it will noticeably slow down your application if you need to repeat the same query many times with different parameters. By using a prepared statement you avoid repeating the analyze/compile/optimize cycle. In short, prepared statements use fewer resources and thus run faster.

I was thinking along the lines that if you have a stack of objects, you'd only need to build the prepared statement once and could reuse it for each object on the stack. I think at the moment you'll have one select, one insert etc for each object. The question would be how to share the prepared statement objects 🙂

I'd like to make it to the PHPLondon meeting. Marcus Baker is a bit of a PHP guru on Sitepoint. Should be doable on the train after work. Now I need to go nurse some bruises, my company went karting last night and damn those karts hurt :p

[dream_scape]

bubblenut wrote:

$dm->map('Recipe')->save($recipe1); // Only written localy
// some stuff
$dm->map('Recipe')->save($recipe2); // Only written locally
// some more stuff
$dm->map('Recipe')->save($recipe3); // Only written locally
// yet more stuff
$dm->map('Recipe')->flush();  // Flushed to memcache and db

I'm gonna be a bit nit-picky here, but not sure I like using a method called flush() to do this.

When I hear "flush" it does mean flow, but I get the impression something is being cleared out; flush the toilet, flush the drain, flush the buffer, flush the IP table, etc... "Flushed to" seems less common usage to me, and wouldn't be my first thought. When I see $dm->map('Recipe')->flush();, my first though is that is not sending something to memcache, but is flushing out Recipe related memcaches, i.e, emptying them.

Maybe modeling more on the traditional transaction model might work better? Something like $dm->map('Recipe')->commit();

Just a thought.

[Shrike]

I think commit is good and illustrative of the action of saving objects, as it implies finality (and of course is in keeping with transactions).

I don't think it's nitpicking, it's important to have a good domain language. Just as save() is the right method name for updating and creating objects.

[bubblenut]

Yea, agreed on commit instead of flush. Also leaves flush open for implemting something which could flush portions of the cache. For example, if the Recipe cache got flushed we could flush all Recipe cache items without having to flush the whole thing. Not sure how that could be implemented but it would certainly be usefull.

I have added the slides for my presentation to my first post. If anyone has the time to have a read through them and comment I would really apprecieate it.

[Shrike]

Are you sure you added them? Can't find anything 🙂

[bubblenut]

Ahh, no, I added a little comment to the first message. I can't upload the slides so I'll have to mail them to you.

As it happens I've pretty much taken this code out of the slides now. I'm still going to offer it online by means of an example but it was too much to go through in a talk in the end.