Building a generic ecommerce system

SeenGee

Hi all,

I'm building my first really big web application, a generic ecommerce solution for my company. Offerring much the same functionality as OScommerce without the need to work around the OScommerce core which I find bulky and difficult to extend. At the moment I am still very much at the planning stage so I am interested in any ideas that people have or experience of similar projects. Essentially what I am creating is the core to an ecommerce system, in its "core" state it will still be a functional ecommerce site on the front and back end, however as clients require we will create new and bespoke modules for specific needs. I am intending on making the site class based where possible and also abstracting where possible. This wouldnt be necessary for the database as the system will only need to run with mySQL but I intend to abstract - for example - the payment process.

In terms of the front end i'm wrestling with the whether to template debate (good article here ), i've pretty much decided on using my own lightweight template class like in the article as Smarty et al does seem overkill. With regard to the wider system framework i've looked briefly at some application frameworks like Kinky and Sourdough but i've never worked with systems like this and i'm not sure how well they would work in this context.

As i say there's a lot to think about, seems better now than when the whole project has collapsed around me.

As i said, any ideas, reflections,warnings will all be appreciated.

Thanks.

btw. will (probably) be running on mysql 3.23.58 and php 4.3.2

MarkR

Having already done this, I can make some suggestions:

Keep it simple
Keep it simple
Reread and pay close attention to points 1 and 2

Using a template system is an extremely good idea. We use Smarty, and I recommend you do so too unless you have a good reason not to.

Target only the most recent version of PHP if at all possible. Don't target more than one major version of PHP. Use an ORM* or other database tool to write queries for you, and/or use prepared queries as necessary. Reuse as much code as possible (within your own app). This will reduce the amount of database persistence code required and increase its correctness.

Try to identify where things are similar and put them into the same class (/ table if using a RDBMS** via an ORM, as we are). The fewer entities the better. We have quite a lot of features in our system, and have 12 tables at the moment. Oscommerce has 48.

A design error made by a lot of shops (including oscommerce) is to consider a basket a different type of entity from an order. A basket is really just an order which hasn't been placed yet.

Oscommerce tries to have an awful lot of functionality - much of it is done very badly, wasn't thought out well and doesn't work. The implementation is terrible, there is a lot of repeated code and it is thoroughly unusable.

Our shop has a lot less functionality than Oscommerce, but is a lot cleaner, more maintainable, more reliable, more secure, prettier and more correct.

Variables / variants add a great deal of complexity - therefore, don't add them unless you REALLY have to. If you do, do at least a couple of days analysis with your other developers / customers beforehand, to study the problem in-depth. Particularly SKUs*** and stock control, but also other issues.

If at all possible, have an automated test tool. We've only started doing this recently.

Pay a great deal of attention to error handling - ideally run all the time with E_ALL and have an error handler that shouts loudly and aborts the page even if an E_NOTICE happens. No errors, warnings or even notices should be present in a properly functioning app.

Test it well on a variety of browsers. Try not to to rely on Javascript, at least on the front-end.

Trust no-one, revalidate every possible tiny bit of data on every form POST.

PSP**** integration is one of the most painful parts of it. Aim to integrate at least one PSP from day one - don't add it as an afterthought. Bear in mind the different types of integration PSPs do.

Peace and good luck.

Mark

Object-relational Mapper
Relational Database Management System (often known as a "Database")
Stock-keeping unit. The code which merchants identify individual lines by.
*** Payment service provider. Paypal, Worldpay, Protx etc.

MarkR

Oh yes, some more things to think about:

Be mindful of localisation and immediately make a decision as to how much localisation you intend to support. Are you planning to have it single-language throughout? Storing product information in multiple languages? Storing text labels in multiple languages? Is the management are multilingual?
Be mindful of encodings and make a decision early on what encoding to use / support and stick to this. Consider tool support here.
Decide up-front how you're going to support variables / variants (If at all). Do a thorough design. It probably won't be right but at least you'll understand the issues a bit better.
Decide how much CMS functionality you want in it (page editing, versioning, multiuser back-end?) and stick to it.

In all of the above, keep in mind the "keep it simple" idea. Unless you have a VERY good reason, choose the easiest answer to each of the above.

It's going to be impossible to make an application which is all things to all people, so you must restrict its scope as much as possible. If in doubt, leave it out!

Mark

SeenGee

hi MarkR, first of all thanks for all the great input.

I am lucky in that I can keep the scope simple for this project in as much as it will always exist on the company server with known PHP/MySQL versions and our system would never be white-labelled for anyone else. This does allow me to work in a controlled and known environment.

its good that you point out localisation as i need to consider not only the company's physical whereabouts but also where they intend to sell to. Error handling is another good one, i've heard good things about a PEAR error handling class that i intend to look into.

a question about Smarty though, would it require a developer to implement as we intend to create a system that our designers can piece together. Also, what advantages did you see in using it over a purely PHP driven template system given that you are simply replacing PHP code with equivalent code in a different language.

thanks again.

dream_scape

Be sure to have alot of time on your hands 😉 I've been developing a system on and off for about 2 years and am just now starting to get to where I want in terms of code and application design.

If this will be something you'll be using as a base to develop shops for others, there are some key things I think take precedence over things that might be more important if it was just solely for you... well one thing mainly:
- make it flexible

Flexibility is key. What do I mean by this? Make is so that the core application is not bogged down in features. Delegate features out to plugins or addons. If a feature likely won't be used by at least 75-80% of customers (or more), don't put it directly in the core. Make it easy(er) to customize and add functionality without having to funk with core code. Meaning a strong plugin API and possibly the ability to override certain parts of the core in a "user" space. This also makes rolling out core updates a zillion times easier, since theoretically, every install, no matter how customized, will have exactly the same core code.

For hierarchial structures like categories, I'd say use the Modified Preorder Tree Traversal (MPTT) method if you can. You can pull out a whole tree with a single query this way. The other major option is the common Adjacency method (as seen in most php ecommerce applications), but with this method, SQL queries quickly escalate (the more categories = more queries to build the tree).

MarkR wrote:
Try to identify where things are similar and put them into the same class (/ table if using a RDBMS** via an ORM, as we are). The fewer entities the better. We have quite a lot of features in our system, and have 12 tables at the moment. Oscommerce has 48.

How are fewer entities better? Unless you are pulling data from one table at a time, one row at a time, it doesn't really make much difference. Most RBDMS are optimized to perform well up to 4 or 5 joins, which is probably about the max you'd do anyways on a single query. Depending on what level of normalization you are going for, 48 tables for an eCommerce application is nothing.

This is one reason I tend to stay away from most PHP ORM solutions, such as Propel. They escalate the number of SQL queries in table relationships, performing one query for each related table for each row, instead of pulling it all in one query and then splitting it out into objects. The n-query method is alright for some applications, but you definitely don't want that in an eCommerce system.

MarkR wrote:
A design error made by a lot of shops (including oscommerce) is to consider a basket a different type of entity from an order. A basket is really just an order which hasn't been placed yet.

Couldn't agree with this more. The way I do it is a parent order class with 2 children, an order basket and an order database class. Though I use the order basket through the checkout process until the order is placed in the database and then from there when the order is accessed use the order database, naturally, one could also delegate out a 3rd child, used during the checkout process. Or take another approach entirely.

MarkR wrote:
Variables / variants add a great deal of complexity - therefore, don't add them unless you REALLY have to. If you do, do at least a couple of days analysis with your other developers / customers beforehand, to study the problem in-depth. Particularly SKUs*** and stock control, but also other issues.

I think that alot of applications have the product options/variants model wrong. They kind of see them as being something optional and almost external to the product, which leads to overly complex application logic to account for when they might occur. It's also a pretty unrealistic model, IMO, and doesn't do a good job of representing what the makeup of a product looks like in reality.

This is one area where I took my cue from ElasticPath, because after much though, I think they've got it right, or much better than other systems. The model they use says that all products have at least one "variation". There is "generic" product and description information that is common to all variations, and then items like price, sku, stock, weight, etc are part of each variation. If you think about this, this is exactly true for every product under the sun.

This approach does complicate the SQL a little bit (not exactly for the faint of heart), but greatly simplifies the application logic as all products follow the same model. Not only that but is also gives greater flexibility over "multi-variation" products; you can either choose to display them as separate products, or as a single product with variants/options; but the basic product model allows for either one and doesn't force your hand to one or the other.

You can go another step further as well and add more flexibility by adding information that might be specific to a product "type". ElasticPath does this in the database with what they refer to as "attributes" I think (which are not like attributes in oscommerce or zen cart). Here I have chosen to have a product plugin API where you can create product plugins that add additional functionality, including adding and displaying additional info, for specific product "types" or product classes as I've called them.

And then again, doing something like this [extending functionality with plugins], allows a basic and relatively simple product model that can be easily expanded upon. For example a site might sell Books, CDs, and DVDs; all of which are sort of similar, yet all are quite different in terms of what you might want to do with them. So you create a plugin for each one to facilitate that; or you might go the database route and create "attribute" sets for books, for cds, and for dvds. But either way, you keep the product model relatively simple and lightweight, only having additional information exactly where it is needed (as opposed to having it there even if it isn't needed).

MarkR wrote:
PSP**** integration is one of the most painful parts of it. Aim to integrate at least one PSP from day one - don't add it as an afterthought. Bear in mind the different types of integration PSPs do.

I'm not sure I agree. I think all PSP integration can be basically boiled down to a few types. You've got your advanced or transparent integration. And you've got you're 3rd party offsite integration, which could be split into a couple of types, depending on how the system notifies you of the response. If you abstract the payment system to a payment plugin API, then you can write hooks in appropriate places for the different integration types (among other things a payment plugin might want to do), and it will or should be relatively easy to write a plugin for any PSP. Getting a good plugin API would be the hard part, since you'll naturally miss some things and will be adding them in as the system evolves.

MarkR

dream.scape wrote:
How are fewer entities better? Unless you are pulling data from one table at a time, one row at a time, it doesn't really make much difference.

Fewer entities are better because you then have to write less code. Less code contains fewer bugs and is easier to write.

All of your optimisation ideas seem to be about runtime performance - which is fine, but personally I find that developer performance is MUCH more important than runtime important.

Therefore my stress to keep things simple.

Runtime performance isn't important enough to worry about most of the time, therefore, don't.

MPTT is a very interesting technique, but I am not about to implement it.

This is one reason I tend to stay away from most PHP ORM solutions, such as Propel. They escalate the number of SQL queries in table relationships, performing one query for each related table for each row, instead of pulling it all in one query and then splitting it out into objects.

But that's just a runtime performance optimisation. As I said, I don't care about runtime performance, and I'll gladly sacrifice a bit for an easier to develop / maintain application.

The n-query method is alright for some applications, but you definitely don't want that in an eCommerce system.

Our ecommerce system uses an ORM, which means that we often have to use the "n-query" method.

True there are a few places where I've used a conventional join instead to improve performance (like joining the catalogue_item and catalogue_item_language tables), but mostly it's ok.

I think that alot of applications have the product options/variants model wrong. They kind of see them as being something optional and almost external to the product, which leads to overly complex application logic to account for when they might occur. It's also a pretty unrealistic model, IMO, and doesn't do a good job of representing what the makeup of a product looks like in reality.

I agree. That's why we don't support variants - because they are complicated. But mostly because it's difficult to explain how they work to Merchants.

You can use a model where every product has at least one variant, and your attributes / variants have the apropriate cross-references etc, but it simply adds complexity, and merchants won't understand it.

Oscommerce showed, based on its incredibly complicated yet totally unusable / non-scalable variants configuration screen, how easy it is to get this wrong.

I'm not sure I agree. I think all PSP integration can be basically boiled down to a few types. You've got your advanced or transparent integration. And you've got you're 3rd party offsite integration, which could be split into a couple of types, depending on how the system notifies you of the response.

Yes, of course. But if you didn't plan it this way, you could run into trouble. Oscommerce is terrible with PSP integration, largely because it doesn't allocate the order ID until it's paid for.

If you abstract the payment system to a payment plugin API,

Which incidentally we do 🙂

We currently have about 4-5 PSPs in there.

then you can write hooks in appropriate places for the different integration types (among other things a payment plugin might want to do), and it will or should be relatively easy to write a plugin for any PSP.

What's happened is we've gradually improved the payment plugin API to add more hooks for more types of payment.

Getting a good plugin API would be the hard part, since you'll naturally miss some things and will be adding them in as the system evolves.

But that's a good thing.

For example, when we started out we didn't anticipate using PSPs with Preauth / repeat capability, but now several of our PSP integrations are using it and our merchants use it.

Mark

dream_scape

Mark, if you don't exactly understand why query escalation is a bad thing, you might want to get together with some database gurus and have some talks. Their perspectives can be very enlightening 😉 Avoiding escalating queries is not a runtime "performance optimization". It is just good coding practice for many.

PS. I did not say don't use ORM. But there is no reason an ORM system cannot do regular queries and then spit the data apart into their objects; unfortunately, many of the pre-built PHP ORM systems don't do this, and at this point they just duck out and take the easy road (that is the road easier on them) and escalate the queries, which leads to incredible inefficient applications that can quickly bring the database on its knees if you're not careful.

MarkR

dream.scape wrote:
Mark, if you don't exactly understand why query escalation is a bad thing, you might want to get together with some database gurus and have some talks. Their perspectives can be very enlightening 😉 Avoiding escalating queries is not a runtime "performance optimization". It is just good coding practice for many.

It's a trade-off. You prefer to make a more complex application that performs better, which is fine. But I prefer an easier to maintain application which is probably correct.

There are some cases where it does blow up into a fair number of queries, this is annoying, however it's not that bad.

Specifically, I have very few cases where "query escalation" as you put it, actually happens in practice.

One case which is a problem at the moment is my search page, which is criminally inefficient, but I don't have a better solution at present.

PS. I did not say don't use ORM. But there is no reason an ORM system cannot do regular queries and then spit the data apart into their objects; unfortunately, many of the pre-built PHP ORM systems don't do this, and at this point they just duck out and take the easy road (that is the road easier on them) and escalate the queries, which leads to incredible inefficient applications that can quickly bring the database on its knees if you're not careful.

The ORM we're using is too braindead to do either of these things. It neither does "query escalation", nor supports joins properly. Basically it is just to speed up the simple operations (I mean in terms of developer time NOT runtime performance).

I'm not very happy with our ORM, but it's difficult to change it retrospectively.

For the commonly used operations which lend themselves to joins, I actually produce custom queries. It's not a very strict ORM model, as we sometimes create objects which contain the result of a join (hence no 1:1 mapping).

Most cases where I can fix things by making a join are not a problem already. The main problem I have is doing hierarchical stuff - which as you pointed out, can be made better by using the "Nested set" / MPTT model.

However, a problem with MPTT is that it's difficult to refactor to it if you already have a parent/child tree, and that it makes some operations remarkably difficult - such as finding immediate children or moving things around. Algorithms for this do exist though and I'm sure these problems have clever solutions.

A hybrid approach might be possible, to maintain some duplicated information to improve performance - but again, it would be require great care to ensure that data do not become inconsistent.

Mark

dream_scape

MarkR wrote:
It's a trade-off. You prefer to make a more complex application that performs better, which is fine. But I prefer an easier to maintain application which is probably correct.

That's really a bit unfair. Have you even seen my application? How the hell can you make a claim like that then?

How exactly does preferring normalized databases and efficient queries translate to "more complex"? Since when does efficiency = complexity and inefficiency = simplicity?

Well first let's not kid ourselves. eCommerce applications are one of the most complex types of web applications. So what we're really talking about is relative complexity.

But wherever you got the idea that efficiency = complex and hard to maintain, send it back because they gravely misinformed you.

MarkR wrote:
However, a problem with MPTT is that it's difficult to refactor to it if you already have a parent/child tree, and that it makes some operations remarkably difficult - such as finding immediate children or moving things around. Algorithms for this do exist though and I'm sure these problems have clever solutions.

A hybrid approach might be possible, to maintain some duplicated information to improve performance - but again, it would be require great care to ensure that data do not become inconsistent.

Actually I do use a hybrid approach. Originally I was using the common adjacency method (parent/child), but then decided to refactor to mptt. The way I did was keep the parent_id field for management, and use the mptt for display. It works well and what I do is anytime there is a category management operation (moving, adding, deleting), it is performed on the parent_id field like was done before. Only I add an extra call then to a method which takes the parent_id column, and uses that to determine the values for mptt. This works so well that I have not yet bothered to refactor the entire way to pure mptt for both management and display.

I've also done things like add a "level" column that denotes which hierarchy level the item sits on (automatically calculated when the mptt data is regenerated). This makes operations like finding immediate children a snap.

Though you do kind of have to enjoy math to like working with mptt. I think the method it uses was developed by a mathematician, and once you catch on, the genius of it is remarkable.

MarkR

dream.scape wrote:
Actually I do use a hybrid approach. Originally I was using the common adjacency method (parent/child), but then decided to refactor to mptt. The way I did was keep the parent_id field for management, and use the mptt for display.

That was exactly what I was thinking of... damn you beat me to it !

It works well and what I do is anytime there is a category management operation (moving, adding, deleting), it is performed on the parent_id field like was done before. Only I add an extra call then to a method which takes the parent_id column, and uses that to determine the values for mptt. This works so well that I have not yet bothered to refactor the entire way to pure mptt for both management and display.

I don't think you should, as it would probably make some operations much harder.

I've also done things like add a "level" column that denotes which hierarchy level the item sits on (automatically calculated when the mptt data is regenerated). This makes operations like finding immediate children a snap.

Ahh yes, that would work. So rather than selecting where parent_id = something, you could do where lft between parent.lft and parent.rgt and level=parent.level+1

Though you do kind of have to enjoy math to like working with mptt. I think the method it uses was developed by a mathematician, and once you catch on, the genius of it is remarkable.

Yes it does seem remarkably clever.

I shall consider it as a future option.

Mark

dream_scape

SeenGee, a few other points.

With eCommerce apps more than most other types, you will want your code to be as secure as possible. There are a number of good articles online about writing secure PHP apps. PHP|A'a Guide to PHP Security is also a good reference, IMO.

You also will want to make sure your data has some basic level of integrity, and ideally more than just a basic level. For MySQL, I recommend using InnoDB table types which will allow true foreign key restrictions and enable transactional behavior.

btw. will (probably) be running on mysql 3.23.58 and php 4.3.2

Ouch just saw this. That is some pretty old technology. You might want to consider running at least the latest MySQL 4.0.x (4.1.x would be better, 5.0 might be pushing it at this time). And you might consider using at least the latest PHP 4.3.x release (4.3.11), or better the latest 4.x release (4.4.2). PHP 5.1 is ideal, IMO, but if you'll be running 4, at least get the latest man 😉

MarkR

I can't see any reason to use any versions less than PHP 5.1 and MySQL 4.1 for new applications.

It will make your life much easier.

You do NOT want the pain of trying to upgrade apps from PHP4 to PHP5 - the OO semantics changed significantly and it will create havoc. Just don't bother targetting any new apps at PHP4.

Likewise, MySQL 4.1 offers some extremely useful features which you'd be an idiot not to want to use (Subqueries, definitely).

Our app is currently targetted at PHP 4.3.11 (NOT 4.4, some semantics changed in a way which breaks things) and it's a pain in the arse, because I can't upgrade it (easily) to PHP 5 as some installations are running on older kit and I don't want to maintain two codebases.

Mark