Test-Server?

[lajkonik86]

When you have a live website. How do you test new code and how do you migrate this successfully to your live website?

[Piranha]

The best way to test new code is to have a development server set up with the same versions of everything. It could be your own computer if you are the only one developing the site, or another development server, the important thing is that you have the same versions of everything (yes, I repeated it on purpose since it is very important). Test, test, test and test on that development server until everything works as expected, then the code is ready for the live website.

A good idea is to sometimes copy the whole database from your live server to the development server, then you get a feeling about how fast your script will be on the live website. Another good idea to get a feeling for the speed is to use an old computer, about 500 MhZ, as a development server. If it is fast there the odds are that it will hold up better to higher traffic on the live server and with lots more in the database.

[edit]How to migrate to the live site I have no idea.[/edit]

[MarkR]

It depends. Of course it's normally very important that your dev server is as close as possible to the production server, configuration-wise.

You could set up a test server with an identical software set to production, for the purpose of doing final testing before deploying changes.

Another option is to create a test environment on the production server itself - a private copy of the site (typically using its own database) which you can run integration testing against.

But of course, you can't run performance testing against that environment because you don't want to place unnecessary load on your production servers.

If you have more than one production server in a cluster, you could try removing one of them, upgrading it individually and testing that before you upgrade the others.

---

As far as production migration is concerned - we normally use a set of scripts which apply (and remove) database changes, and have a shell script which runs them as necessary. We take the production site down by setting a flag somewhere which causes users to see a "temporarily out of service" message (NB: This may or may not be acceptable depending on your customers' SLA), then run the script which applies the database changes and copies the new files into production.

We then do some cursory tests while the site is still down (the flag still lets us view the site on production), before putting it back up.

If you need your site to run with no scheduled downtime, it could be tricker.

Running a site with no scheduled downtime is pretty much impossible, as some things, especially schema changes on big tables, are bound to create problems.

If you do an ALTER TABLE on a table with 100 million rows on it, that table will become locked for quite some time, during which time PHP pages attempting to access the table will get blocked - resulting in downtime.

There is no easy way around this.

Mark

[lajkonik86]

great answers, thanks guys

[dalecosp]

MarkR wrote:

Another option is to create a test environment on the production server itself - a private copy of the site (typically using its own database) which you can run integration testing against.

But of course, you can't run performance testing against that environment because you don't want to place unnecessary load on your production servers.

We've used this approach before. In DNS, we created "test.example.com" with it's A record set to the server. In the PHP header file for the development site, something like this:

if (file_exists("./debug") {
   if ($_SERVER['REMOTE_ADDR']1=$mycompany_ip ||$_SERVER['REMOTE_ADDR']1=$client_ip ) {
      header("location:http://www.example.com");
   }
}

As far as production migration is concerned - we normally use a set of scripts which apply (and remove) database changes, and have a shell script which runs them as necessary. We take the production site down by setting a flag somewhere which causes users to see a "temporarily out of service" message (NB: This may or may not be acceptable depending on your customers' SLA), then run the script which applies the database changes and copies the new files into production.

We then do some cursory tests while the site is still down (the flag still lets us view the site on production), before putting it back up.

If you need your site to run with no scheduled downtime, it could be tricker.

Running a site with no scheduled downtime is pretty much impossible, as some things, especially schema changes on big tables, are bound to create problems.

If you do an ALTER TABLE on a table with 100 million rows on it, that table will become locked for quite some time, during which time PHP pages attempting to access the table will get blocked - resulting in downtime.

There is no easy way around this.

Mark

That's insightful. Obviously this applies more strongly to sites where users are actually making database changes. It might be possible to do a little mucking around with the DB on a site in real-time if the data is primarily static and used for lookups (and if the words "100 million" could never be used in the same sentence with any DB terminology). I'll need to think about this if/when I ever get another project ready to go.

As far as the actual work with Apache, that should be as easy as editing httpd.conf and issuing "apachectl graceful". That, and maybe a lot of marketing 😃

[lajkonik86]

so far it looks like it will be having a test and development server
keeping track of which files on the development server are different
when updating do so at 12,am in maintenance mode and copy and paste the stuff.

messing up a file or two or forgetting a database change would be a mess though.

Better keep track of the database change commands i use and run those.

[MarkR]

You should not have to keep track of what files have changed. That's what your source code control system is for.

You do use one, right?

Then a script which takes a snapshot of your source code, packages it up and installs it in a single, quick operation should do.

It's useful to have it also remove obsolete files which have been deleted in your source code.

But be aware if you're using any data files, not to overwrite them.

Database schema changes are another issue and must be carefully controlled.

When you're working with multiple developers, of course each developer should have their own copy of the application to work on (in your SCM, of course) - and their own database too. So when one developer makes a schema change, they must script it and notify all the other developers so that they can update their own database schemas as necessary when they update their dev area.

Mark

[lajkonik86]

hey,

no have never used one of those, googled it a bit and found
Subversion and CVS
Are they any good?
What are these programs supposed to do, its not entirely clear to me.

Thanks🙂

[Sxooter]

MarkR wrote:

Running a site with no scheduled downtime is pretty much impossible, as some things, especially schema changes on big tables, are bound to create problems.

If you do an ALTER TABLE on a table with 100 million rows on it, that table will become locked for quite some time, during which time PHP pages attempting to access the table will get blocked - resulting in downtime.

There is no easy way around this.

Really? hmmm:

select count(*) from a1;
  count
----------
 10000000

\timing
Timing is on.

alter table a1 add column extra text;
ALTER TABLE
Time: 949.583 ms
insert into a1 (i1,i2,extra) values (23,45,'more text');
INSERT 0 1
Time: 81.104 ms
select currval('a1_id_seq');
 currval
----------
 10000001
Time: 13.455 ms
alter table a1 drop column extra;
ALTER TABLE
Time: 55.070 ms
delete from a1 where id=10000001;
DELETE 1
Time: 16.243 ms

 select 949.583+81.104+13.455+55.070+16.243;
 ?column?
----------
 1115.455

Took one second.

Oh, you meant MySQL with myisam tables.  That is an issue.

Now, something like create index might block, then you'd have to wait for the index creation to finish, right?  Oh wait, no you don't: [URL="http://www.postgresql.org/docs/8.2/static/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY"]http://www.postgresql.org/docs/8.2/static/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY[/URL]

But I do agree with your point that real 24/7 operation is very hard to implement.  We use Oracle where I work and honestly, it's basically impossible to run Oracle 9 with no down time.

[MarkR]

Sxooter wrote:

Took one second.

Oh, you meant MySQL with myisam tables. That is an issue.

Obviously your experience may vary. But as far as I'm aware, there are SOME types of "ALTER TABLE" with some table types in SOME databases, where the ALTER TABLE can take a long time and will block writes (or perhaps reads) to that table in the interim.

If your application can handle absolutely no downtime, this is a significant issue.

However, it is possible that some kinds of ALTER TABLE, especially on smaller tables, won't take very long. These are acceptable.

With MySQL, taking backups is also an issue, as there is no standard way to take a coherent backup without locking out the entire database - replication is the only solution here (but of course, to set it up requires a backup)

Mark

[Sxooter]

Yeah, in almost any system there are going to be things that make it hard to make it truly 24/7.

As for databases, the pgsql team has done a LOT to eliminate those kinds of things from the database.

DDL in a transaction can be an issue if you're sitting thinking about it while another transaction is pending, it may block until you commit or roll back, but if you run it as a script, where all or nothing will be committed then the blocking will only last as long as the DDL is running, which runs lightening fast unless you're creating indexes, which can be created concurrently which is slower but non-blocking.

The real issue on 24/7 with any database based app is generally patching. You've pretty much got to take down the db to patch either the db or the application running on top of it, especially if there's DDL that needs to run along with the code update in the app.

[piersk]

I recently bought this book by one of the guys that wrote Flickr. There's a really good section on deploying web sites in that.

[Sxooter]

lajkonik86 wrote:

hey,

no have never used one of those, googled it a bit and found
Subversion and CVS
Are they any good?
What are these programs supposed to do, its not entirely clear to me.

Subversion is freaking awesome.

It's about version control.

You write an app. You make a change, you find out, six weeks later you introduced a bug in abc_xyz.php that you need to backout. With version control you can do things like:

svn log abc_xyz.php

and it will show you a lot of all the changes, with revision numbers, then you can:

svn diff -r1234:1235 abc_xyz.php

and it will show you the diffs between the two versions of abc_xyz.php

[laserlight]

no have never used one of those, googled it a bit and found
Subversion and CVS
Are they any good?
What are these programs supposed to do, its not entirely clear to me.

You could start by reading the Subversion Book. That said, I would attempt to subvert you by recommending the distributed version control system Bazaar instead. I found Subversion unsatisfactory as I wanted to have full access to version control of the same project on multiple computers that may be offline while I work.

[MarkR]

Obviously choice of a source code control system will be one you want to make carefully. Look for the feature set you feel you need.

A SCM system is only part of your overall development / deployment strategy, and if not used correctly will not be advantageous.

Consider how you want to work overall, thinking about the bigger picture.

If you have more than one developer, they MUST each have their own development copy of everything. You may also want to consider creating one or more staging, testing, demo environments for internal use on non-production systems. The requirements will depend exactly what sort of application it is.

You will want some system of rolling out updates to your staging, demo, test and production systems which is highly structured and correctly deploys everything in an automated or semi-automated (at least well documented) fashion.

Otherwise, you're going to end up with "just some files all over the place" which is not a robust mechanism for maintaining a working web application.

Mark