[Resolved] Optimal multi-directory structure/scheme

acrabadabra

At the moment I am developing a site that probably will extensively use literally thousands of images. While considering how to store those images MySQL seems to be the easiest and fastest method. However, considering the high bandwidth expenses (costs) on a regular host with script and db support, I decided to use a fileserver, that is far cheaper per gig bandwidth and storage being used.

This faces me with the problem of how to store the files effectively in multi-directories, so the redundancy will be taken to a minimum.

Easiest would be to use the corresponding ID of the db as filenames. How and where do I start to setup a directory structure that can easily grow and still would be fast?

(also see: http://www.phpbuilder.com/board/showthread.php?threadid=10250409)

Thanks for any advice in advance!

vaaaska

When a user uploads the image don't they need to select a directory? You could store the path and image name and id in the db.

Or, you could actually have php read the file folder(s) contents and create a tree list for the user to access instead.

It depends upon how you are trying to get your user connected to the images.

acrabadabra

It is for news/article purposes, and amounts can and will exceed thousands of images.

Therefor:
- The user had no saying about the location of the file
- the System can fully take control of directory storage, file naming etc.

So, actually (as Vaaaska mentions), renaming the file to the ID and store it in a directory would be enough. But the only reason why I want to have a directory structure is to speed up things, since finding a file in a directory seems to be done linear and therefor in huge lists can be slow.

Eg.
/image-store/a/0/a0000.jpg
is faster found than
/image-store/a0000.jpg
(if image-store would be the only directory being used for all files)

Weedpacket

Originally posted by acrabadabra
since finding a file in a directory seems to be done linear and therefor in huge lists can be slow.

Although this depends on the filesystem being used: many of the more modern ones maintain a btree index of the directory contents, making search time logarithmic and effectively becoming BLOB-oriented databases. Nesting directories will overlay a search trie on top of this.

acrabadabra

Originally posted by Weedpacket
Although this depends on the filesystem being used: many of the more modern ones maintain a btree index of the directory contents, making search time logarithmic and effectively becoming BLOB-oriented databases. Nesting directories will overlay a search trie on top of this.

Okay. So, actually you are saying that most good providers would have installed a system using this btree index system, and therefor I should not worry about where to drop the images?

Does this take the limitation of the amount of files (apart from the amount of nodes in a Unix/Linux system) in one directory away too? (Eg. 1,000,000 files in one directory should give no problem)

That would be great news though!

acrabadabra

Okay, I went to my hosting provider and they told me the system has a Linux EXT3 file system, meaning (according to them) I could best structure it like a max of 1,000 files and directories in each directory.

So.. unfortunately my question as stated, will remain.

Anyone?

Weedpacket

Well, to draw out your example, you could use bits of the id to provide the directory name. Will you be storing billions of images? If not, a simple integer ID would be enough. That can give nine decimal digits. With a thousand files per directory, there's an obvious triple-depth structure there by taking the ID number in groups of three digits.

I'd actually start from the right and work towards the left: that will keep the files uniformly distributed, instead of having one directory max out before starting another.

// ensure we have nine digits.
$filename = str_pad($id_number, 9, '0', STR_PAD_LEFT);
$filename = strrev($filename);
$filename = substr($filename,0,3) . '/' . substr($filename,3,3) . '/' . $id_number . '.jpg';

Smart would be to contruct directories as and when needed. When saving image # 64322 (which maps to 223/460/64322.jpg), you'd first look to see if directory 223 exists. If not, create it. Then go into directory 223. If 460 doesn't exist, create it, and then save 64322.jpg in there. That will save creating over a million directories in advance. Image #1 will be in 100/000/1.jpg, Image #2 will be in 200/000/2.jpg, and so on.

Fixing the directory structure in advance would be better than rearranging files and directories whenever some threshold is crossed. Even so, if you do manage to have so many filenames that you need ten digits to accommodate them all, you can easily go from 233/460/64322.jpg to 233/460/000/64322.jpg and thus also have room for 233/460/000/1000064332.jpg.

Hiding the image-file retrieval behind a script (so that the src= attribute refers to a PHP script) will hide these shenanigans from the user.

acrabadabra

Wow! It all looks so easy when you see it 🙂
This is exactly what I was looking for! Thanks a billion!

PS. To answer your suggestion about the retrieval hiding, I won't do that, since I will be storing these images on an external cheap hosting party, seperately from my fast and expensive script/db hosting.