limiting number of images in a single directory

sneakyimp

i'm working on an event listing site. when someone lists an event, they can upload 3 images. Rather than uploading every single image to a single directory, i'd like to break the uploaded images out into a bunch of different directories -- otherwise, any attempts to view the contents of the user image folder could result in a really LONG wait as we list a billion images.

I'd like to limit the contents of any given directory to a thousand items. I'd also like to support a very large number of events and images (1 billion? 10 billion?). I realize that many images might overload a single drive on any server, but am still determined to do it.

Seems to me that some sort of recursive function might be necessary? It's easy enough to turn an event id into a folder. it's also not so hard to make a new subdirectory so there are no more than 1,000 events in a directory...what's more difficult is when we exceed 1 million images -- then my top level directory would have more than a thousand images in it.

any suggestions?

jcanker

Personally, I think the easiest method is to try to not be quite so dynamic with it but planning and laying everything out first...

If I understand what you are saying, if you indeed end up with that many events, even if you dynamically create directories you'll end up with thousands of folders. Instead, I'd try to really get a gauge on how many events you'll really have (What do we mean by events, anyway? Any chance that it's stuff that can be auto-archived at a certain time to keep the "active" working directories fresh?) and set up the structure accordingly to anticipate the size. For instance,

Create a series of directories (you can even code the directory creation in php so as to not have to manually do it) such as 000, 001, 002, 003,...997,998,999.

Randomly generate a three alphanumeric code ( or just numeric) when the event is created. File it away in the appropriate directory, and log the location in the db so your content-creating php code knows where to find the info.

Not a perfect system, as some files will inherently hold more than others, but you'll get a better spread than if you did 1,000 files in the first folder before creating a new directory. Besides, to limit the amount of files, you'd have to have it count every file in every folder every time there was an upload unless you put a marker in the db somewhere to track it.

I'm a fan of dynamically creating things in php, but I also believe that content should be dynamically created while the "containing" objects, such as the db and file structure, should be carefully planned out otherwise it runs amok over time.

Hope that helped and that I actually answered the question you were asking 🙂

sneakyimp

an event is something like
Joey Fest
saturday september 9, 2006

any event will have no more than a handful of images associated with it.

i have literally no idea how many events there might be at any given time.

jcanker

Well, I'd just do what I mentioned before...

create a php page that does something like:

$i=1;
while($i < 1000)
{
mkdir($i, 0777) //the 0777 is ignored if you are doing this on IIS
$i++;
}

that will create all your directories for you so you don't have to do the tedious job of creating them (you may need to alter the path such as "./events/usrposted/".$i Don't forget the quotes if you do set a path

then on the page that allows users to post up, include in the parsing of the posted data a method to automatically select the folder to write to by using

rand(1,999);

to decide which folder to post in. Don't forget to also log that event in the db so it knows which folder to pull the event from when the user wants to retrieve it. I'm assuming that you'll want to list events by date range, not by folder, since the users won't care about past events (I'm assuming this, since you didn't go into detail about the actual purpose of the site--maybe they DO care about "past" events...)

and if it were me, I'd create an admin-accessible page that allows me to enter a date and search the db for all items that occurred before that date so I can either remove them from the database or set a "1" in an "archive" column in the db table so that when I search I can use WHERE archive !=1 or WHERE archive = 0 to keep the users from having to deal with archived events. If you start getting short on disk space, you can also write the code to identify those datafiles as well and delete them from the directories. But I'm probably going in a direction you don't want to, since you said you wanted billions of entries, which would indicate that your users probably are interested in those past events--photos from the events, perhaps? Hey, look at what foolishness went down at Joey Fest on September 9,2006!!! 😃

jcanker

I might also add at this point that I got off track from your original vision of having all those images available to present to the user at once...I think that's a bad idea. Instead, pull the event name/info from the db and present that list. Remember that no matter how fast you may be able to push out a page of 1,000 photos to your users, very few of them will be able to absorb that at any usable rate, and your php will probably time out --unless you change it in the .ini file, but even if you do, what user wouldn't say, "F*** this!" and close the browser window after two minutes of downloading only to have achieved loading 40% of the page?

Give the users a list that's hyperlinked to the event by passing ?<variable> in the url in the anchor tag instead of bombarding them with images they don't want--they'll appreciate you for it.

sneakyimp

i had considered something like your scheme but ruled it out. although it's unlikely that my site will exceed 1 million images, then the scheme you suggested would begin to exceed 1,000 items per directory.

it is nice and simple though.

i've been working on one that will derive a folder path from the event_id of a particular event. each event will have its own directory to avoid image name collisions. additionally, a user can find the directory containing an event's images knowing only the event id without having to use any code or checking the database:

http://phpbuilder.com/board/showthread.php?t=10329055

jcanker

Just read your other thread, and I was wondering why you wanted to limit the folder to 1000 items....it's all clear now🙂

But one problem with the "every event gets it's own folder"
FTP clients will still have to connect up to the containing folder; once you get 1,000 events, you have 1,000 folders, ie, 1,000 items in the ftp root. You're back to square one, aren't you?

if 1,000 folders aren't enough (that is enough for 1,000,000 items according to the 1,000 items per) then you can just add one more layer of complexity to the $i function by adding in a while for $d (for directory 😉 )and have $d create 0-999 directories, each one of which is creating $i subdirectories with its own while statement for $i. Then you have 1,000 X 1,000 folders each containing 1,000 items--that allows for a billion items that ftp and dreamweaver users should be able to deal with. Change your rand() statement to pull up 2 random numbers, one for the containing directory, one for the subdirectory, and you'll be fine. What are the odds with 1,000,000 directory combinations that you'll hit 1,000 items? If you really want to check against it, create a function on your administration page that looks for subdirectories with over 1,000 items, moves a chunk of it to another file with space, and updates the database with the new dir info

As far as name collisions, that should be checked for in the db regardless, not by parsing filenames in a folder. Quick on a db (especially if you index the column), thrashing to the disk by checking filenames in the directory.

Users can still find whatever events they want by knowing the eventid --you just have to use the eventid as the primary key (which you should be doing anyway!) and using that as your call to the DB--any way you want the user to be able to access it is up to you. Don't overestimate the ability of your users, however; How the heck are they supposed to know what the eventid is for their specific event? They'd have to query the db at some point to get that info. You might as well serve it up to them in the first place. If they do know the eventid, it's just as easy to let them pass it in the url with a %dir%/?eventid=999 type of entry.

sneakyimp

But one problem with the "every event gets it's own folder"
FTP clients will still have to connect up to the containing folder; once you get 1,000 events, you have 1,000 folders, ie, 1,000 items in the ftp root. You're back to square one, aren't you?

Actually, the code is smart enough to break an event id more than 6 digits enough times that you avoid more than 1000 items per folder. if event id is 123456789 then you get a folder name like 123/456/789/123456789.

As far as name collisions, that should be checked for in the db regardless, not by parsing filenames in a folder. Quick on a db (especially if you index the column), thrashing to the disk by checking filenames in the directory.

A name collision is only a problem for a given folder. It is extremely likely that two different users of a single brand of camera will upload an identical filename. because of my event id parsing scheme, only photos relating to a particular event will be stored in a given directory.

HOWEVER, a user might have two different images with the same filename in different folders. I guess I should automatically rename any colliding files. rats.

The issue about finding files relating to a particular event ID is not an issue for users but for developers. The entire site is driven by a search mechanism which is pretty comprehensive. Users search for events based on a bunch of different criteria. When they find the event, the pics are there.

My concern on that front related more to a web developer's perspective. Suppose there is a big where the image gets uploaded by the dbase query to create a record fails? It should be easy in that case to locate the folder where the images went to see if something was weird about the filename. this should be easy because for event id 1234567 you can just search like this:

find * -name '1234567'

granted, you might find duplicate items for short event ids but that doesn't bother me.

Ultimately, I am worried about the disk drive performance for my scheme. Does creating all these directories slow things down or will the directory records get cached somewhere? If anyone can answer that, I'd be quite happy.