use ftp or http based file-up/downloads?

Anon

Hello,
I'm making a site where users have their own documents and other files.
When it is finished it's going to be used by quite alot of users (10.000+) with everybody their own share of diskspace. It is not going to be used intensively by eacht apart user, but there's going to be many users tho.

So I need a way to store their files. Now I wonderd wheter to use an ftp-server to handle the files, or go with a directory-tree based style, say http-based instead of ftp.

Also the users may only be able to view their files from the websites, they are not allowed to log in the ftp-server. Becasue everything has to be know in the database.
So their should be, for security reasons, an other password for the user when logging in to the ftp than the one they use to log in to the website.

And if it can be done (which i couldn't find out), the website (php) needs to create an account on the ftp-server for each user when they register. I couldn't find somethin like that, so i'll ask here if this is possible.

I did some research, and mostly i came to the conclusion that ftp wasn't really needed. So i think that http-based is much easier to use with php, also because their is 1 server and not 2.

So i'm asking if you have an opinion in wether to use ftp or http.

thanks for reading.

nashirak

Ftp is a rather old protocol. My preference would be to just use http? Why? I dont like giving people real accounts on my machine, and for the fact that it would be easier to use http. Yes, its possible to give people accounts on you maching (I speak on *nix). You kind have to do a little trick though. You have to have php right to some file and right say a shell scritp or perl script that runs as root in your cron tab, that mabye every 10 minutes or so would activate those accounts. (At least thats how I would do it).

So my opinion and preference would be http.

Hope that Helps!

Anon

yes thanks, clear explanation.
The solution to get accounts on the ftp is nice 🙂

Indeed it is better not to have people have real accounts on the server.

So thanks for your opinion, i'm gettin more sure now to use http.

[deleted]

Forget HTTP, use FTP.

HTTP was never designed to do file uploads, FTP was designed to do nothing BUT file uploads.

FTP is faster and more secure than HTTP.
FTP does not require 'real' accounts on the server, just an FTP username and password.

In short: FTP already does everything you plan to build in HTTP, and it does it better than you can every make it in HTTP.

Fopr one thing, FTP can transfer directories, which HTTP cannot.

Anon

good one.
"HTTP was never designed to do file uploads, FTP was designed to do nothing BUT file uploads."
Was why I came to the idea of ftp.
But is the difference so significant?

The files are not going to big tens of mb's big 1 mb may be relatively big file on the server. So maybe that gives ftp a little less the option to go for?

ftp is more secure?
hm well i can do a force transfer/download to the client with php using fileheaders. The user now never know where the file came from. The file can even have a different name on the server than it says in the save/download dialog.

I don't need to transfer directories, only files.

And yes you are right about the user account difference between ftp and server.
I already thought so, but the post of Nashirak made me think otherwise.
But still i would need another script running putting the usernames in the ftp-server.

Which I think is a bit too much work when it can be done with only http.

Maybe you noted about this post that I rather not use ftp, mainly because of the extra hassle.

But try to convince me.

Anon

another thing:
"ftp is more secure"

when a user gets to the site and asks a list of their files for example. With ftp there has to come another connection to the server to request the list.
Now is it a good idea to give this user another name and pass on the ftp-server than the one thay use to logon the website?

They don't have to manually anter the ftp login and pass when they are logged in on the website. Cause the name and pass for the ftp-server is in the database for every user.

or would it be useless to have other login and pass when users don't have to enter them manualy ?

[deleted]

"But is the difference so significant? "

time it and see.

There is a reason why the worlds biggest download centers use FTP instead of HTTP, and it is not because HTTP is a lot better than FTP. 🙂

"ftp is more secure?
hm well i can do a force transfer/download to the client with php using fileheaders. The user now never know where the file came from. The file can even have a different name on the server than it says in the save/download dialog."

That is a cute feature, but has nothing to do with security. In order to get to the download, you'll need some sort of link with parameters, that is where the hacker will strike, because you will make a mistake there.
With FTP, they can enter filenames all day long, they will never get anything you don't want them to get, because FTP has been around as long as the internet has.

"I don't need to transfer directories, only files."

Give it a few months and I can almost guarantee you you will need it. 🙂

"Which I think is a bit too much work when it can be done with only http."

That's no argument. Putting in one extra script is hardly a big issue when you intend to write an entire FTP-replacement from scratch. 🙂

"Maybe you noted about this post that I rather not use ftp, mainly because of the extra hassle."

Yes I noted. In today's WWW World it seems everything must be done through HTTP. And when you ask why, the answer is always 'because I already know how to do it' or 'it looks better' or 'the competition is doing it'.
The argument is never 'because it takes only half the resources of my server' or 'becuase it uses true binary transfers instead of encoded ascii' or 'because it is much faster'.

"But try to convince me."

Nah, I've made my point (several times on this forum) so if this doesn't convince you, then I wish you all the best in re-inventing the wheel :-)

Anon

great post 🙂

but to be honest I did think about the resources and binary/ascii uploading, cause I did some research before i posted, but well i didn't mention it 🙁.

Which brings me to a question I wanted to ask: If I want to upload to a ftp-server with php I have to give a parameter wether the file is binary or ascii. Is there something to check the file to get the right transfering method?
Or do i have to make a script which check the extensions?
With http it is just upload and done, no difference, right?

And about the force download u are probably right. good arguments 🙂

thx

Anon

a few things about hacking.
(I'm not that trained in linux)

When I want to use a ftp-server I have to create accounts on it. But does the cron job have to run as root? This isn't quite safe right?

And if one can hack the header, and thus knowing the file's true name and place, isn't it possible to hack the ftp-server?
Or is the difference between the 2 hacks big?

Question about .htaccess and .htpasswd
Now if I choose using http-based file-serving I can protect the different users directories with login's and passwords stored in a .htpasswd file.
So if one wants to access a file a popup comes which asks for the name an pass.
But I don't want this pop-up to come if the user is already logged in.
Is there a way to "disable"/bypass the .htaccess if the user is logged in?

Anon

Hm i just thought of some protection of the documents.

use a referrer check in an .htaccess file

let's say when a user is logged in, then goes to www.domain.tld/groupe/username
his files are in www.domain.tld/groupe/username/files

in this files directory I can place a htaccess file containing the following.

AuthUserFile /dev/null
AuthGroupFile /dev/null

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^{http://www.domain.tld/groupe/username/}
RewriteCond %{HTTP_REFERER} !^{http://domain.tld/groupe/username/}

RewriteRule /* http://www.domain.tld/restricted/index.html [R,L]

Now only the logged in user can log access the files.
I think this is a good solution 🙂

haven't tried it tho

[deleted]

"But does the cron job have to run as root?"

not really

"This isn't quite safe right?"

yes and no.
The script does need extra privileges, but the script will only accept data in the correct format. Obviously you'd build some protection in so it will never try to set the password of system-users like root/ftp/www etc.

"And if one can hack the header, and thus knowing the file's true name and place, isn't it possible to hack the ftp-server?"

The whole point of having an FTP server is that people can simply browse to the correct location and download/upload whatever files they need. Which files they can access depends on your security settings, and those depend only on the username and password, nothing more.

"Question about .htaccess and .htpasswd"

'obviously' with 10k users, using htaccess is out of the question. Just reading the .htaccess file would take more resources than you'd care to spend on it.
Instead, you'd have to create a fileserving script that get's it's username and password from a custom-made authorization scheme (such are readili available on the net but will need tweaking for your app).
Then you must make sure that not a single page on your server can be accessed without proper logon information.

[deleted]

"use a referrer check in an .htaccess file"

no you can't. Well, technically you can, but the REFERER field is optional, browsers don't have to send it, so you cannot depend on it. (what's more, it is childishly easy to fake the REFERER)

nashirak

OK the only reason I suggested the HTTP approach was the following:

His specifications stated that he wanted them to be logged into a database
His specifications also stated that he did not want them logging into the server manually to download an upload files.

I agree if he left both of these out he would be better off using FTP. But since its going to be tough (especially letting them only logging in over there webpages), I think that the HTTP way would be better. Even though yes FTP would be better suited for File Transfer.
Hope that clarifies!

Anon

"And if one can hack the header, and thus knowing the file's true name and place, isn't it possible to hack the ftp-server?"

I didn't meant a direct connection between the two. But more that if a person is capable of getting the real header, then hacking the ftp-server could also be done?
Well i presume getting the real file-location is a easier then actually hacking the ftp-server, but still.

About the other post about the .htaccess and the referrer: Thanks for the info that is easy to work around.

And indeed it is quite some work to do for the server with the use of htaccess and htpasswd, that's also why I wondered about the referrer thing.

But there isn't really a safe way (well compared to ftp) to store files http-based?.

I mean I could chmod the files not accessable with php when the user logs off (I think that's possible).
Or would that be too much to ask the server capacity/speed-wise.
I don't know that much about php and file handling.

Well ftp seems more and more the way to go 😉
It's just that http-based files can't be as good protected as ftp-based right?
Well maybe they can, but doing so is going to cost the server relatively a lot of power.
And even if it doesn't make any big difference it seems to me that when using a ftp-server it is easier to place/split the web-server and ftp-server on different machines than doing so when http-fileserver and web-server. This may be handy if the server-load is getting high.
Am I right?

Anon

indeed.
In the database there should be a record of the files. Each file has a (maybe unique) ID.
The user should be able (via a webpage) to give certain users rights to view certain files, as a reference. These rights are stored for each file(ID) in the database.

And also it is very important that no-one can login to the ftp-server without any connection to the website. Because if this was possible then the file-ID's wouldn't be right anymore.

I'm planning to name the files on the server something like this "<ID><time<date>" or "<ID><randomString>" (time and date to make sure the file name is hard guesable, but a random string would be better maybe)
And then in the database the real name is stored.
And with this use the force download idea.

I think this is pretty secure using http.
And if the server has an easy time we could also add .htaccess

As for ftp.
Since the user isn't allowed to make a connection manually it is best that for each user is a seperate login and pass. But the user is not allowed to know these, cause then the link with the ID could get screwed.
So only php can use this ftp-connection to get the files.

This may be the savest way. (have php make a connection with the ftp server, to the user, unknown login and pass.)

But on the other hand, looking only at security, http is quite good to. Even if they can get the file's real name on the server then the other files would be hard to guess.

So seeing it now I think http is a good option for my question.
I mean I don't really see why I should add an ftp-server when this is sufficient.
Also keep in mind that the files are mainly Ms-word documents, so they aren't very big mostly.

ps. It is possible that I sometimes speak against myself, that's cause I haven't got everything clear. Listing all/most possibilities here is e big help 🙂

[deleted]

"then hacking the ftp-server could also be done"

Nope. Like I said, if the FTP server does not allow it, you'd have to hack your way through the ftp server and into the OS to change the user's permissions. Like I mentioned, the worlds largest repositories use FTP without any such problems.

"I mean I could chmod the files not accessable with php when the user logs off (I think that's possible)."

ump no. if you do it HTTP style, you are dealing with one user on your system, and one user only: the webserver. The webserver reads and writes all files, and so all files must be accessible to the webserver.

When a user logs on, the webserve reads his rights from the database. It then uses those rights to decide which of the files (which all have the same read/write rights) the user can see/read/write.

Anon

you can also set the user's shell (listed in /etc/passwd ) to usr/bin/ftpd or whatever your ftp server is. That way they automatically log in to ftp. Of course if there is an exploit for the FTP server then they may get shell access.

If you want to keep track of who logs in when and retrieves what, and you aren't familiar with the logs or don't want to bother writing another script to parse them and update the database, using http put/get and having your script add an entry to the DB at the same time will probably just as easy--without having to create accounts.

Anon

well I now know I\'m goint to use http-based file serving.
The main reason is because the user is not allowed to connect to the ftp-server manualy.

I figured that when a file was needed by a user which resides on the ftp-server, the problem exists. The problem is that when the user recieves the requested file he has to have some sort of connection woth the ftp right?
So with this url, which contains username and password, he now knows how to access the ftp-server.
A workaroudn would be to let php get the file form the ftp-server, and then copy it to the server. And when this is done send the file to the user. This method is a bit clumsy i think.

So I\'ll stick with http.

thx for all replies.