How would you do this ?

[jad]

I'm building an RSS aggregator so I'm trying to find out the best way to parse users account feeds equally so Lets say we have 20.000 user with average of 10 feeds in account so we have about
200.000 feed

How would you schedule the parsing process to keep all accounts always updated without killing the server? NOTE: that some of the 200.000 feeds might be shared between more than one user

Now, what I was thinking of is to split users into
1-) Idle users (check their account once a week, no traffic on their RSS feeds)
2-) Idle++ (check their account once a week, but got traffic on their RSS feeds)
2-) Active users (Check their accounts regularly and they got traffic on their RSS feeds)

NOTE: The week is just an example but at the end it’s going to be dynamic ratio

so with this classification I can split the parsing power and time to
1-) 10% idle users
2-) 20% idle++ users
3-) 70% active users.

NOTE: There is another factors that should be included but I don’t want to get the idea messy now (CPU usage, Memory usage, connectivity issues (if feed site is down) in general the MAX execution time for the continues parsing loop shouldn’t be more than 30 minutes 60 minutes) Actually I’m thinking of writing a daemon to do it “just keep checking CPU/memory” and excute whenever a reasonable amount of resource available without killing the server.

Please elaborate.

[sneakyimp]

This could depend on a number of things. In may cases, it is a good idea to only fetch information for users when they come asking for it. If these RSS feeds you are talking about are too slow to be fetched when a user shows up, then it would be smart to find a way to update them while the user isn't around.

On the other hand, if the user shows up on Wednesday and all of his RSS information is old now (perhaps from the previous friday) and he will assume his RSS is out of date and will tend to think of your information as out of date.

You might want to ask yourself a few questions:

• How often are the RSS feeds updated? If they are constantly updated, it would probably be wise to only fetch them when a user asks for them by visiting your site or whatever.

• When they are updated do they trigger any event you can use to fetch the feed? That way, you would always have the latest feed. On the other hand, you might be generating unnecessary traffic because those idle users never come to check on their feeds so you are fetching them for nothing.

• Do all 200,000 RSS feeds come from the same location? Do they all get updated at the same time? In this case it's probably unwise to try and fetch RSS feeds when they are updated.

If it's not possible to fetch feeds when users ask for them, you can consider some other options:
1) an event triggered by the feed source...an email or page access or something
2) a cron job - to occur every minute/hour/day/week/month
3) a probability function - whenever some visitor hits a page you can have a 1 in 1000 chance (or 1 in 100 or 1 in 100000) that your page will go fetch some RSS feeds. which feeds and how many are up to you.

just some ideas.

[Piranha]

An idea I got when reading what sneakyimp wrote is that if the parsing takes to long to do when a user visits but it still is pretty fast for that users feeds (say 10-30 seconds) it should be possible to do it when the user logs in.

I don't know anything about this way to handle things, so please excuse me if it is a stupid idea :rolleyes:

[MarkR]

Obviously this will need to be done via cron or some other scheduled task (maybe a permanently running task which wakes up when there's work to do).

If you are going to need to check 200k feeds on a daily / weekly basis, the job which does it will certainly need some form of parallelisation.

This should in practice not be particularly difficult. I'd recommend that you choose how many processes you want to use (perhaps at runtime), then go through the database assigning a task ID to each of the feeds.

Then fork N child processes (Perhaps you can do this through the shell not through PHP) and have each one process the feeds with its task ID.

Note that if you're fetching these over HTTP, you may want to use a custom HTTP implementation so that you can tune the timeout etc, very carefully. I did this when I wrote a spider.

Clearly proper error handling must be done at every level, e.g. if a feed seems unavailable you can schedule it for another attempt later, or if a feed gives a badly formed document, or a http error, some method of retrying and recording persistent failures, potentially removing bad feeds from the list.

Presumably all these feeds you're adding will be validated for correctness before they're flagged as being "valid".

---

It's complicated, but there's nothing that isn't technically feasible in there.

Mark