It's generally not a good idea to have multiple values in a single field.
If you want to keep track of who rated who (and the score), you could create a table to store this information as you said. Hundreds of thousands of records is not a problem for a good relational database like MySQL or PgSQL.
If speed is a concern, one suggestion is to create two new fields next to the items you are rating.
Example Table:
ItemID itemname Ratings TimesRated
When you do an insert, insert into both tables. One holding who rated who and what, then in the item table add the rating to the total and increment TimesRated by 1.
update <items_table>
set Ratings = Ratings + $newRating
, TimesRated = TimesRated + 1
where ItemID = $id
Then on your select query (or in PHP if you'd like), Ratings/TimesRated = Average Rating. I assume you'll be showing ratings more than the listing of who rated who, so this may help alleviate some strain.
However, there is also the trade-off of having to do two inserts and two selects (if showing both things). So this will all really depend on your situation. Hopefully this gives you another option to consider.