fwrite - Writing a google merchant feed

Damian_

I have no issues writing a couple a thousands products into a txt file.

But as soon as it gets too big it just stops no errors occir just doesn't write

Any ideas on how to write large files on fwrite? i currently have about 50,000 - 90,000 lines to write for google merchant

autocolismo

Never tested it but try using 'x' on fread,

$file=fopen("myFile.txt","x");

http://www.w3schools.com/PHP/php_file.asp

dalecosp

PHP Manual wrote:
If handle was fopen()ed in append mode, fwrite()s are atomic (unless the size of string exceeds the filesystem's block size, on some platforms, and as long as the file is on a local filesystem). That is, there is no need to flock() a resource before calling fwrite(); all of the data will be written without interruption.

Hmm, I wonder if [man]file_put_contents/man exhibits the same behavior?

If it is a blocksize problem, you might consider chunking your data into sections and appending:

$fp = fopen('data.txt', 'w');
fwrite($fp, $chunk1);
fwrite($fp, $chunk2);
fclose($fp);

dalecosp

autocolismo;11025443 wrote:
Never tested it but try using 'x' on fread,
http://www.w3schools.com/PHP/php_file.asp

Careful! You don't want to become a W3 Fool! 😉 😃

autocolismo

dalecosp;11025453 wrote:
Careful! You don't want to become a W3 Fool! 😉 😃

:eek::p

Damian_


$File = "gprods.txt"; 

$Data = "";
$Data .= "id	title	description	product_type	link	image_link	additional_image_link	condition	availability	price	brand	mpn	\n";

	 $File = "gprods.txt"; 

				 $Handle = fopen($File, 'w');


foreach ($makeid1 as $u => $id ) 
{

	$imgpath = "http://#########hidden web address#########/live/images/Products/";
	($prod->getProducts($id, $db, $user));
	$description = $prod->aryProducts[$id]->description;
	$model = $prod->aryProducts[$id]->model;
	$MANUFACTURERITEMCODE = $prod->aryProducts[$id]->MANUFACTURERITEMCODE;
	$LONGDESCRIPTION = $prod->aryProducts[$id]->longDescription;
	$google_product_category = $prod->aryProducts[$id]->manufacturerCategoryID;		
	$listprice = round(($prod->aryProducts[$id]->listPrice)*(1.20),0);
	$pr = round((($prod->getProductPrice($id, $db, $user))*1.20),0);
	$manufacturerName = $prod->aryProducts[$id]->manufacturerName;

					$Data .= $id."	";
					$Data .= $description."	";
					$Data .= $LONGDESCRIPTION."	";
					//     echo   "<td>".$google_product_category."</td>";
					$Data .= $product_type."	";
					$Data .= "http://#########hidden web address#########".$id."	";
					$Data .= $imgpath.$manufacturerName."/P".$id."_L00.jpg	";
					$Data .= $imgpath.$manufacturerName."/P".$id."_L01.jpg	";
					$Data .= "new	";
					$Data .= "in stock	";
					$Data .= $pr." GBP	";
					$Data .= $manufacturerName."	";
					$Data .= $model."	";
					$Data .= "\n";


					  fwrite($Handle, $Data, 400000000);  
					  print "Data Written $id <br />";







}
oci_free_statement($arrayids);


	 fclose($Handle);

So I have made some progress after the hours of tinkering bascially the script works! I get the format i need for google!

Now i have the problem where it writes this line in sometimes "$Data .= "id title description product_type link image_link additional_image_link condition availability price brand mpn \n";"

I dont know why? it seems random?

All i did was take the fclose and fopen outside of the foreach loop. which looks like it works because it returned a 500mb document where before when they were inside the loop i was getting 615k document.

Weedpacket

autoclismo wrote:
Never tested it but try using 'x' on fread,

The difference between 'x' and 'w' being that with 'x' opening the file for writing will fail if the file already exists – I don't see how that is relevant here.

Damian wrote:
I dont know why? it seems random?

I do note that every time you you add a line you write the entire file again. You're writing lines 1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 6, ..... You never reset the value $Data after writing its contents.

But don't worry:

                          fwrite($Handle, $Data, 400000000);

You won't write more than ~400MB for each id.

(Incidentally, [man]fputcsv[/man] can output tab-delimited text.)

Damian_


                      fwrite($Handle, $Data, 400000000);

I was getting frstrated and trying everything i could, i just left it in, i have removed that since!

I do note that every time you you add a line you write the entire file again. You're writing lines 1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 6, ..... You never reset the value $Data after writing its contents.

Whenyou say reset the value $Data what would you recommend? shall i just take out the fwrite from the loop?

thanks

Weedpacket

Or change

                        $Data .= $id."    ";

                        $Data = $id."    ";

so that [font=monospace]$Data[/font] starts out fresh for each line.

Damian_

I had a breakthrough, bit annoyed that i didn't think of this earlier, a big DOH moment for me.

So basically i opened with $Handle to write the fields description in first and then closed it, then i opened a $Handle2 to Append the loop after the field descriptions, don't know if this is a good way of doing it, but it works!!

Thanks for your help

		 $File = "gprods.txt"; 

$fields = "id	title	description	product_type	link	image_link	additional_image_link	condition	availability	price	brand	mpn	\n";
fwrite($Handle, $fields);  

fclose($Handle);


$Handle2 = fopen($File, 'a');
foreach ($makeid1 as $u => $id ) 
{

	$imgpath = "###web#";
	($prod->getProducts($id, $db, $user));
	$description = $prod->aryProducts[$id]->description;
	$model = $prod->aryProducts[$id]->model;
	$MANUFACTURERITEMCODE = $prod->aryProducts[$id]->MANUFACTURERITEMCODE;
	$LONGDESCRIPTION = $prod->aryProducts[$id]->longDescription;
	$google_product_category = $prod->aryProducts[$id]->manufacturerCategoryID;
	$listprice = round(($prod->aryProducts[$id]->listPrice)*(1.20),0);
	$pr = round((($prod->getProductPrice($id, $db, $user))*1.20),0);
	$manufacturerName = $prod->aryProducts[$id]->manufacturerName;

					$Data = $id."	";
					$Data .= $description."	";
					$Data .= $LONGDESCRIPTION."	";
					//     echo   "<td>".$google_product_category."</td>";
					$Data .= $product_type."	";
					$Data .= "##web##?PRODUCTID=".$id."	";
					$Data .= $imgpath.$manufacturerName."/P".$id."_L00.jpg	";
					$Data .= $imgpath.$manufacturerName."/P".$id."_L01.jpg	";
					$Data .= "new	";
					$Data .= "in stock	";
					$Data .= $pr." GBP	";
					$Data .= $manufacturerName."	";
					$Data .= $model."	";
					$Data .= "\n";


					  fwrite($Handle2, $Data);  
					  print "Data Written $id <br />";







}
oci_free_statement($arrayids);


	 fclose($Handle2);

Weedpacket

Damian wrote:
don't know if this is a good way of doing it,

I don't even see how it would fix anything.

Damian_

I'm still having issues with the loop, still confused on how fwrite writes the $data, just when I think it's ok, I find that it has duplicated dated!! So not sure what's up with it! Unless anyone here can explain what is up with they loop?

Weedpacket

Um, "duplicated dated"?

The following is not a solution. I just wrote it for my own edification as to what your code is doing.

$imgpath = "###web#";

foreach($makeid1 as $id)
{
    // If you're getting these $ids from a database query
    // then using them in a loop to run a whole bunch more queries
    // is a sign that the original query could have been improved to
    // retrieve the information you actually wanted
    //
    // Are you sure all the $id values in $makeid1 are unique?
    $prod->getProducts($id, $db, $user);

// How many entries does aryProducts have? What are all the others?
$product = $prod->aryProducts[$id];

$description = $product->description;
$model = $product->model;
// pick a variable naming convention and stick with it - your
// code uses four in the following three lines:
// $everythinglowercase
// $EVERYTHINGUPPERCASE
// $wordsCamelCased
// $words_underscore_separated
$longDescription = $product->longDescription;
$googleProductCategory = $product->manufacturerCategoryID;
$listPrice = round($product->listPrice);
$price = round($prod->getProductPrice($id, $db, $user) * 1.2); // Another database query?
$manufacturerName = $product->manufacturerName;

$Data = join("\t", array(
    $id,
    $description,
    $longDescription,
    $product_type,
    "{$imgpath}?PRODUCTID={$id}",
    "{$imgpath}{$manufacturerName}/P{$id}_L00.jpg",
    "{$imgpath}{$manufacturerName}/P{$id}_L01.jpg",
    "new",
    "in stock",
    "{$price} GBP",
    $manufacturerName,
    $model
)) . "\n";
fwrite($Handle2, $Data);

print "Data written $id <br>";
}

Damian_

Thanks for the feedback!

I see your concerns 🙂. I have double checked all ids are unique they are called from a primary key in a database I have also tested them individually! As for the rest I have to use classes from what the previous developer made unfortunately for me he was a really good java developer and used the same logic in his php programming so I have spent hours trawling through pages of codes and library's of classes, to pick out what I need!

When I get back to work on Monday I can show more snippets of the code before the loop, but i think that it's fine, maybe i am missing a step somewhere!