Trim the HTML and save content to txt file

[Nightmaster]

Hello there,

I have a large HTML file, that looks like this:

<p class="title">Paragraph 1</p>
<p class="number">1</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>

<p class="title">Paragraph 2</p>
<p class="number">2</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>
....etc...

I want to save content of each section in new .txt file, so I have txt files named with a content of "title" element (for example: Paragraph 1.txt, Paragraph 2.txt etc) and the content of first file should be everything under the first title class, upto the next "title" tag.

So the example for the above code should be:
Filename: Paragraph 1.txt
Content:

<p class="number">1</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>

Do note that sometimes there's 3 "normal" classes in P tag, sometimes more, it's not the same length for each paragraph.

Any suggestions?

[NogDog]

Here's what I came up with for this Sunday code challenge. 😉

<?php

// create some test data:
$text = <<<EOD
<p class="something">foo</p>
<p class="title">Paragraph 1</p>
<p class="number">1</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>

<p class="title">Paragraph 2</p>
<p class="number">2</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>
<p class="something_else">bar</p>
EOD;

$tempFile = tempnam('.', 'test');
file_put_contents($tempFile, $text);

// now the actual processing:
$fh = fopen($tempFile, 'r');
$paragraph = false;
$data = array();
while(($line = fgets($fh)) != false) {
    if(preg_match('/class="[^"]*\btitle\b[^"]*">([^<]+)</', $line, $matches)) {
        $paragraph = str_replace(' ', '_', $matches[1]);
    }
    elseif($paragraph and preg_match('/class="[^"]*\b(number|normal)\b/', $line)) {
        $data[$paragraph][] = trim($line);
    }
}
fclose($fh);
unlink($tempFile);
foreach($data as $paragraph => $lines) {
    file_put_contents($paragraph, implode(PHP_EOL, $lines));
}

// look at results:
foreach($data as $paragraph => $lines) {
    echo "<pre>$paragraph:\n".htmlentities(file_get_contents($paragraph))."</pre>\n";
}

// clean-up -- probably don't want this in your script!!
foreach($data as $paragraph => $lines) {
    unlink($paragraph);
}

[Nightmaster]

Woa, that's great, thanks!

Now, I forgot to mention, in some tags there's also empty a tags which should be removed because as far as I can see it won't show anything if there's such thing, for example:

<p class="something">foo</p>
<p class="title"><a id="str_93" name="str_93"></a>Paragraph 1</p>
<p class="number"><a id="clan_131" name="clan_131"></a>1</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>

<p class="title"><a id="str_94" name="str_94"></a>Paragraph 2</p>
<p class="number"><a id="number_131" name="number_131"></a>2</p>
<p class="normal">text, text...</p>
<p class="normal">more text, text...</p>
<p class="normal">more text, text...</p>
<p class="something_else">bar</p>

Also, after everything else - I need to create a .txt files with content from the above paragraphs if possible.

Thanks once again, you rock!

[NogDog]

Those empty tags are probably there for in-page navigation, e.g. if you stick a "#number_131" at the end of the URL, the browser will scroll down to the element with that ID. If you decide you want to get rid of those, you could apply preg_replace() to the string before saving it in the array. As far as adding a ".txt" suffix, you could just append it:

file_put_contents($paragraph.'.txt', implode(PHP_EOL, $lines)); 

[Nightmaster]

Hey again,

Thanks for your answer, got it!

I changed the last part of your code to this:

// clean-up -- probably don't want this in your script!!
foreach($data as $paragraph => $lines) {
    unlink($paragraph);
	$lines = preg_replace('/<p class="normal">(.*?)<\/p>/', '$1', $lines);
	file_put_contents($paragraph.'.txt', implode(PHP_EOL, $lines));  

} 

That should remove p with class=normal from the output, and it works fine in some cases and in some not.

here's the part of the html:

<p class="title"><a id="str_105" name="str_105"></a>Neovlaš&#263;eno prisluškivanje i snimanje </p>
<p class="number"><a id="clan_143" name="clan_143"></a>&#268;lan 143 </p>
<p class="normal">(1) Ko posebnim ure&#273;ajima neovlaš&#263;eno prisluškuje ili snima razgovor, izjavu ili kakvo saopštenje koji mu nisu namenjeni,<br>
	kazni&#263;e se nov&#269;anom kaznom ili zatvorom od tri meseca do tri godine. 
</p>
<p class="normal">(2) Kaznom iz stava 1. ovog &#269;lana kazni&#263;e se i ko omogu&#263;i nepozvanom licu da se upozna sa razgovorom, izjavom ili saopštenjem koji su neovlaš&#263;eno prisluškivani, odnosno tonski snimani. </p>
<p class="normal">(3) Ako je delo iz st. 1. i 2. ovog &#269;lana u&#269;inilo službeno lice u vršenju službe,<br>
	kazni&#263;e se zatvorom od šest meseci do pet godina. 
</p>

And here's what i have as output in txt file:

<p class="normal">(1) Ko posebnim ure&#273;ajima neovlaš&#263;eno prisluškuje ili snima razgovor, izjavu ili kakvo saopštenje koji mu nisu namenjeni,<br>
(2) Kaznom iz stava 1. ovog &#269;lana kazni&#263;e se i ko omogu&#263;i nepozvanom licu da se upozna sa razgovorom, izjavom ili saopštenjem koji su neovlaš&#263;eno prisluškivani, odnosno tonski snimani. 
<p class="normal">(3) Ako je delo iz st. 1. i 2. ovog &#269;lana u&#269;inilo službeno lice u vršenju službe,<br>

However in this example it works fine:

<p class="title"><a id="str_12" name="str_12"></a>Važenje krivi&#269;nog zakonodavstva Srbije za stranca koji u&#269;ini krivi&#269;no delo u inostranstvu </p>
<p class="number"><a id="clan_9" name="clan_9"></a>&#268;lan 9 </p>
<p class="normal">(1) Krivi&#269;no zakonodavstvo Srbije važi i za stranca koji van teritorije Srbije u&#269;ini prema njoj ili njenom državljaninu krivi&#269;no delo i kad nisu u pitanju krivi&#269;na dela navedena u &#269;lanu 7. ovog zakonika, ako se zatekne na teritoriji Srbije ili bude izru&#269;en Srbiji. </p>
<p class="normal">(2) Krivi&#269;no zakonodavstvo Srbije važi i za stranca koji prema stranoj državi ili prema strancu u&#269;ini u inostranstvu krivi&#269;no delo za koje se po zakonu zemlje u kojoj je u&#269;injeno može izre&#263;i kazna zatvora od pet godina ili teža kazna, ako se zatekne na teritoriji Srbije, a ne bude izru&#269;en stranoj državi. Ako ovim zakonikom nije druk&#269;ije odre&#273;eno, sud u takvom slu&#269;aju ne može izre&#263;i težu kaznu od one koja je propisana zakonom zemlje u kojoj je krivi&#269;no delo u&#269;injeno. </p>

Output in file:

(1) Krivično zakonodavstvo Srbije važi i za stranca koji van teritorije Srbije učini prema njoj ili njenom državljaninu krivično delo i kad nisu u pitanju krivična dela navedena u članu 7. ovog zakonika, ako se zatekne na teritoriji Srbije ili bude izručen Srbiji. 
(2) Krivično zakonodavstvo Srbije važi i za stranca koji prema stranoj državi ili prema strancu učini u inostranstvu krivično delo za koje se po zakonu zemlje u kojoj je učinjeno može izreći kazna zatvora od pet godina ili teža kazna, ako se zatekne na teritoriji Srbije, a ne bude izručen stranoj državi. Ako ovim zakonikom nije drukčije određeno, sud u takvom slučaju ne može izreći težu kaznu od one koja je propisana zakonom zemlje u kojoj je krivično delo učinjeno. 

I think that it will remove P tag if the whole paragraph is in one line, and if it's divided it won't. Also there's that <br> tag that should be removed. I also noticed that content after the <br> tag is cut-off and the content is not showed.

[Nightmaster]

Okay, got br tags removed with str_replace.
However, still have the problem with paragraphs which is not closed with </p> in the same line where <p> starts and the content in new line is not copied to txt files.

Example HTML:

<p class="normal">(1) Ko nemo&#263;no lice koje mu je povereno ili o kojem je ina&#269;e dužan da se stara ostavi bez pomo&#263;i u stanju ili prilikama opasnim za život ili zdravlje,<br>
	kazni&#263;e se zatvorom od tri meseca do tri godine. 
</p>
<p class="normal">(2) Ako je usled dela iz stava 1. ovog &#269;lana nastupilo teško narušenje zdravlja ili druga teška telesna povreda ostavljenog lica,<br>
	u&#269;inilac &#263;e se kazniti zatvorom od jedne do pet godina. 
</p>
<p class="normal">(3) Ako je usled dela iz stava 1. ovog &#269;lana nastupila smrt ostavljenog lica,<br>
	u&#269;inilac &#263;e se kazniti zatvorom od jedne do osam godina. 
</p>

This is what I get in .txt file:

<p class="normal">(1) Ko nemo&#263;no lice koje mu je povereno ili o kojem je ina&#269;e dužan da se stara ostavi bez pomo&#263;i u stanju ili prilikama opasnim za život ili zdravlje,
<p class="normal">(2) Ako je usled dela iz stava 1. ovog &#269;lana nastupilo teško narušenje zdravlja ili druga teška telesna povreda ostavljenog lica,
<p class="normal">(3) Ako je usled dela iz stava 1. ovog &#269;lana nastupila smrt ostavljenog lica,

Also, the html for the <p class="normal"> is not removed in this case, guess because there's no </p> in the same line.

[Derokorian]

Nightmaster;11055587 wrote:

Okay, got br tags removed with str_replace.
However, still have the problem with paragraphs which is not closed with </p> in the same line where <p> starts and the content in new line is not copied to txt files.

Sounds like you want to use the multi-line option in preg. Try adding the m modified after the closing delimiter. Though I warn you, its possible you need to change your captures to be non-greedy as well.

[Nightmaster]

Sorry, but I'm not familiar with php that much, can you please specify a bit more what should i add and where?
Thanks, and sorry for bothering.

[Derokorian]

Sure! Change this:

    $lines = preg_replace('/<p class="normal">(.*?)<\/p>/', '$1', $lines); 

to this:

    $lines = preg_replace('/<p class="normal">(.*?)<\/p>/m', '$1', $lines); 

I'm not 100% sure it will work, but I'm at least 90%.

[NogDog]

And as the regexp's get more complicated, you may want to start thinking about doing this via the DOM, instead.

[Nightmaster]

Still get the same result in output txt file as before, don't see any change.

[Nightmaster]

Sorry for bumping, but I still need a hand here, I tried few other methods that I came up with by searching for similar scripts, but couldn't find a way to fix the problem I'm experiencing. Any help please?