wiki scrape and change of scrape size

[synistics]

I am working with someone elses code and his explanation has not worked and he has not responded in over 7 weeks so I need to ask here, a total newbie, with limited understanding of coding

Breaking it apart, this is a Drupal module for calling information from different sources, one being Wikipedia.

This is code portion that looks to pull the raw:

function createfromweb_operator_wikipedia_execute($query) {
$result = array();

unset($_SESSION['createfromweb']['operator']);

$gurl = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
$gurl.= urlencode('site:wikipedia.org/wiki/ '.$query); // intitle, quotes, ... ?
$gjson = file_get_contents($gurl);
$_SESSION['createfromweb']['operator']['result1'] = $gjson;
$gres = json_decode($gjson);
$url = (string)$gres->responseData->results[0]->unescapedUrl; // get first google match
$rawurl = str_replace(".wikipedia.org/wiki/", ".wikipedia.org/w/index.php?title=", $url);
$rawurl.= "&action=raw&section=0";

The instructions to pull the entire page from Wikipedia are change the last line to:

   $rawurl.="&printable=yes";

This does not work. I have tried the following:

    $rawurl.= "&action=raw";

$rawurl.= "&action&printable=yes";

all three of these modifications return a blank. The value set =0 tells the code to pull the first section of a wikipedia page and manually changing that I can move down the page, but it is not capturing anything else. I also tried to make that a variable or a range, but it broke the pull as well, returning a blank.

The code then goes through some strip and replace functions ending in the creation of the new page:

    $result['title'] = $title;
$result['body'] = trim($body);

with the instructions for $printable=yes the $result['body'] = trim line is to be changed to

   $result['body']

this just returns an error code for the next line of code.

I tried this function, but all it does is sidestep the "trim" functions and drops the limited raw into the page.

$result['body'] = $body;

Thoughts would be appreciated, guidance on this one is a real need.

Thanks for any help.

---

FULL CODE

<?php

// $Id: operator_wikipedia.inc,v 1.2.4.2 2009/02/17 13:38:44 brevity Exp $

/* TODO FormAPI image buttons are now supported.
FormAPI now offers the 'image_button' element type, allowing developers to
use icons or other custom images in place of traditional HTML submit buttons.

$form['my_image_button'] = array(
'#type' => 'image_button',
'#title' => t('My button'),
'#return_value' => 'my_data',
'#src' => 'my/image/path.jpg',
); */

/ TODO New user_mail_tokens() method may be useful.
user.module now provides a user_mail_tokens() function to return an array
of the tokens available for the email notification messages it sends when
accounts are created, activated, blocked, etc. Contributed modules that
wish to make use of the same tokens for their own needs are encouraged
to use this function. /

/ TODO
There is a new hook_watchdog in core. This means that contributed modules
can implement hook_watchdog to log Drupal events to custom destinations.
Two core modules are included, dblog.module (formerly known as watchdog.module),
and syslog.module. Other modules in contrib include an emaillog.module,
included in the logging_alerts module. See syslog or emaillog for an
example on how to implement hook_watchdog.
function example_watchdog($log = array()) {
if ($log['severity'] == WATCHDOG_ALERT) {
mysms_send($log['user']->uid,
$log['type'],
$log['message'],
$log['variables'],
$log['severity'],
$log['referer'],
$log['ip'],
format_date($log['timestamp']));
}
} /

/ TODO Implement the hook_theme registry. Combine all theme registry entries
into one hook_theme function in each corresponding module file.
function operator_wikipedia_theme() {
return array(
);
} /

function createfromweb_operator_wikipedia_name() {
return "some content from wikipedia article";
}

function createfromweb_operator_wikipedia_execute($query) {
$result = array();

unset($_SESSION['createfromweb']['operator']);

$gurl = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
$gurl.= urlencode('site:wikipedia.org/wiki/ '.$query); // intitle, quotes, ... ?
$gjson = file_get_contents($gurl);
$_SESSION['createfromweb']['operator']['result1'] = $gjson;
$gres = json_decode($gjson);
$url = (string)$gres->responseData->results[0]->unescapedUrl; // get first google match
$rawurl = str_replace(".wikipedia.org/wiki/", ".wikipedia.org/w/index.php?title=", $url);
/**$rawurl.= "&action=raw&section=0";*/
$rawurl.= "&action=raw";
/**$rawurl.= "&action=raw";*/

$title = trim(preg_replace("#.*?wikipedia.org/wiki/|_#", " ", $url));

drupal_set_message("getting: ".$url); // " -> ".$rawurl);

$raw = (file_get_contents($rawurl)) or drupal_set_message("could not retrieve wikidata", ERROR);

#$_SESSION['createfromweb']['operator']['result2'] = $raw;

$raw = preg_replace("|<!--(.*?)-->|","",$raw); // no html comments
$raw = preg_replace("/\[\[(.*?)(\|(.*?))?\]\]/", "$1", $raw); // no wiki/alias links
$raw = preg_replace("/\[(.*?) (.*?)\]/", "$1", $raw); // only http urls

preg_match_all('/\{((?>[^{}]+)|(?R))*\}/x', $raw, $boxes);
foreach($boxes[1] as $box) {
	if (preg_match('/\{Infobox/', $box)) {
		$infobox = $box;
		break;
	}
}

#$_SESSION['createfromweb']['operator']['infobox'] = $infobox;
$body = preg_replace('/{{.*?}}/ms','',$raw); //rest w/out boxes
$body = preg_replace("/'''([^']*?)'''/", "<strong>$1</strong>", $body);	
$body = preg_replace("/''([^']*?)''/", "<em>$1</em>", $body);
$infobox = preg_replace("|<br ?/?>|","; ", $infobox);  // html linebreaks to semicolons
$infobox = strip_tags($infobox);	
$result = getBoxProperties($infobox);
$result['title'] = $title;
/**$result['body'] = trim($body);*/
$result['body'] = $body;


$_SESSION['createfromweb']['operator']['result'] = $result;	
//$result = array($result);
return $result;

}

/**
 * [url]http://code.google.com/p/linuxpedia/[/url]
 * Retrieves properties defined in an infobox as an associative array
 *
 * @param   $box    Infobox code
 * @param   $toLower    Whether to convert all predicate keys to lowercase
 * @return  Associative array with predicates as keys
 */
function getBoxProperties($box, $toLower = false) {

    /* Remove outside curly brackets */
    $box = substr($box, 1, strlen($box) - 2);

    /* Remove HTML comments */
    $box = preg_replace('/<\!--[^>]*->/mU', '', $box);

    /* Split triples; ignoring triples in subtemplates */
    $triples = preg_split('/\| (?! [^{]*\}\} | [^[]*\]\] )/x',$box);

    $a = array();

    foreach ($triples as $triple) {
            $predObj = split('=',$triple,2);

            if (count($predObj) == 2 && ($pred = trim($predObj[0])) != "" && ($obj = trim($predObj[1])) != "")
            {
                $key = ($toLower ? strtolower($pred) : $pred);
                $a[$key] = $obj;
            }
    }

    return $a;
}

?>

[shawty]

This is most likley not a PHP issue, but a Google search API issue. Looking at your description:

function createfromweb_operator_wikipedia_execute($query) {
$result = array();

unset($_SESSION['createfromweb']['operator']);

$gurl = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
$gurl.= urlencode('site:wikipedia.org/wiki/ '.$query); // intitle, quotes, ... ?
$gjson = file_get_contents($gurl);
$_SESSION['createfromweb']['operator']['result1'] = $gjson;
$gres = json_decode($gjson);
$url = (string)$gres->responseData->results[0]->unescapedUrl; // get first google match
$rawurl = str_replace(".wikipedia.org/wiki/", ".wikipedia.org/w/index.php?title=", $url);
$rawurl.= "&action=raw&section=0";

The instructions to pull the entire page from Wikipedia are change the last line to:

$rawurl.="&printable=yes";

This does not work. I have tried the following:

The code seems to be creating a URL to pass to google, something like (and I'm making a bit of a guess here :-) )

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=site:wikipedia.org/wiki/myqueryhere

The php call to 'file_get_contents' then uses the url file method, to get that page of data from google.

Once you get to the line following 'file_get_contents' you in effect already have the file data, so quite why the code is messing about adding/removing and transforming another URL I'm not quite sure. It looks to me like the URL your editing is really just for show as i don't see it being used to fetch more data, however if the variable $rawurl is being created to be used then:

$wikipage = file_get_contents($rawurl);

Should fill the variable $wikipage with the wikipedia page content.

[synistics]

What happens in the process is the web person puts in a search item, say Beach Boys. This then reaches out ultimately to wikipedia, so why the google search is there I have never understood.

It then returns the first portion of the wiki page which is controlled with the

$rawurl.= "&action=raw&section=0";

changing the value to 1 gives you the next block on the wiki, but of course, eliminates the first block.

This allows you to populate content on the web page using wikipedia as the source.

The issue is how to adjust the code to take the whole wiki page rather than just the first block, or even add just a range 0-3 for an example to scrape the info.

Thanks again

Synistics

[shawty]

The first thing you need to do is figure out what the URL looks like, if it where me, I'd "print" $rawurl somewhere I could read it, then paste it into my browser address bar and see what I get, then once i have it in the address bar I'd play with the different parameters there, and see what effect they have on the page.

Once you do this, you'll have a better understanding on what each of the options do. I don't know wikipedia's API some i can't help with that part, but i can see that:

$raw = (file_get_contents($rawurl)) or drupal_set_message("could not retrieve wikidata", ERROR);

seems to be the line where the content is grabbed.

As for google, there forcing a search of the keyword specified but only matching in wikipedia, and then ONLY taking the first match on the page, so i'm guessing there using google to perform the search on wikipedia.

[synistics]

I found a note from the code writer regarding the google function

"the wiki page is found via a google search on site:wikipedia.org"

Syn

[shawty]

Pretty much what i thought :-D

[synistics]

From my side, I just simply hacking away at this until I can make it work, but I have ZERO coding experience unless you cound Commodre64 Basic in the 80's.

What I have got going for me is the logic to walk through and understand. Sorry if it sounds like I am looking for a teacher, I am not, just need enough to get onto the next problem.

Down to the question, I get the idea of the print, what would the code look like and I assume I am just printing to an html page

My stab would be

$rawurl.= "&print wiki.html";

BTW Thanks

Synisitics

[synistics]

Tried it and nothing, just a blank output, so that code stab was wrong.

Syn

[shawty]

It doesn't really matter where you print it, you want it to output the variable on your web page where you can see it, if it where me i'd alter my code to something like this:

print($rawurl);
die();

$raw = (file_get_contents($rawurl)) or drupal_set_message("could not retrieve wikidata", ERROR);

This IS going to stop your web-app executing at that point in the script, but it should display in your output somewhere, something similar to this:

http://www.wickipedia.org/w/blah&action=raw&secton=0

Once you have the URL, then copy it into your browser address bar and start playing. change action=raw to action=plain or action=explosion or anything else that might make sense, or try &section=1, section=10 etc.

At this stage, you need to figure out exactly what the address is that PHP is using, and how, do a google see if you can find any docs on wikipedia urls.

[synistics]

Right code in and search for ABC, the American Broadcasting Corporation, the result is:

http://en.wikipedia.org/w/index.php?title=American_Broadcasting_Company&action=raw&section=0

As you can see it gives me the index.php and this is the notepad output of that:

{{Distinguish2|the [[Australian Broadcasting Corporation]]}}
{{Infobox Network
|network_name = American Broadcasting Company (ABC)
|network_logo = [[Image:American Broadcasting Company Logo 2007.png|170px]]
|country = [[United States]]
|network_type = [[Terrestrial television|Broadcast]] [[radio network]] and <br />[[television network]]
|available = [[United States]], [[Canada]], [[Mexico]], [[United Kingdom]], the [[Philippines]], [[Japan]], [[South Korea]], [[Germany]], [[Italy]], [[France]], [[Belgium]], [[Spain]], [[Iceland]], [[Hungary]], [[Sweden]], [[Russia]], [[Poland]], [[Israel]], [[Iraq]], and the [[Caribbean]].
|slogan = ''Start Here''
|owner = [[The Walt Disney Company]]
|parent = [[Disney-ABC Television Group]]
|key_people = [[Robert Iger]], CEO of The Walt Disney Company<br />[[Anne Sweeney]], co-chair Disney Media Networks and President, Disney-ABC Television Group<br />[[Anton Vassilenko]], Media, ABC Entertainment
|launch_date = October 12, 1943 (radio network)<br />April 19, 1948 (television network)
|founder = (as independent company) <br />[[Edward Noble]]
|past_names = NBC [[Blue Network]]
|brand = "America's Broadcasting Company", <br />"This is ABC"
|website = [http://abc.go.com/ www.abc.com]|
}}

The '''American Broadcasting Company''' ('''ABC''') is an [[United States|American]] [[television network]]. Created in 1943 from the former [[NBC]] [[Blue Network|Blue radio network]], ABC is owned by [[The Walt Disney Company]] and is part of [[Disney-ABC Television Group]]. It first broadcast on television in 1948. Corporate headquarters are in [[Manhattan]] in [[New York City]], while programming offices are in [[Burbank, California|Burbank]], [[California]] adjacent to the [[Walt Disney Studios (Burbank)|Walt Disney Studios]] and the [[Walt Disney Company]] corporate headquarters.

The current slogan is "Start Here", except television idents continue to use the former "America's Broadcasting Company" slogan. Before the "America's Broadcasting Company" slogan, television idents used the full corporate name from 1997-2000.

The formal name of the operation is '''American Broadcasting Companies, Inc.''', and that name appears on copyright notices for its in-house network productions and on all official documents of the company, including paychecks and contracts. A separate entity named '''ABC Inc.''', formerly Capital Cities/ABC Inc., is that firm's direct parent company, and that company is owned in turn by Disney. The network is sometimes referred to as the '''Alphabet Network''', due to the letters "ABC" being the first three letters of the Latin alphabet, in order.

So it would appear it is having an issue with generic things, I tried a specific, The Beatles

http://en.wikipedia.org/w/index.php?title=The_Beatles&action=raw&section=0

again the index.php which is giving me this:

{{use dmy dates}}{{semiprotected|small=yes}}
{{otheruses4|the band}}
<!-- PLEASE read the standards for this infobox before editing it -->
{{Infobox musical artist
| Name = The Beatles<!-- PLEASE discuss at Talk:The Beatles#Beatles logo in infobox before adding logo -->
| Img = The Fabs.JPG
| Img_capt = The Beatles in 1964.<br />'''Top:''' [[John Lennon]], [[Paul McCartney]]<br />'''Bottom:''' [[George Harrison]], [[Ringo Starr]]
| Img_size = 220
| Background = group_or_band
| Alias = "The Fab Four," "The Lads from Liverpool"
| Origin = [[Liverpool]], England
| Genre = [[Rock music|Rock]], [[Pop music|pop]]
| Musical_style = [[Rock music|Rock]], [[Pop Rock]]<!-- This is the consensus listing, otherwise there would be very many genres listed and fought over. Please change consensus before adding to this list. -->
| Years_active = 1960–1970
| Label = [[EMI]]<br>[[Parlophone Records|Parlophone]]<br>[[Capitol Records|Capitol]]<br>[[Odeon Records|Odeon]]<br>[[Apple Records|Apple]]<br>[[Vee-Jay Records|Vee-Jay]]<br>[[Polydor Records|Polydor]]<br>[[Swan Records|Swan]]<br>[[Tollie Records|Tollie]]<br>[[United Artists Records]]
| Associated_acts = [[The Quarrymen]]<br>[[Plastic Ono Band]]<br>[[The Dirty Mac]]<br>[[Wings (band)|Wings]]<br>[[Traveling Wilburys]]<br>[[Rory Storm and the Hurricanes]]<br>[[Ringo Starr and His All-Starr Band]]
| URL = [http://www.beatles.com/ www.beatles.com]
| Current_members = [[John Lennon]]<br />[[Paul McCartney]]<br />[[George Harrison]]<br />[[Ringo Starr]]
| Past_members = [[Pete Best]]<br />[[Stuart Sutcliffe]]
}}
'''The Beatles''' were a [[rock music|rock]] and [[pop music|pop]] band from [[Liverpool]], England that formed in 1960. During their career, the group primarily consisted of [[John Lennon]] (rhythm guitar, vocals), [[Paul McCartney]] (bass guitar, vocals), [[George Harrison]] (lead guitar, vocals) and [[Ringo Starr]] (drums, vocals). Although their initial musical style was rooted in 1950s [[rock and roll]] and [[skiffle]], the group worked with different musical [[music genre|genres]], ranging from [[Tin Pan Alley]] to [[psychedelic rock]]. Their clothes, style and statements made them trend-setters, while their growing social awareness saw their influence extend into the social and cultural revolutions of the 1960s. After the band broke up in 1970, all four members embarked upon successful solo careers.

The Beatles were one of the most commercially successful and critically acclaimed bands in the [[history of music|history of popular music]], [[List of best-selling music artists#1 billion records or more|selling over one billion records]] internationally.<ref name="Worldwide sales">{{cite web|url=http://www.emigroup.com/NR/exeres/60E71CC3-405A-42F4-A013-1B20496FAE04.htm|title=1960-1969|publisher=EMI Group Ltd.|accessdate=31 May 2008}}</ref> In the United Kingdom, The Beatles released more than 40 different [[The Beatles discography#Singles|singles]], [[The Beatles discography|albums]], and [[The Beatles discography#Extended plays (EPs)|EPs]] that reached [[UK Singles Chart|number one]], earning more number one albums (15) than any other group in UK chart history. According to the [[Recording Industry Association of America]], The Beatles have sold more albums in the United States than any other band.<ref name="riaa">{{cite web |url=http://www.riaa.com/newsitem.php?news_year_filter=1999&resultpage=2&id=3ABF3EC8-EF5B-58F9-E949-3B57F5E313DF |title=The American Recording Industry Announces its Artists of the Century |publisher=Recording Industry Association of America (RIAA) |date=10 November 1999|accessdate=26 June 2007}}</ref> In 2004, ''[[Rolling Stone]]'' magazine ranked The Beatles number one in its list of 100 Greatest Artists of All Time.<ref>{{cite web| title = The Immortals: The First Fifty| work = Rolling Stone Issue 946| publisher = Rolling Stone| url=http://www.rollingstone.com/news/story/5939214/the_immortals_the_first_fifty | accessdate = 8 July 2008 }}</ref> According to that same magazine, The Beatles' innovative music and cultural impact helped define the 1960s, and their influence on pop culture is still evident today. In 2008, ''[[Billboard (magazine)|Billboard]]'' magazine released a list of top-selling Hot 100 artists to celebrate the chart's fiftieth anniversary; The Beatles topped it.<ref>{{cite web | url = http://www.billboard.com/bbcom/specials/hot100/charts/top100-artists-20.shtml | title = The Billboard Hot 100 All-Time Top Artists (20-01) | date = 11 September 2008 | accessdate = 13 September 2008}}</ref>

when I swap out the rawurl for the $printable=yes this is the output:

http://en.wikipedia.org/w/index.php?title=The_Beatles&printable=yes

which gives me a full readable page in a web browser.

next step?

Syn

[shawty]

Ok, after going through the code i can see nothing wrong with it, and after looking at the URL's produced, they both produce content, generated from wikipedia, which does get returned by the function.

You likley have one of two scenarios.

A) Wikipedia have changed the raw/section and other web parameters, and without the wikipedia docs, it's impossible to know what should be returned and how.

😎 Wikipedia have changed the format of the page so that it no longer comes back as what you expect, or they only deem that is all the information your going to be allowed to obtain.

If you are expecting the function to return raw HTML (Which I'm guessing is what you think should have been returned) then the closest iv'e been able to get is to change

$raw = (file_get_contents($rawurl)) or drupal_set_message("could not retrieve wikidata", ERROR);

to

$raw = (file_get_contents($url)) or drupal_set_message("could not retrieve wikidata", ERROR);

This then retrieves just the plain wikipedia article without any special tags or other special stuff, it also doesn't transform the URL it just basicly sends the address in a standard plain form, EG:

http://en.wikipedia.org/wiki/Cheddar_cheese

There does still look to be some custom tags in there, such as words surrounded by [[ & ]] , which I'm guessing are specific to wikipedia.

I really don't think however there is any faults with the code, it really does seem to be a wikipedia thing.

My next step if it was me, would be to have a dig round on wikipedia itself and see if i could turn up any documentation on the page url formats and such like.

[synistics]

Worked with your suggestion, no go, blank output.

I played with the print line and worked my way down through the code looking for a place to break. It continued to work, and it should it was printing the rawurl.

I put in print(result) and it of course returned the word ARRAY which is close to the beginning of the code.

If I try and print $result= trim body I get a Wiki error page, if I try print the session line, same error page.

I need to pop out for a couple of hours, can to screen captures but I think it is in the module fill, see the zip attached

http://drupal.org/project/createfromweb

http://ftp.drupal.org/files/projects/createfromweb-6.x-1.3.tar.gz

[shawty]

I put in print(result) and it of course returned the word ARRAY which is close to the beginning of the code.

You'll find that it printed that beacuse it was an array you where trying to print.

try using

print_r($result);

Instead. If it is an array, then that will print out the elements.

Don't worry about the popping out thing... :-) It's nearly midnight here in the UK, i'll be gone by you get back :-)

[synistics]

Well I want to thank you for the help.

On some other occassion we can talk about the UK TV I watch, been a bit of a weak season over there since Christmas. Add to that the weak US TV and I spend a lot of time in front of the computer.

I am off to give that instructions a go, will post the outcome and perhaps talk with you tomorrow.

Syn

[synistics]

Array ( [title] => Neil Young [body] =>
Wikimedia Foundation
Error

English

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.

You may be able to get further information in the #wikipedia channel on the Freenode IRC network.

The Wikimedia Foundation is a non-profit organisation which hosts some of the most popular sites on the Internet, including Wikipedia. It has a constant need to purchase new hardware. If you would like to help, please donate.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: GET http://en.wikipedia.org/w/index.php?title=Neil_Younghttp://en.wikipedia.org/wiki/Neil_Young&printable=yes, from 99.198.111.18 via sq25.wikimedia.org (squid/2.7.STABLE6) to ()
Error: ERR_ACCESS_DENIED, errno No at Tue, 26 May 2009 02:37:44 GMT

And that in multilanguage

so print_r failed

[shawty]

Not quite Syn....

Wikipedia failed.

:-)

So pretty much as i guessed, the problems your having are not with your code, but something to do with wikipedia.

outside of PHPBuilder... you can find my blog here:
http://cid-4515677bdf99b35f.spaces.live.com/default.aspx?wa=wsignin1.0&sa=522026211

[synistics]

I popped over to the blog, will catch up with you off line later.

Travel day(s) for me here, haul ass across the continent and back again in 24 hours!

I am just having a look at the project while I have a pint in the airport!

Catch up with you later and again, thanks for the help.

Syn