Searchengine that will index drop-downs

Anon

Hi all,

I'm looking for a search engine that will index the links I have on a site in drop-down boxes. So far all the scripts I have tried simply ignore the drop-down menu links and do not spider the pages linked in there.

Does anyone know of such search engine that will allow this? The drop-down links/content is updated dynamically so it would be a crazy job to enter an url for spidering one by one...and then have to do this every time something changes...

Any tips would be welcome.
Thanks,

Stepan

[deleted]

"Does anyone know of such search engine that will allow this?"

nope.

"The drop-down links/content is updated dynamically so it would be a crazy job to enter an url for spidering one by one..."

it would be equally crazy to create a script that will read a form and start trying to fill it out, hoping that it will somehow end up on another page. :-)

Anon

Hi!

What is the syntax of the URL links you use in there? Is it the valid syntax of type "http://" + short uri? If you only have www.hotmail.com, that could be a problem.

According to logic, the indexing script shouldn't care whether it is reading a <a href=""> type of link or a link inside your onSelect or onSubmit javascript parameter. It deals with the page as pure text, so I disagree with Vincent on point 2: every indexing engine should do this. You can try to trick it by perhaps listing all the links separately in another place, perhaps in the form:
<a href="http://www.hotmail.com"></a> or something like this. This could be done at the very bottom of the page - or perhaps - and I think it's a rather clever solution - in an invisible IE layer which is bounded by <div> tags and is listed at the very bottom of the page.

Hope it helps,
Stas

Anon

This is the syntax..the urls are quite long.

Stanislav Okhvat wrote:

Hi!

What is the syntax of the URL links you use in there? Is it the valid syntax of type "http://" + short uri? If you only have www.hotmail.com, that could be a problem.

Hope it helps,
Stas

Anon

Hi!

It seems that when the forum saves the thread to the database, it strips some of the HTML.

The options should be of this type:

<select name=site SIZE=1 onChange="formHandler(this.form);" class=menu>
<option value="http://www.servver.com/cgi-bin/parse.cgi?type=012&titel=News">News Title</option>
</select>

If this doesn't work, try the second approach I suggested.

Best regards,
Stas

[deleted]

"According to logic, the indexing script shouldn't care whether it is reading a <a href=""> type of link or a link inside your onSelect or onSubmit javascript parameter"

Which logic? :-)

Tiny problem: javascript must be analyzed to find out what it does with the values in the option tags. Now I'm sure there are a lot of nifty searchengines about, but I don't think any of them will bother to load the page and see what happens when every possible combination of form buttons/dropdowns and whatnot are activated.

Anon

Hi, Vincent!

According to my logic, that is ... :-)

I see your concern, but Stepan has submitted clear HTML source for these dropdowns, and the links follow the URI syntax. The search engine does not navigate the page being analyzed in a graphical web browser - it pulls the raw source of the page, analyzes it, perhaps via regex, finds all links on the page and starts new indexing sessions for these URIs. If the URI is valid, it should follow it. Why do you think many webmasters put the following right before the closing </BODY> tag:
<a href="http://www.somesite.com/indextop.php"> </a>
This makes the web crawler analyze a specially prepared "top" index which has more meta information than the main index.

I understand that perhaps this is because the webcrawler searches for <a href=" and </a> delimiters before spinning off a new request. But it is possible to fix this by making an invisible layer, just as I said. This layer will not be processed by old browsers and will still present the info in the way a web spider expects it.

Best regards,
Stas

[deleted]

My only concern is that searchengines will not read FORMs or Javascript to find new links, because submitting forms can seriously f*ck up the website (imagine a searchengine trying to find new links by filling out a three-page questionnaire in all possible ways...) and javascript is just too complex to analyze for 2.07 billion pages.

I agree that any other time of link, visible or invisible, hidden or not, will be found and processed, but only those that are clearly links, as in an A HREF tag.

Anon

Hi, Vincent!

Frankly, I am clueless as to how the search engines work (only guessing), but if I were the one to develop it (:-), I'd too look for <a></a> delimiters. Less headache.

It still seems to me they could at least handle the case Stepan is worrying his head off about. Sounds logical. Just another alternative branch in the regex sounds pretty easy to implement and wouldn't throw off their servers.

Best,
Stas

Anon

actually, I've found an engine that filters anything as a link that starts with http://
just what I needed...

🙂

thanks for all input though.

stepan