filter_var results a bit weird for FILTER_VALIDATE_URL

[Weedpacket]

That's why FILTER_FLAG_SCHEME_REQUIRED/FILTER_FLAG_HOST_REQUIRED don't do anything.

diff museum.php.net/php5/php-5.2.0/ext/filter/logical_filters.c museum.php.net/php5/php-5.2.1/ext/filter/logical_filters.c > diff.txt

...
488,491c457,460
<	if (
< 		((flags & FILTER_FLAG_SCHEME_REQUIRED) && url->scheme == NULL) ||
< 		((flags & FILTER_FLAG_HOST_REQUIRED) && url->host == NULL) ||
< 		((flags & FILTER_FLAG_PATH_REQUIRED) && url->path == NULL) ||
< 		((flags & FILTER_FLAG_QUERY_REQUIRED) && url->query == NULL)
<	) {
<bad_url:
<		php_url_free(url);
<		RETURN_VALIDATION_FAILED
---
	if (
		url->scheme == NULL || 
		/* some schemas allow the host to be empty */
		(url->host == NULL && (strcmp(url->scheme, "mailto") && strcmp(url->scheme, "news") && strcmp(url->scheme, "file"))) ||
		((flags & FILTER_FLAG_PATH_REQUIRED) && url->path == NULL) || ((flags & FILTER_FLAG_QUERY_REQUIRED) && url->query == NULL)
	) {
bad_url:
		php_url_free(url);
		RETURN_VALIDATION_FAILED

Thanks to Ilia: https://bugs.php.net/bug.php?id=39898

They're not "on by default", they're hardcoded and factored out.

Absolute URIs only, please. I insist.

[sneakyimp]

Weedpacket;11061897 wrote:

That section doesn't have anything to do with the presence of [font=monospace]////[/font]....

In my first round of tests, filter_var is flunking //example.com/:

// bool(false)
var_dump(filter_var("//cdn.example.com/js_file.js", FILTER_VALIDATE_URL, FILTER_FLAG_HOST_REQUIRED | FILTER_FLAG_PATH_REQUIRED));

Weedpacket;11061897 wrote:

There seem to be quite a few problems with FILTER_VALIDATE_URL.

I'd like to gingerly approach the PHP devs about fixing it. Perhaps we can make our own list of flagrant problems.

Derokorian;11061903 wrote:

It says in the docs that SCHEME and HOST are both applied by default.

Wondering a) if flags actually work and b) is it possible to ignore the scheme? I think flags may not work. I'm mistrustful of docs.

Derokorian;11061903 wrote:

In fact we can see here that those constants aren't even referenced while path and query are. If scheme or host are not present, it fails.

PHP source code relies very heavily on macros and and its parameter parsing code is pretty funky but yeah, I don't see those constants referenced in that function's source.

Weedpacket;11061907 wrote:

Yes; and for the other half, RFC3986 notes among its lists of changes that its predecessor didn't allow [font=monospace]about:[/font], and that it had trouble distinguishing "host" and path.

Really it shouldn't be too hard to translate the ABNF of 3986 into a parser to replace whatever PHP's using now (just checked: the filter basically calls [man]url_parse[/man], which is hand-rolled; https://bugs.php.net/bug.php?id=72301 is right about there being a re2c-based parser and how nothing seems to use it). Attempting to change it might cause squawks about BC breakage, but I suspect that it's not that well-used as it is so won't be breaking anything that is not probably broken already.

Thanks for this diligence, although I'm not exactly sure parsers and BC breakage.

Weedpacket;11061907 wrote:

Wow:
I'm not seeing something.
BRB

:eek:

Weedpacket;11061909 wrote:

That's why FILTER_FLAG_SCHEME_REQUIRED/FILTER_FLAG_HOST_REQUIRED don't do anything.

diff museum.php.net/php5/php-5.2.0/ext/filter/logical_filters.c museum.php.net/php5/php-5.2.1/ext/filter/logical_filters.c > diff.txt

...
488,491c457,460
<	if (
< 		((flags & FILTER_FLAG_SCHEME_REQUIRED) && url->scheme == NULL) ||
< 		((flags & FILTER_FLAG_HOST_REQUIRED) && url->host == NULL) ||
< 		((flags & FILTER_FLAG_PATH_REQUIRED) && url->path == NULL) ||
< 		((flags & FILTER_FLAG_QUERY_REQUIRED) && url->query == NULL)
<	) {
<bad_url:
<		php_url_free(url);
<		RETURN_VALIDATION_FAILED
---
	if (
		url->scheme == NULL || 
		/* some schemas allow the host to be empty */
		(url->host == NULL && (strcmp(url->scheme, "mailto") && strcmp(url->scheme, "news") && strcmp(url->scheme, "file"))) ||
		((flags & FILTER_FLAG_PATH_REQUIRED) && url->path == NULL) || ((flags & FILTER_FLAG_QUERY_REQUIRED) && url->query == NULL)
	) {
bad_url:
		php_url_free(url);
		RETURN_VALIDATION_FAILED

Thanks to Ilia: https://bugs.php.net/bug.php?id=39898

They're not "on by default", they're hardcoded and factored out.

Absolute URIs only, please. I insist.

This is not cool. I suppose I'll send an email to PHP devs about this now ten-year-old bug. This function looks pretty broken to me. If anyone has any requests or wants to chime in on what we would like from this function, I'll try and roll it into a polite email calling for some action.

[Weedpacket]

Hah, it looks like Derokorian and I managed to be in the same place at the same time. I took longer to write things up.

First of all, I think a working filter should validate against RFC3986: it's not just a cosmetic difference, as there are problems with RFC2396 that the former spec calls out.

Validating against RFC3986 will allow specifying that the filter validates URIs, not merely URI-references, which is probably what was intended.

The validation rule becomes "all URIs have a scheme part" (and in fact that's the only part it requires), followed by further checks such as "if the scheme is followed by '//' then an authority part must be present", and so forth.

It's not the place for a URL validator to pick and choose based on specific schemes (http: vs. file: vs. mailto: etc.). The "decision made" in the bug tracker:

[e-mail] wrote:

- remove option of having host & scheme optional
- No option to make them optional (what you have is not a url then)

is itself refuted in Illia's fix (Oops, sometimes a URL doesn't have a host part! Quick, bodge a special case! Have we got them all?)

Yes, maybe 'Every developer expects "blahblub" to be not valid.' but the fact is that RFC3986 allows an empty string as a valid URI-reference. Picking and choosing about which bits to accept or deny isn't going to produce anything intuitive because it would require the user to intuit which bits the implementer chose.

The problem with validating a URI-reference is that it could be a relative URI. By definition, such a beast can't be validated without resolving it against the base URI it is supposed to be relative to. So if relative URIs are to be accepted, it would be necessary for the user to pass a suitable base URI ([noparse]http://www.example.com/user[/noparse], [noparse]mailto:[/noparse], or whatever), and apply the RFC's resolution rules, failing the input string if resolution fails to produce a valid URI.

Any URI that passes the grammar test should not be invalidated on additional grounds if the scheme is unrecognised by the implementer (if they don't recognise the scheme, how do they know what's valid and what's not?) If the implementer does recognise the scheme, then additional checks may be performed. That means implementing the relevant sections of [noparse]RFC7230 (http🙂, RFC6068 (mailto🙂, RFC8089 (file🙂 RFC5538 (news🙂[/noparse] and so on, depending on how energetic the implementer feels. The more esoteric the scheme, the more likely the user is in a better position to validate, and can use FILTER_CALLBACK for further refinement. It's not up to the implementer to discard a "pkcs11:" URI (RFC7512) just because it doesn't include a host part and they've never personally seen one themselves.

I think the strongest (most useful and least surprising) filter would be to make it validate against RFC3986's "URI" production (basically, absolute-URI with optional #fragment). More generic would be FILTER_FLAG_RELATIVE_URI allowing URI-references ("blahblub", "localhost", "" and all). Being able to provide a base URI the filter can resolve them against would allow validating URI-references as stringently as absolute URIs.


This has given me an itch to scratch.

[sneakyimp]

Weedpacket, that all sounds right to me -- what I understand of it anyway. Please tell me you are writing this function 😃

[Weedpacket]

sneakyimp;11061969 wrote:

Please tell me you are writing this function 😃

Well, since you mentioned it... here's an experimental thing to see what sort of work it would entail.

<?php
namespace URI;

class URIException extends \DomainException
{
}

class URI
{

public static function validate(string $str): bool
{
	try
	{
		$parse = static::parse($str);
	}
	catch(URIException $e)
	{
		return false;
	}
	return true;
}

protected $scheme;

protected $userinfo;

protected $host;

protected $port;

protected $path;

protected $query;

protected $fragment;

private static function bad_pct_encoded($str): bool
{
	// Look for a % not followed by two hexdigits.
	return (bool)preg_match('/%(?![A-F0-9]{2})/i', $str);
}

private static function proper_path(string $path): bool
{
	if($path[0] == '/')
	{
		$path = substr($path, 1);
	}
	$parts = explode('/', $path);
	if($parts[0] == '')
	{
		return false;
	}
	foreach($parts as $part)
	{
		if(!self::bad_pct_encoded($part))
		{
			return false;
		}
	}
	return true;
}

private static function proper_host(string $host): bool
{
	// Regular name
	if(!preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $host) && !self::bad_pct_encoded($host))
	{
		return true;
	}
	// IPv4 literal
	if(preg_match('/^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$/', $host, $match))
	{
		return !($match[1] > 255 || $match[2] > 255 || $match[3] > 255 || $match[4] > 255);
	}
	// Other IP address literals
	if($host[0] != '[' || $host[-1] != ']')
	{
		return false;
	}
	$host = substr($host, 1, -1);
	// IPvFuture
	if($host[0] == 'v' || $host[0] == 'V')
	{
		return preg_match('/^v[0-9a-f]+[!$&\'()*+,\-.0-9:;=_a-z~]+$/i', $host);
	}
	// IPv6
	// Regex from http://home.deds.nl/~aeron/regex/
	return (bool)preg_match('/^(((?=.*(::))(?!.*\3.+\3))\3?|([\dA-F]{1,4}(\3|:\b|$)|\2))(?4){5}((?4){2}|(((2[0-4]|1\d|[1-9])?\d|25[0-5])\.?\b){4})$/i', $host);
}

private static function parse_authority(string $authority): array
{
	// Peel any userinfo off start of authority.
	if(preg_match('/^([a-z0-9\-._~!$&\'()*+,;=:%]*)@/i', $authority, $match))
	{
		$authority = substr($authority, strlen($match[0]));
		$userinfo = $match[1];
	}
	else
	{
		$userinfo = null;
	}
	if(!preg_match('/^[!$%&\'()*+,\-.0-9:;=\[\]_a-z~]*$/i', $authority))
	{
		throw new URIException('Invalid authority part');
	}
	// Peel any port off end of authority
	if(preg_match('/(?<!:):([0-9]+)$/', $authority, $match))
	{
		$host = substr($authority, 0, -strlen($match[0]));
		// The port is a number...
		$port = (int)$match[1];
		if($port > 65535)
		{
			throw new URIException('Invalid port number');
		}
		// ...But is represented by a string of digits.
		$port = (string)$port;
	}
	else
	{
		$host = $authority;
		$port = null;
	}

	if($userinfo !== null && (self::bad_pct_encoded($userinfo) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $userinfo)))
	{
		throw new URIException('Invalid userinfo part');
	}
	if(!self::proper_host($host))
	{
		throw new URIException('Invalid host part');
	}

	return [$userinfo, $host, $port];
}

public static function parse(string $str): URI
{

	// Remove this test if and when IRIs are implemented.
	if(preg_match('/[^ -~]/', $str))
	{
		throw new URIException('URI supports ASCII only');
	}

	// Initial parsing regexp from RFC3986 Appendix B
	// Slightly modified to make scheme mandatory (we only deal with
	// absolute URIs, not URI-references)
	if(!preg_match('~^([^:/?#]+):(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?~', $str, $match))
	// parentheses---1--------1-2--3-------32-4------45--6-----65-7-8--87--
	{
		throw new URIException('Invalid URI');
	}

	$scheme = $match[1];
	$authority = isset($match[2]) ? $match[3] : null;
	$path = $match[4];
	$query = isset($match[5]) ? $match[6] : null;
	$fragment = isset($match[7]) ? $match[8] : null;

	if(!preg_match('/^[a-z][a-z0-9\-+.]*$/i', $scheme))
	{
		throw new URIException('Invalid scheme component');
	}
	if($query !== null && (self::bad_pct_encoded($query) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $query)))
	{
		throw new URIException('Invalid query component');
	}
	if($fragment !== null && (self::bad_pct_encoded($fragment) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $fragment)))
	{
		throw new URIException('Invalid fragment component');
	}

	if($authority !== null && $authority !== '') // hmm...
	{
		[$userinfo, $host, $port] = self::parse_authority($authority);
	}
	else
	{
		$userinfo = $host = $port = null;
		if(!($path == '' || $path == '/' || !self::proper_path($path)))
		{
			throw new URIException('Invalid path component');
		}
	}

	return new self($scheme, $userinfo, $host, $port, $path, $query, $fragment);
}

private static function canonical_pct_encoding($str)
{
	return ($str === null) ? null : preg_replace_callback('/%[0-9a-f]{2}/i', function ($s)
	{
		return strtoupper($s[0]);
	}, $str);
}

protected function __construct($scheme, $userinfo, $host, $port, $path, $query, $fragment)
{
	// Scheme names are canonically lower case. (RFC3986 §3.1, para.2)
	$scheme = strtolower($scheme);
	// Host names are canonically lower case (§3.2.2) apart from %-encoding
	if($host !== null)
	{
		$host = strtolower($host);
	}
	// %-encoding canonically uses uppercase A-F (§2.1)
	$userinfo = self::canonical_pct_encoding($userinfo);
	$query = self::canonical_pct_encoding($query);
	$fragment = self::canonical_pct_encoding($fragment);
	$host = self::canonical_pct_encoding($host);
	$path = self::canonical_pct_encoding($path);

	$this->scheme = $scheme;
	$this->userinfo = $userinfo;
	$this->host = $host;
	$this->port = $port;
	$this->path = $path;
	$this->query = $query;
	$this->fragment = $fragment;
}

public function __toString()
{
	$uri = $this->scheme . ':';
	if($this->host !== null)
	{
		$uri .= '//';
		if($this->userinfo !== null)
		{
			$uri .= $this->userinfo . '@';
		}
		$uri .= $this->host;
		if($this->port !== null)
		{
			$uri .= ':' . $this->port;
		}
	}
	$uri .= $this->path;
	if($this->query !== null)
	{
		$uri .= '?' . $this->query;
	}
	if($this->fragment !== null)
	{
		$uri .= '#' . $this->fragment;
	}
	return $uri;
}
}


class HttpUri extends URI
{

// http-URI = "http:" "//" authority path-abempty [ "?" query ] [ "#"
// fragment ]
public function parse(string $str): HttpUri
{
	parent::parse($str);
	// We have to have an authority (rather, a host, because we've parsed
	// userinfo and port already).
	if($this->scheme !== 'http' || $this->scheme !== 'https')
	{
		throw new URIException('scheme part does not identify http(s) URI');
	}
	if($this->host === null)
	{
		throw new UriException('http(s) URIs require an authority part');
	}
}
}

[sneakyimp]

Weedpacket;11062021 wrote:

Well, since you mentioned it... here's an experimental thing to see what sort of work it would entail.

Amazing!

Trying to use it (PHP 7.0.15-0ubuntu0.16.04.4) like so:

require_once "weedpacket.php";

$urls = array(
        "",
        "Buy It Now",
        "localhost/foo/bar",
        "blarg",
        "blarg/",
        "blarg/some/path/file.ext",
        "http://google.com",
        "http://google.com/",
        "http://google.com/some/path.ext",
        "http://google.com/some/path.ext?foo=bar",
        "example.com",
        "example.com/",
        "example.com/some/path/file.ext",
        "example.com/some/path/file.ext?foo=bar",
        "example.com:1234",
        "example.com:1234/",
        "example.com:1234/some/path/file.ext",
        "example.com:1234/some/path/file.ext?foo=bar",
        "//foobar.com",
        "//foobar.com/",
        "//foobar.com/path/file.txt",
        "//cdn.example.com/js_file.js"

);

foreach($urls as $u) {
	echo "url: $u\n";
	try {
		$v = URI\URI::parse($u);
		echo "OK?\n";
	} catch (Exception $e) {
		echo "EXCEPTION: " . $e->getMessage() . "\n";
	}
	echo "\n\n"; 
}

yields some errors:

PHP Parse error:  syntax error, unexpected '=' in /home/jaith/2017-05-16-url-check/weedpacket.php on line 179

Change that line to this:

list($userinfo, $host, $port) = self::parse_authority($authority);

solves the problem (at least temporarily -- not sure my change preserves the operation intended?

But then I get this error:

PHP Fatal error:  Cannot make static method URI\URI::parse() non static in class URI\HttpUri in /home/jaith/2017-05-16-url-check/weedpacket.php on line 275

The problem seems to be that URI::parse is a static function and called statically, whereas HttpUri::parse is not static and refers to $this so it cannot be static.

I modified the HttpUri class to try and remedy the situation:

class HttpUri extends URI
{

// http-URI = "http:" "//" authority path-abempty [ "?" query ] [ "#"
// fragment ]
public static function parse(string $str): HttpUri
{
    $v = parent::parse($str);
    // We have to have an authority (rather, a host, because we've parsed
    // userinfo and port already).
    if($v->scheme !== 'http' || $v->scheme !== 'https')
    {
        throw new URIException('scheme part does not identify http(s) URI');
    }
    if($v->host === null)
    {
        throw new UriException('http(s) URIs require an authority part');
    }
}
}

But this leads to complaints about the type hinting:

PHP Fatal error:  Declaration of URI\HttpUri::parse(string $str): URI\HttpUri must be compatible with URI\URI::parse(string $str): URI\URI in /home/jaith/2017-05-16-url-check/weedpacket.php on line 275

remove type hint from URI::parse and things seem better.

The output:

url: 
EXCEPTION: Invalid URI. Preliminary regex failed.


url: Buy It Now
EXCEPTION: Invalid URI. Preliminary regex failed.


url: localhost/foo/bar
EXCEPTION: Invalid URI. Preliminary regex failed.


url: blarg
EXCEPTION: Invalid URI. Preliminary regex failed.


url: blarg/
EXCEPTION: Invalid URI. Preliminary regex failed.


url: blarg/some/path/file.ext
EXCEPTION: Invalid URI. Preliminary regex failed.


url: http://google.com
OK?


url: http://google.com/
OK?


url: http://google.com/some/path.ext
OK?


url: http://google.com/some/path.ext?foo=bar
OK?


url: example.com
EXCEPTION: Invalid URI. Preliminary regex failed.


url: example.com/
EXCEPTION: Invalid URI. Preliminary regex failed.


url: example.com/some/path/file.ext
EXCEPTION: Invalid URI. Preliminary regex failed.


url: example.com/some/path/file.ext?foo=bar
EXCEPTION: Invalid URI. Preliminary regex failed.


url: example.com:1234
OK?


url: example.com:1234/
OK?


url: example.com:1234/some/path/file.ext
OK?


url: example.com:1234/some/path/file.ext?foo=bar
OK?


url: //foobar.com
EXCEPTION: Invalid URI. Preliminary regex failed.


url: //foobar.com/
EXCEPTION: Invalid URI. Preliminary regex failed.


url: //foobar.com/path/file.txt
EXCEPTION: Invalid URI. Preliminary regex failed.


url: //cdn.example.com/js_file.js
EXCEPTION: Invalid URI. Preliminary regex failed.

[sneakyimp]

The modified code:

namespace URI;

class URIException extends \DomainException
{
}

class URI
{

public static function validate(string $str): bool
{
    try
    {
        $parse = static::parse($str);
    }
    catch(URIException $e)
    {
        return false;
    }
    return true;
}

protected $scheme;

protected $userinfo;

protected $host;

protected $port;

protected $path;

protected $query;

protected $fragment;

private static function bad_pct_encoded($str): bool
{
    // Look for a % not followed by two hexdigits.
    return (bool)preg_match('/%(?![A-F0-9]{2})/i', $str);
}

private static function proper_path(string $path): bool
{
    if($path[0] == '/')
    {
        $path = substr($path, 1);
    }
    $parts = explode('/', $path);
    if($parts[0] == '')
    {
        return false;
    }
    foreach($parts as $part)
    {
        if(!self::bad_pct_encoded($part))
        {
            return false;
        }
    }
    return true;
}

private static function proper_host(string $host): bool
{
    // Regular name
    if(!preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $host) && !self::bad_pct_encoded($host))
    {
        return true;
    }
    // IPv4 literal
    if(preg_match('/^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$/', $host, $match))
    {
        return !($match[1] > 255 || $match[2] > 255 || $match[3] > 255 || $match[4] > 255);
    }
    // Other IP address literals
    if($host[0] != '[' || $host[-1] != ']')
    {
        return false;
    }
    $host = substr($host, 1, -1);
    // IPvFuture
    if($host[0] == 'v' || $host[0] == 'V')
    {
        return preg_match('/^v[0-9a-f]+[!$&\'()*+,\-.0-9:;=_a-z~]+$/i', $host);
    }
    // IPv6
    // Regex from http://home.deds.nl/~aeron/regex/
    return (bool)preg_match('/^(((?=.*(::))(?!.*\3.+\3))\3?|([\dA-F]{1,4}(\3|:\b|$)|\2))(?4){5}((?4){2}|(((2[0-4]|1\d|[1-9])?\d|25[0-5])\.?\b){4})$/i', $host);
}

private static function parse_authority(string $authority): array
{
    // Peel any userinfo off start of authority.
    if(preg_match('/^([a-z0-9\-._~!$&\'()*+,;=:%]*)@/i', $authority, $match))
    {
        $authority = substr($authority, strlen($match[0]));
        $userinfo = $match[1];
    }
    else
    {
        $userinfo = null;
    }
    if(!preg_match('/^[!$%&\'()*+,\-.0-9:;=\[\]_a-z~]*$/i', $authority))
    {
        throw new URIException('Invalid authority part');
    }
    // Peel any port off end of authority
    if(preg_match('/(?<!:):([0-9]+)$/', $authority, $match))
    {
        $host = substr($authority, 0, -strlen($match[0]));
        // The port is a number...
        $port = (int)$match[1];
        if($port > 65535)
        {
            throw new URIException('Invalid port number');
        }
        // ...But is represented by a string of digits.
        $port = (string)$port;
    }
    else
    {
        $host = $authority;
        $port = null;
    }

    if($userinfo !== null && (self::bad_pct_encoded($userinfo) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $userinfo)))
    {
        throw new URIException('Invalid userinfo part');
    }
    if(!self::proper_host($host))
    {
        throw new URIException('Invalid host part');
    }

    return [$userinfo, $host, $port];
}

public static function parse(string $str)
{

    // Remove this test if and when IRIs are implemented.
    if(preg_match('/[^ -~]/', $str))
    {
        throw new URIException('URI supports ASCII only');
    }

    // Initial parsing regexp from RFC3986 Appendix B
    // Slightly modified to make scheme mandatory (we only deal with
    // absolute URIs, not URI-references)
    if(!preg_match('~^([^:/?#]+):(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?~', $str, $match))
    // parentheses---1--------1-2--3-------32-4------45--6-----65-7-8--87--
    {
        throw new URIException('Invalid URI. Preliminary regex failed.');
    }

    $scheme = $match[1];
    $authority = isset($match[2]) ? $match[3] : null;
    $path = $match[4];
    $query = isset($match[5]) ? $match[6] : null;
    $fragment = isset($match[7]) ? $match[8] : null;

    if(!preg_match('/^[a-z][a-z0-9\-+.]*$/i', $scheme))
    {
        throw new URIException('Invalid scheme component');
    }
    if($query !== null && (self::bad_pct_encoded($query) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $query)))
    {
        throw new URIException('Invalid query component');
    }
    if($fragment !== null && (self::bad_pct_encoded($fragment) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $fragment)))
    {
        throw new URIException('Invalid fragment component');
    }

    if($authority !== null && $authority !== '') // hmm...
    {
        list($userinfo, $host, $port) = self::parse_authority($authority);
    }
    else
    {
        $userinfo = $host = $port = null;
        if(!($path == '' || $path == '/' || !self::proper_path($path)))
        {
            throw new URIException('Invalid path component');
        }
    }

    return new self($scheme, $userinfo, $host, $port, $path, $query, $fragment);
}

private static function canonical_pct_encoding($str)
{
    return ($str === null) ? null : preg_replace_callback('/%[0-9a-f]{2}/i', function ($s)
    {
        return strtoupper($s[0]);
    }, $str);
}

protected function __construct($scheme, $userinfo, $host, $port, $path, $query, $fragment)
{
    // Scheme names are canonically lower case. (RFC3986 §3.1, para.2)
    $scheme = strtolower($scheme);
    // Host names are canonically lower case (§3.2.2) apart from %-encoding
    if($host !== null)
    {
        $host = strtolower($host);
    }
    // %-encoding canonically uses uppercase A-F (§2.1)
    $userinfo = self::canonical_pct_encoding($userinfo);
    $query = self::canonical_pct_encoding($query);
    $fragment = self::canonical_pct_encoding($fragment);
    $host = self::canonical_pct_encoding($host);
    $path = self::canonical_pct_encoding($path);

    $this->scheme = $scheme;
    $this->userinfo = $userinfo;
    $this->host = $host;
    $this->port = $port;
    $this->path = $path;
    $this->query = $query;
    $this->fragment = $fragment;
}

public function __toString()
{
    $uri = $this->scheme . ':';
    if($this->host !== null)
    {
        $uri .= '//';
        if($this->userinfo !== null)
        {
            $uri .= $this->userinfo . '@';
        }
        $uri .= $this->host;
        if($this->port !== null)
        {
            $uri .= ':' . $this->port;
        }
    }
    $uri .= $this->path;
    if($this->query !== null)
    {
        $uri .= '?' . $this->query;
    }
    if($this->fragment !== null)
    {
        $uri .= '#' . $this->fragment;
    }
    return $uri;
}
}


class HttpUri extends URI
{

// http-URI = "http:" "//" authority path-abempty [ "?" query ] [ "#"
// fragment ]
public static function parse(string $str): HttpUri
{
    $v = parent::parse($str);
    // We have to have an authority (rather, a host, because we've parsed
    // userinfo and port already).
    if($v->scheme !== 'http' || $v->scheme !== 'https')
    {
        throw new URIException('scheme part does not identify http(s) URI');
    }
    if($v->host === null)
    {
        throw new UriException('http(s) URIs require an authority part');
    }
}
}

[sneakyimp]

OK I've been running some tests and things are looking mostly good, but I see some behavior which seems noteworthy:
urls containing trailing spaces are passed as valid, e.g.:

$url = "http://example.com/lots of  space ";

Is this intentional? I see some discussion of whitespace in Appendix C of RFC 3986, but I kind of thought that spaces would become + or %20 in a proper URL. Seems really weird to be talking about urls broken by line wraps.
* urls with newline or tab chars have incorrect exception message:

URI supports ASCII only

Also, NO ONE has responded to my email the PHP internals list. If anyone is a member of this list, feel free to chime in and support me.

[Derokorian]

sneakyimp;11062035 wrote:

OK I've been running some tests and things are looking mostly good, but I see some behavior which seems noteworthy:
urls containing trailing spaces are passed as valid, e.g.:

$url = "http://example.com/lots of  space ";

Is this intentional? I see some discussion of whitespace in Appendix C of RFC 3986, but I kind of thought that spaces would become + or %20 in a proper URL. Seems really weird to be talking about urls broken by line wraps.
* urls with newline or tab chars have incorrect exception message:

URI supports ASCII only

Also, NO ONE has responded to my email the PHP internals list. If anyone is a member of this list, feel free to chime in and support me.

What's not valid about it? That it hasn't been encoded with + or %20 yet? I'm not sure what the problem is, every tool I know will automatically change that at time of request, including curl, wget, and browsers.

[sneakyimp]

Derokorian;11062039 wrote:

What's not valid about it? That it hasn't been encoded with + or %20 yet? I'm not sure what the problem is, every tool I know will automatically change that at time of request, including curl, wget, and browsers.

I guess I'm showing my age. Browsers used to barf on this kind of thing. I haven't tested curl with it.

[sneakyimp]

I spent some more time working with Weedpacket's code and I fixed a couple more things and added some comments and tweaks.

/**
 * A handful of quick classes to validate URLS according to RFC 3986
 * @see https://tools.ietf.org/html/rfc3986
 * written by Weedpacket to make up for broken filter_var functionality
 * @see http://board.phpbuilder.com/showthread.php?10395633-filter_var-results-a-bit-weird-for-FILTER_VALIDATE_URL
 */
namespace Weedpacket;

class URIException extends \DomainException
{
}

class URI
{

/**
 * When the static validate() function is called, this var will hold the last validation error (if any)
 * @var string
 */
private static $last_validate_error;

protected $scheme;

protected $userinfo;

protected $host;

protected $port;

protected $path;

protected $query;

protected $fragment;

/**
 * Validates the supplied string to determine if it's a valid uri or not
 * @param string $str
 * @return bool
 */
public static function validate(string $str): bool
{
	try
	{
		self::$last_validate_error = NULL;
		$parse = static::parse($str);
	}
	catch(URIException $e)
	{
		self::$last_validate_error = $e->getMessage();
		return false;
	}
	return true;
}


/**
 * Look for a % not followed by two hexdigits.
 * @param mixed $str
 * @return bool
 */
private static function bad_pct_encoded($str): bool
{
	return (bool)preg_match('/%(?![A-F0-9]{2})/i', $str);
}

private static function proper_path(string $path): bool
{
	if($path[0] == '/')
	{
		$path = substr($path, 1);
	}
	$parts = explode('/', $path);
	if($parts[0] == '')
	{
		return false;
	}
	foreach($parts as $part)
	{
		if(!self::bad_pct_encoded($part))
		{
			return false;
		}
	}
	return true;
}

private static function proper_host(string $host): bool
{
	// Regular name
	if(!preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $host) && !self::bad_pct_encoded($host))
	{
		return true;
	}
	// IPv4 literal
	if(preg_match('/^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$/', $host, $match))
	{
		return !($match[1] > 255 || $match[2] > 255 || $match[3] > 255 || $match[4] > 255);
	}
	// Other IP address literals
	if($host[0] != '[' || $host[-1] != ']')
	{
		return false;
	}
	$host = substr($host, 1, -1);
	// IPvFuture
	if($host[0] == 'v' || $host[0] == 'V')
	{
		return preg_match('/^v[0-9a-f]+[!$&\'()*+,\-.0-9:;=_a-z~]+$/i', $host);
	}
	// IPv6
	// Regex from http://home.deds.nl/~aeron/regex/
	return (bool)preg_match('/^(((?=.*(::))(?!.*\3.+\3))\3?|([\dA-F]{1,4}(\3|:\b|$)|\2))(?4){5}((?4){2}|(((2[0-4]|1\d|[1-9])?\d|25[0-5])\.?\b){4})$/i', $host);
}

private static function parse_authority(string $authority): array
{
	// Peel any userinfo off start of authority.
	if(preg_match('/^([a-z0-9\-._~!$&\'()*+,;=:%]*)@/i', $authority, $match))
	{
		$authority = substr($authority, strlen($match[0]));
		$userinfo = $match[1];
	}
	else
	{
		$userinfo = null;
	}
	if(!preg_match('/^[!$%&\'()*+,\-.0-9:;=\[\]_a-z~]*$/i', $authority))
	{
		throw new URIException('Invalid authority part');
	}
	// Peel any port off end of authority
	if(preg_match('/(?<!:):([0-9]+)$/', $authority, $match))
	{
		$host = substr($authority, 0, -strlen($match[0]));
		// The port is a number...
		$port = (int)$match[1];
		if($port > 65535)
		{
			throw new URIException('Invalid port number');
		}
		// ...But is represented by a string of digits.
		$port = (string)$port;
	}
	else
	{
		$host = $authority;
		$port = null;
	}

	if($userinfo !== null && (self::bad_pct_encoded($userinfo) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%]/i', $userinfo)))
	{
		throw new URIException('Invalid userinfo part');
	}
	if(!self::proper_host($host))
	{
		throw new URIException('Invalid host part');
	}

	return [$userinfo, $host, $port];
}

/**
 * Parses the supplied string $str and returns a structured object representing the parsed url
 * @param string $str
 * @throws URIException if the supplied string is not a valid URI
 * @return static
 */
public static function parse(string $str)
{

	// Remove this test if and when IRIs are implemented.
	if(preg_match('/[^ -~]/', $str))
	{
		throw new URIException('URI supports ASCII only');
	}

	// Initial parsing regexp from RFC3986 Appendix B
// https://tools.ietf.org/html/rfc3986
// here is original pattern, which apparently allows optional scheme
// NOTE: to use this, we would need to adjust indexes below :(
//$pattern = '~^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?~';
//			  12			3  4		  5	   6  7		8 9
	// Slightly modified to make scheme mandatory (we only deal with
	// absolute URIs, not URI-references)
$pattern = '~^([^:/?#]+):(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?~';
	if(!preg_match($pattern, $str, $match))
	// parentheses---1--------1-2--3-------32-4------45--6-----65-7-8--87--
	{
		throw new URIException('Invalid URI. Preliminary regex failed.');
	}

	$scheme = $match[1];
	$authority = isset($match[2]) ? $match[3] : null;
	$path = $match[4];
	$query = isset($match[5]) ? $match[6] : null;
	$fragment = isset($match[7]) ? $match[8] : null;

	if(!preg_match('/^[a-z][a-z0-9\-+.]*$/i', $scheme))
	{
		throw new URIException('Invalid scheme component');
	}
	if($query !== null && (self::bad_pct_encoded($query) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $query)))
	{
		throw new URIException('Invalid query component');
	}
	if($fragment !== null && (self::bad_pct_encoded($fragment) || preg_match('/[^a-z0-9\-._~!$&\'()*+,;=:@%\/?]/i', $fragment)))
	{
		throw new URIException('Invalid fragment component');
	}

	if($authority !== null && $authority !== '') // hmm...
	{
		list($userinfo, $host, $port) = self::parse_authority($authority);
	}
	else
	{
		$userinfo = $host = $port = null;
		if(!($path == '' || $path == '/' || !self::proper_path($path)))
		{
			throw new URIException('Invalid path component');
		}
	}

	return new static($scheme, $userinfo, $host, $port, $path, $query, $fragment);
}

private static function canonical_pct_encoding($str)
{
	return ($str === null) ? null : preg_replace_callback('/%[0-9a-f]{2}/i', function ($s)
	{
		return strtoupper($s[0]);
	}, $str);
}

/**
 * Called by the parse() function in this class and its extending classes, this constructs a structured URI object
 * @param unknown $scheme
 * @param unknown $userinfo
 * @param unknown $host
 * @param unknown $port
 * @param unknown $path
 * @param unknown $query
 * @param unknown $fragment
 */
protected function __construct($scheme, $userinfo, $host, $port, $path, $query, $fragment)
{
	// Scheme names are canonically lower case. (RFC3986 §3.1, para.2)
	$scheme = strtolower($scheme);
	// Host names are canonically lower case (§3.2.2) apart from %-encoding
	if($host !== null)
	{
		$host = strtolower($host);
	}
	// %-encoding canonically uses uppercase A-F (§2.1)
	$userinfo = self::canonical_pct_encoding($userinfo);
	$query = self::canonical_pct_encoding($query);
	$fragment = self::canonical_pct_encoding($fragment);
	$host = self::canonical_pct_encoding($host);
	$path = self::canonical_pct_encoding($path);

	$this->scheme = $scheme;
	$this->userinfo = $userinfo;
	$this->host = $host;
	$this->port = $port;
	$this->path = $path;
	$this->query = $query;
	$this->fragment = $fragment;
}

public function __toString()
{
	$uri = $this->scheme . ':';
	if($this->host !== null)
	{
		$uri .= '//';
		if($this->userinfo !== null)
		{
			$uri .= $this->userinfo . '@';
		}
		$uri .= $this->host;
		if($this->port !== null)
		{

			$uri .= ':' . $this->port;
		}
	}
	$uri .= $this->path;
	if($this->query !== null)
	{
		$uri .= '?' . $this->query;
	}
	if($this->fragment !== null)
	{
		$uri .= '#' . $this->fragment;
	}
	return $uri;
}

public function validate_error() {
	return self::$last_validate_error;
}
}

/**
 * Class to extend URI and require that the supplied string is an HTTP/HTTPS url
 *
 */
class HttpUri extends URI
{

// http-URI = "http:" "//" authority path-abempty [ "?" query ] [ "#"
// fragment ]
public static function parse(string $str): HttpUri
{
	$v = parent::parse($str);
	// We have to have an authority (rather, a host, because we've parsed
	// userinfo and port already).
	if($v->scheme !== 'http' && $v->scheme !== 'https')
	{
		var_dump($v);
		throw new URIException('scheme part does not identify http(s) URI');
	}
	if($v->host === null)
	{
		throw new UriException('http(s) URIs require an authority part');
	}

	return $v;
}
} // class HttpUri

[sneakyimp]

I ran a comparison test against filter_var for a small set of sample urls. My code takes care to add a scheme before checking these urls, and their results are nearly identical, but filter_var rejects urls with spaces in them. This script:

require_once "weedpacket.php";

$urls = array(
        "",
        "Buy It Now",
        "localhost/foo/bar",
        "blarg",
        "blarg/",
        "blarg/some/path/file.ext",
        "http://google.com",
        "http://google.com/",
        "http://google.com/some/path.ext",
        "http://google.com/some/path.ext?foo=bar",
        "example.com",
        "example.com/",
	"http://example.com/with/space ", // ends with a space
	"http://example.com/lots of  space ", // ends with a space
	"http://example.com/with/tab \t", // ends with a tab
	"http://example.com/with/newline\n", // ends with a newline
        "example.com/some/path/file.ext",
        "example.com/some/path/file.ext?foo=bar",
        "example.com:1234",
        "example.com:1234/",
        "example.com:1234/some/path/file.ext",
        "example.com:1234/some/path/file.ext?foo=bar",
	"http://www.foo.bar.example.com/",
        "//foobar.com",
        "//foobar.com/",
        "//foobar.com/path/file.txt",
        "//cdn.example.com/js_file.js",
	"http://www.example.com/v1/AUTH_svc.pdoc00001/PIX63/44749c00-9d96-48bb-8a35-68d418733114.JPG"

);

foreach($urls as $u) {

if (!$u) {
	// if the url is empty don't change it
} elseif (preg_match('#^//[^/]#', $u)) {
	// urls starting with exactly two slashes, prepend http:
	$u = "http:" . $u;
} elseif(!preg_match('#^http(s*)://#', $u)) {
	// urls that don't start with http:// or slashes
	$u = "http://" . $u;
}

$a = Weedpacket\HttpUri::validate($u);
$b = filter_var($u, FILTER_VALIDATE_URL);

if ($a != $b) {
	echo "DIFFERENT\n";
	var_dump($u);
	echo "a: $a\n";
	echo "b: $b\n\n";
}

}

The output:

DIFFERENT
string(30) "http://example.com/with/space "
a: 1
b: 

DIFFERENT
string(34) "http://example.com/lots of  space "
a: 1
b: 

Seems to me that the basic 3986 regex should end with an end-of-line signifier ($). As it stands, the input could end with literally anything and still get passed?

[sneakyimp]

Some may recall hackers specifying multiple urls in a link to trick users and/or open additional windows. Weedpacket's function appears to pass this:

$u ="http://www.example.com/ http://dangerous.hackersite.org";
$v = Weedpacket\HttpUri::validate($u);
var_dump($v); // returns TRUE;

where as filter_var flunks it.

[Weedpacket]

As well as most everything else (there's a lot of regex stuff going on and there's probably a lot of redundancy and the whole approach could be tossed) I am still hemming and hawing about encodings and whether to apply them at what point (see RFC3986 §2.3 para.2 and §2.4). You can see that validation and parsing are separated so that a user could extract useful parts of a given URI to allow application-specific checks. The user implementing such checks shouldn't have to fret about percent-encoding and whether "example.com" is equivalent to "exa%4dple.com" or not (or whether "foo+bar" is equivalent to "foo%20bar"), but "/foo/bar/" is a different beast from "/foo%2Fbar/" (although a subclass may parse a hierarchical path into something other than a raw string; maybe an array that in this case would include the element "foo/bar"). At the moment, what canonicalisation is done happens in the URI constructor since that's where all the pieces come together.

Error message

public static function parse(string $str): URI
{
	if(preg_match('/\p{Cc}/', $str))
	{
		throw new URIException('Unprintable control character found');
	}
	// Remove this test if and when IRIs are implemented.
	if(preg_match('/[^ -~]/', $str))
	{
		throw new URIException('URI supports ASCII only');
	}
...

Spaces do need encoding in URIs.

foreach($parts as $part)
{
	if(preg_match('/[^%0-9a-z:@!$&\'()*+,;=]/i', $part) || !self::bad_pct_encoded($part))
	{
		return false;
	}
}

IRIs make for a whole additional level of complexity. See [url]https://www.аррӏе.com/[/url] for details.

[Weedpacket]

The HttpUri class is one I dashed off then and there as a sketch of what a subclass might do - it wasn't in my codebase and it should relly have been in a comment. I don't think it's complete, and of course it uses the parent class wrong. It should be static, assign the return value of [noparse]parent::parse[/noparse] to a variable, check instance properties of that variable, and then return it.

The thing is, the interface is nowhere near complete: accessor mechanisms are needed for the URI object returned by parse() to have any use outside its subclasses, obviously, and there may be calls in future for mutator access as well (especially if there's going to be any support for URI-references and base resolution). Beyond the original "Is this URI syntactically valid? Yes/No" problem the question of what it should do and how it should be told to do it is open.