Debunking Stroustrup's debunking of the myth “C++ is for large, complicated, programs only”

Question

Stroustrup has recently posted a series of posts debunking popular myths about C++. The fifth myth is: “C++ is for large, complicated, programs only”. To debunk it, he wrote a simple C++ program downloading a web page and extracting links from it. Here it is:

#include <string>
#include <set>
#include <iostream>
#include <sstream>
#include <regex>
#include <boost/asio.hpp>

using namespace std;

set<string> get_strings(istream& is, regex pat)
{
    set<string> res;
    smatch m;
    for (string s; getline(is, s);)  // read a line
        if (regex_search(s, m, pat))
            res.insert(m[0]);              // save match in set
    return res;
}

void connect_to_file(iostream& s, const string& server, const string& file)
// open a connection to server and open an attach file to s
// skip headers
{
    if (!s)
        throw runtime_error{ "can't connect\n" };

    // Request to read the file from the server:
    s << "GET " << "http://" + server + "/" + file << " HTTP/1.0\r\n";
    s << "Host: " << server << "\r\n";
    s << "Accept: */*\r\n";
    s << "Connection: close\r\n\r\n";

    // Check that the response is OK:
    string http_version;
    unsigned int status_code;
    s >> http_version >> status_code;

    string status_message;
    getline(s, status_message);
    if (!s || http_version.substr(0, 5) != "HTTP/")
        throw runtime_error{ "Invalid response\n" };

    if (status_code != 200)
        throw runtime_error{ "Response returned with status code" };

    // Discard the response headers, which are terminated by a blank line:
    string header;
    while (getline(s, header) && header != "\r")
        ;
}

int main()
{
    try {
        string server = "www.stroustrup.com";
        boost::asio::ip::tcp::iostream s{ server, "http" };  // make a connection
        connect_to_file(s, server, "C++.html");    // check and open file

        regex pat{ R"((http://)?www([./#\+-]\w*)+)" }; // URL
        for (auto x : get_strings(s, pat))    // look for URLs
            cout << x << '\n';
    }
    catch (std::exception& e) {
        std::cout << "Exception: " << e.what() << "\n";
        return 1;
    }
}

Let's show Stroustrup what small and readable program actually is.

1. Download http://www.stroustrup.com/C++.html

2. List all links:

   http://www-h.eng.cam.ac.uk/help/tpl/languages/C++.html
   http://www.accu.org
   http://www.artima.co/cppsource
   http://www.boost.org
   ...
   

You can use any language, but no third-party libraries are allowed.

Winner

C++ answer won by votes, but it relies on a semi-third-party library (which is disallowed by rules), and, along with another close competitor Bash, relies on a hacked together HTTP client (it won't work with HTTPS, gzip, redirects etc.). So Wolfram is a clear winner. Another solution which comes close in terms of size and readability is PowerShell (with improvement from comments), but it hasn't received much attention. Mainstream languages (Python, C#) came pretty close too.

Answer 1

Wolfram

This feels like complete cheating

Import["http://www.stroustrup.com/C++.html", "Hyperlinks"]

So just add some honest parsing on top

Cases[
 Import["http://www.stroustrup.com/C++.html", "XMLObject"],
 XMLElement["a", {___, "href" -> link_, ___}, ___] :> 
  link /; StringMatchQ[link, RegularExpression["((http://)?www([./#\\+-]\\w*)+)"]]
, Infinity]

Answer 2

C++

#include <boost/asio.hpp>
#include <regex>
#include <iostream>
int main() {
    std::string server = "www.stroustrup.com";
    std::string request = "GET http://" + server + "/C++.html HTTP/1.0\r\nHost: " + server + "\r\n\r\n";
    boost::asio::ip::tcp::iostream s{server, "http"};
    s << request;
    std::regex pat{R"((http://)?www([./#\+-]\w*)+)"};
    std::smatch m;
    for (std::string l; getline(s, l);)
        if (std::regex_search(l, m, pat))
            std::cout << m[0] << "\n";
}

The main shortcoming is the awkward nature of boost::asio, I'm sure it can be even shorter with a better library.

Answer 3

Pure Bash on Linux/OS X (no external utilities)

HTTP client software is notoriously bloated. We don't want those kinds of dependencies. Instead we can push the appropriate headers down a TCP stream and read the result. No need to call archaic utilities like grep or sed to parse the result.

domain="www.stroustrup.com"
path="C++.html"
exec 3<> /dev/tcp/$domain/80
printf "GET /$path HTTP/1.1\r\nhost: %s\r\nConnection: close\r\n\r\n" "$domain" >&3
while read -u3; do
    if [[ "$REPLY" =~ http://[^\"]* ]]; then
        printf '%s\n' "$BASH_REMATCH"
    fi
done

Meh - I suppose it could be more readable...

Answer 4

Python 2

import urllib2 as u, re
s = "http://www.stroustrup.com/C++.html"
w = u.urlopen(s)
h = w.read()
l = re.findall('"((http)s?://.*?)"', h)
print l

Lame, but works

Answer 5

C#

using System;
using System.Net;
using System.Text.RegularExpressions;

class Program {
    static void Main() {
        string html = new WebClient().DownloadString("http://www.stroustrup.com/C++.html");
        foreach (Match match in Regex.Matches(html, @"https?://[^""]+"))
            Console.WriteLine(match);
    }
}

Answer 6

"No third-party" is a fallacy

I think the "no third-party" assumption is a fallacy. And is a specific fallacy that afflicts C++ developers, since it's so hard to make reusable code in C++. When you are developing anything at all, even if it's a small script, you will always make use of whatever pieces of reusable code are available to you.

The thing is, in languages like Perl, Python, Ruby (to name a few), reusing someone else's code is not only easy, but it is how most people actually write code most of the time.

C++, with its nearly impossible-to-maintain-compatible-ABI-requirements makes that a much tougher job, you end up with a project like Boost, which is a monstrous repository of code and very little composability outside it.

A CPAN example

Just for the fun of it, here goes a CPAN-based example, with proper parsing of the html, instead of trying to use regex to parse html

#!/usr/bin/perl
use HTML::LinkExtor;
sub callback {
   my ($tag, %links) = @_;
   print map { "$_\n" } values %links
}
$p = HTML::LinkExtor->new(\&callback, "http://www.stroustrup.com/C++.html");

Answer 7

UNIX shell

lynx -dump http://www.stroustrup.com/C++.html | grep -o '\w*://.*'

Also finds an ftp:// link :)

Another way, without relying on :// syntax:

lynx -dump -listonly http://www.stroustrup.com/C++.html | sed -n 's/^[ 0-9.]\+//p'

Answer 8

CSS 3

* {
  margin: 0;
  padding: 0;
}
*:not(a) {
  font: 0/0 monospace;
  color: transparent;
  background: transparent !important;
}
a {
  content: "";
}
a[href*="://"]::after {
  content: attr(href);
  float: left;
  clear: left;
  display: block;
  font: 12px monospace;
  color: black;
}

This code can be used as a user style to display only absolute links on a page in an unformatted list. It may not work correctly if your browser enforces minimum font size.

It works correctly with http://www.stroustrup.com/C++.html (note !important on background). In order to work on other pages with more styles, it must be extended (reset more properties, mark properties as important etc.).

Alternative version which includes relative links except intrapage links starting with hashes (it relies on a hard-coded absolute link, unfortunately):

* {
  margin: 0;
  padding: 0;
}
*:not(a) {
  font: 0/0 monospace;
  color: transparent;
  background: transparent !important;
  float: none !important;
  width: auto !important;
  border: none !important;
}
a {
  content: "";
}
a::after {
  display: none;
}
a:not([href^="#"])::after {
  content: attr(href);
  float: left;
  clear: left;
  display: block;
  font: 12px monospace;
  color: black;
}
a:not([href*="://"])::after {
  content: "http://www.stroustrup.com/" attr(href);
}

Answer 9

Clojure

(->> (slurp "http://www.stroustrup.com")
     (re-seq #"(?:http://)?www(?:[./#\+-]\w*)+"))

Answer 10

Emacs Lisp

(with-current-buffer (url-retrieve-synchronously "http://www.stroustrup.com/C++.html")
  (while (re-search-forward "https?://[^\\\"]*")
    (print (match-string 0))))

Answer 11

Scala

"""\"(https?://.*?)\"""".r.findAllIn(scala.io.Source.fromURL("http://www.stroustrup.com/C++.html").mkString).foreach(println)

Answer 12

PHP 5

<?php
preg_match_all('/"(https?:\/\/.*?)"/',file_get_contents('http://www.stroustrup.com/C++.html'),$m);
print_r($m[1]);

Answer 13

PowerShell

Text search for all fully-qualified URLs (including JavaScript, CSS, etc.):

[string[]][regex]::Matches((iwr "http://www.stroustrup.com/C++.html"), '\w+://[^"]+')

Or to get links in anchor tags only (includes relative URLs):

(iwr "http://www.stroustrup.com/C++.html").Links | %{ $_.href }

Shorter versions from comments:

(iwr "http://www.stroustrup.com/C++.html").Links.href

(iwr "http://www.stroustrup.com/C++.html").Links.href-match":"

Answer 14

Node.js

var http = require('http');

http.get('http://www.stroustrup.com/C++.html', function (res) {
    var data = '';
    res.on('data', function (d) {
        data += d;
    }).on('end', function () {
        console.log(data.match(/"https?:\/\/.*?"/g));
    }).setEncoding('utf8');
});

Answer 15

D

import std.net.curl, std.stdio;
import std.algorithm, std.regex;

void main() {
foreach(_;byLine("http://www.stroustrup.com/C++.html")
    .map!((a)=>a.matchAll(regex(`<a.*?href="(.*)"`)))
    .filter!("a")){ writeln(_.front[1]); }
}

Answer 16

Haskell

Some troubles with "\w" in Text.Regex.Posix

import Network.HTTP
import Text.Regex.Posix
pattern = "((http://)?www([./#\\+-][a-zA-Z]*)+)"
site = "http://www.stroustrup.com/C++.html"

main = do
    file <- getResponseBody =<< simpleHTTP (getRequest site)
    let result = getAllTextMatches $ file =~ pattern
    putStr $ unlines result -- looks nicer

Answer 17

Ruby

require 'net/http'
result = Net::HTTP.get(URI.parse('http://www.stroustrup.com/C++.html'))
result.scan(/"((http)s?://.*?)"/)

Answer 18

PHP

As far as I can tell, most modern PHP installations come with DOM processing, so here's one that actually traverses the anchors inside the HTML:

foreach (@DOMDocument::loadHTMLFile('http://stroustrup.com/C++.html')->getElementsByTagName('a') as $a) {
    if (in_array(parse_url($url = $a->getAttribute('href'), PHP_URL_SCHEME), ['http', 'https'], true)) {
        echo $url, PHP_EOL;
    }
}

The inner loop could be shortened to:

preg_match('~^https?://~', $url = $a->getAttribute('href')) && printf("%s\n", $url);

Answer 19

Unix Shell

wget -q -O - http://www.stroustrup.com/C++.html | sed -n '/http:/s/.*href="\([^"]*\)".*/\1/p' | sort

Though i have to admit this doesn't work if there's more than one link on a line.

Answer 20

Java

import java.util.regex.*;
class M{
    public static void main(String[]v)throws Throwable{
        Matcher m = Pattern.compile( "\"((http)s?://.*?)\"" )
            .matcher(
                 new Scanner(
                         new URL( "http://www.stroustrup.com/C++.html" )
                             .openStream(),
                         "UTF-8")
                     .useDelimiter("\\A")
                     .next());
        while(m.find())
            System.out.println(m.group());
    }
}

Answer 21

Groovy

"http://www.stroustrup.com/C++.html".toURL().text.findAll(/https?:\/\/[^"]+/).each{println it}

Answer 22

SQL (SQL Anywhere 16)

Define a stored procedure to fetch the web page

CREATE OR REPLACE PROCEDURE CPPWebPage()
URL 'http://www.stroustrup.com/C++.html'
TYPE 'HTTP';

Produce the result set using a single query

SELECT REGEXP_SUBSTR(Value,'"https?://[^""]+"',1,row_num) AS Link  
FROM (SELECT Value FROM CPPWebPage() WITH (Attribute LONG VARCHAR, Value LONG VARCHAR) 
      WHERE Attribute = 'Body') WebPage, 
      sa_rowgenerator( 1, 256 ) 
WHERE Link IS NOT NULL;

Limitations: This produces up to 256 links. If more links exist, then bump up the 256 to an appropriate value.

Answer 23

CoffeeScript / NodeJS

require('http').get 'http://www.stroustrup.com/C++.html', (r) ->
    dt = '';
    r.on 'data', (d) -> dt += d
    r.on 'end' , (d) -> console.log dt.match /"((http)s?:\/\/.*?)"/g

Answer 24

Perl

use LWP;
use feature 'say';

my $agent = new LWP::UserAgent();
my $response = $agent->get('http://www.stroustrup.com/C++.html');

say for $response->content =~ m<"(https?://.+?)">g;

Answer 25

R

html<-paste(readLines("http://www.stroustrup.com/C++.html"),collapse="\n")
regmatches(html,gregexpr("http[^([:blank:]|\\\"|<|&|#\n\r)]+",html))

...although R is written mainly in C... so probably a few lines of C code behind those 2 lines of R code.

Answer 26

Objective-C

NSString *s;
for (id m in [[NSRegularExpression regularExpressionWithPattern:@"\"((http)s?://.*?)\"" options:0 error:nil] matchesInString:(s=[NSString stringWithContentsOfURL:[NSURL URLWithString:@"http://www.stroustrup.com/C++.html"]])]){
    NSLog(@"%@",[s substringWithRange:[m range]]);
}

Answer 27

Go

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "regexp"
)

func main() {
    resp, err := http.Get("http://www.stroustrup.com/C++.html")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    defer resp.Body.Close()
    data, _ := ioutil.ReadAll(resp.Body)
    results := regexp.MustCompile(`https?://[^""]+`).FindAll(data, -1)
    for _, row := range results {
        fmt.Println(string(row))
    }
}

P.S. this code reads entire source into memory, so consider using regexp.FindReaderIndex to search in stream, that'll make the app bulletproof.

Answer 28

Tcl

package require http
set html [http::data [http::geturl http://www.stroustrup.com/C++.html]]
puts [join [regexp -inline -all {(?:http://)?www(?:[./#\+-]\w*)+} $html] \n]

Answer 29

CJam

CJam does not have regex so I had to use a different approach in this one:

"http://www.stroustrup.com/C++.html"g''/'"*'"/(;2%{_"http://"#!\"https://"#!e|},N*

I first convert all ' to ", then I split on all ", take every alternative string and then finally filter that list for strings starting with http:// or https://. After that, simply print each filtered string on a new line.

Try it using the Java interpreter like

java -jar cjam-0.6.2.jar file.cjam

where file.cjam has the contents of the code above.

Answer 30

F#

This code could be far shorter but I would write something like this if I ever expected to have to read or use this code again so it has many unnecessary type annotations. It demonstrates the use of an active pattern MatchValue to enable pattern-matching against the standard CLR type Match

open System.Net

let (|MatchValue|) (reMatch: Match) : string = reMatch.Value

let getHtml (uri : string) : string = 
    use webClient = WebClient() in
        let html : string = webClient.DownloadString(uri)
        html

let getLinks (uri : string) : string list =
    let html : string = getHtml uri
    let matches : MatchCollection = Regex.Matches(html, @"https?://[^""]+") 
    let links = [ for MatchValue reMatch in matches do yield reMatch ]
    links

let links = getLinks "http://www.stroustrup.com/C++.html" 
for link in links do
    Console.WriteLine(link)

Edit I made getLinks its own function

Answer 31

Rebol

parse read http://www.stroustrup.com/C++.html [
    any [
        thru {<a href="} copy link to {"} (print to-string link)
    ]
]

Answer 32

Delphi

program Links;
{$APPTYPE CONSOLE}
{$R *.res}
uses
  System.SysUtils, idHTTP, RegularExpressions;
var
  client: TidHTTP;
  match : TMatch;
begin
  client := TidHTTP.Create;
  try
    match := TRegEx.Create('<a(.*)href="(.*)">(.*)<\/a>', [roIgnoreCase, roMultiline])
      .match(client.Get('http://www.stroustrup.com/C++.html'));
    with match do
      while Success do
      begin
        if Groups.Count >= 3 then
          if copy(lowercase(Groups[2].Value), 1, 4) = 'http' then
              writeln(Groups[2].Value);
        match := NextMatch;
      end;
  finally
    client.Free;
  end;
end.

This would work in (I suppose) Delphi XE and later. It requires no component other than those that are already installed in a default setup (namedly indy and regular expressions). Even tho delphi is quite close to C++ in general structure, I guess this task turned out to be a bit shorter.

Answer 33

JS/jQuery

$.get('http://www.stroustrup.com/C++.html',{},function(s){
    $(s).find('a').each(function() {
        console.log($(this).attr('href'))
    })
})

Answer 34

Ruby

Another Ruby solution:

require 'open-uri'
open('http://www.stroustrup.com/C++.html', 'r:iso-8859-1:utf-8') do |f|
  puts f.read.scan(%r{"(https?://www[^"]*)"}).sort
end

Answer 35

Scala

object Downloader extends App {
    val s = io.Source.fromURL("http://www.stroustrup.com/C++.html", "iso-8859-1").mkString // load URL to String
    val regex = """((http://)?www([./#\+-]\w*)+)""".r                                      // create and compile regexp
    println(regex.findAllIn(s).mkString("\n"))                                             // print matches
}

Answer 36

F#

open System.Net
open System.Text.RegularExpressions

let html = (new WebClient()).DownloadString("http://www.stroustrup.com/C++.html") in
    Regex.Matches(html, @"https?://[^""]+") |> Seq.cast<Match> |> Seq.iter (printfn "%A")

Taken from the C# version.

Answer 37

SmallTalk (Pharo 3)

Hurray for []! Also, we should totally add a Smalltalk version ;)

@Bersaelor at Objective-C answer.

I know basics of Smalltalk - syntax of language and some tutorials. I decide that it's good place for practice. I have already installed Pharo 3.0.

but no third-party libraries are allowed

Okay. On downloaded image I found Zinc-HTTP and regex packages. Probably I should read about "third-party" meaning.

So, code:

(ZnClient new get: 'http://www.stroustrup.com/C++.html')
   regex: '((http\://)?www([./#+-]\w*)+)' matchesDo: [ :x | Transcript show: String cr, x ].

Code with exception handlings:

|response|
[  
response := (ZnClient new url: 'http://www.stroustrup.com/C++.html'; get; response).
response isSuccess 
  ifFalse: [ Transcript show: 'Bad status : ', (response status asString) ]
  ifTrue:  [ response contents regex: '((http\://)?www([./#+-]\w*)+)' matchesDo: 
           [ :x | Transcript show: String cr, x ]]
]
    on: NameLookupFailure do: [ Transcript show: 'Connection problem...' ].

If you try to check me out you will get an exception. Something like UTF8EncoderException: errorIllegalLeadingByte. The first thing I thought that the package pretty outdated. But then realized that it downloads other sites well. Then I thought it is not always well copes with utf8. In debug received byte = 150 (1001 0110) - it's bad if it is the first byte in sequence. I spent a some time on the localization error (moved up in call stack and found parsed line). So:

Lang.Next'14 Keynote: What � if anything � have we learned from C++? A 68 minute talk incl. Q&A.

You should see here squares or diamonds with question marks, depending on your browser. Trap from Stroustrup. At best, I had to write an exception handler in the package at the point where the line is created. But i just comment exception call and return ? character.

Also in code which calc the sequence length I return 1 in place of exception call (seems browsers does same).

P.S. some notes about Pharo (if anyone interested):

• Too much GUI
• Frendly for beginners.
• Nice package searching
• Nice highlighting and code autocompletion (not nice enough for serios IDE like IDEA but better than what I saw a few years ago in the same Pharo). Autocompletion sometimes trying deduce type - very thankless job.
• Very chaotic GUI. At IDE we used to see tiled windows system, but there is only floating litle windows.
• Where is imports, namespaces? There are many entities already. Is never a collision occurs?

Answer 38

Vimscript

function! Cpp()
    " grab the page in a new buffer in a new window in a new tab page
    tabedit http://www.stroustrup.com/C++.html

    " delete all lines that don't contain at least one 'http' hyperlink
    v/"http/d

    " only keep the hyperlink on every line
    %s/^.\+="\([^"]\+\)".\+$/\1
endfunction

Answer 39

Rust

Here is Rust solution:

extern crate reqwest;
extern crate select;
extern crate regex;

use select::document::Document;
use select::predicate::Name;
use regex::Regex;

fn main() {
    scrape_links("http://www.stroustrup.com/C++.html");
}

fn scrape_links(url: &str) {
    let resp = reqwest::get(url).unwrap();
    assert!(resp.status().is_success());
    let re = Regex::new(r"((http://)?www([./#\+-]\w*)+)").unwrap();
    Document::from_read(resp)
        .unwrap()
        .find(Name("a"))
        .filter_map(|n| n.attr("href"))
        .filter(|text| re.is_match(text))
        .for_each(|x| println!("{}", x));
}

Answer 40

MATLAB

Is quite straightforward with urlread and regexp:

url = 'http://www.stroustrup.com/C++.html';
links = regexp(urlread(url), '<a href="http://([^"]*\.*)">', 'tokens');

Answer 41

Python 2

I don't like using regex on HTML for established reasons, so here's an ungolfed HTMLParser approach:

from HTMLParser import HTMLParser
import urllib2 as u

class LinkFinder(HTMLParser):
    def handle_starttag(self, tag, attrs):
        if tag=='a':
            attrdct = dict(attrs)  # attrs is a list of ('key', 'value') tuples
            if 'href' in attrdct:
                href = attrdct['href']
                print href

url = "http://www.stroustrup.com/C++.html"
contents = u.urlopen(url).read()
LinkFinder().feed(contents)

Note this also gives local links such as index.html and anchors à la #learning. If you only want absolute links, prepend print href by

                if ':/' in href:  # also handles ftp, https etc.
                    print href

while for only omitting the anchors, use

                if !href.startswith('#'):
                    print href

Answer 42

Lua

Here's a lua solution complete with error checking and duplicate URL elimination like Stroustrups's C++ version.

Just made it in under 10 lines

local http, urlunique = require 'socket.http', {}

local body, resp, _, respmsg = http.request "http://www.stroustrup.com/C++.html"
assert(resp == 200, respmsg or resp)
for each in body:gmatch 'https?://[^%s<>"]+' do
  if not urlunique[each] then
    urlunique[each] = true
    print(each)
    end end

Here's another version using string.gsub fitting in just 6 lines!

local http, urlunique = require 'socket.http', {}

local body, resp, _, respmsg = http.request "http://www.stroustrup.com/C++.html"
assert(resp == 200, respmsg or resp)
body:gsub ('https?://[^%s<>"]+', function(r) urlunique[r] = true end)
for url in pairs(urlunique) do print(url) end

Answer 43

VBScript in Windows Script Host

That is, if this is stored in links.vbs file, run it via cscript /nologo links.vbs.

sub writeline( s ): WScript.StdOut.WriteLine s : end sub
function re( s ): set re = new RegExp: re.pattern = s: re.global = true: end function

set http = createobject( "Msxml2.XMLHTTP" )
http.open "GET", "http://www.stroustrup.com/C++.html", false: http.send
set links = re( "\w+://[^\""]+" ).execute( http.responseText )
for each link in links: writeline( link ): next

---

Addendum:

While the above lists all full links, which seems to be the goal, Stroustrup’s code additionally pares it down to unique links, and here’s a version that does that:

sub writeline( s ): WScript.StdOut.WriteLine s : end sub
function re( s ): set re = new RegExp: re.pattern = s: re.global = true: end function

set http = createobject( "Msxml2.XMLHTTP" )
http.open "GET", "http://www.stroustrup.com/C++.html", false: http.send
set links = re( "\w+://[^\""]+" ).execute( http.responseText )

set unique_links = createobject( "Scripting.Dictionary" )
on error resume next
for each link in links: unique_links.add ucase(link), link & "": next
for each link in unique_links.items(): writeline( link ): next

This reduces the number of output lines from 81 to 77.

Answer 44

R

I'm super new to regex so I gave this my best shot... any improvements appreciated!

grep("(http)s?://.*?", readLines("http://www.stroustrup.com/C++.html"), value = T)

readLines() just dumps the HTML source into a character vector. I then used grep() to find the URLs. The problem I ran into was that HTML element tags as well as corresponding link text were included in the output. substring() could be used to trim some of them I guess but it wouldn't work in all cases. If anyone knows a better way please let me know - especially if I could use a better regex.

Answer 45

Python 2.7

Some readable Python code.

import urllib2
import re
page = urllib2.urlopen("http://www.stroustrup.com/C++.html").read()
for link in re.findall('"(http[s]?://.*?)"', page):
    print link

Answer 46

(C++)--

aka C

I'm suprised no one has done C yet. The code is nice and clean.

linkfetch.c:

#include "inet_utils.h"
main(){
  char* SITE="www.stroustrup.com";
  char* PAGE="C++.html";
  char* REGX="((http://)?www([./#\\+-]\\w*)+)";
  int s = connect_to(SITE);
  FILE* f = fetch_page(s,SITE,PAGE);
  if (f) list_matches(f,REGX);
  else return printf("Can't fetch page %s/%s\n",SITE,PAGE);
  fclose(f); close(s);
}

That is, clean assuming you also write these simple utiliies:

inet_utils.h

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <netdb.h>
#include <regex.h>

#define SZ 1024 //good default buffer size
#define ERROR(s) (puts(s)&&0)
#define VP(e) ((void*)(long)(e))

inline int connect_to(char* w){
  //:returns socket connected to website w.
  struct sockaddr_in a;
  int s=socket(AF_INET, SOCK_STREAM, 0);                       // make socket
  struct hostent* h = gethostbyname(w);                        // lookup host
  if (!h) return ERROR("No Such Host");                        // check err
  a.sin_family=AF_INET;                                        // set ip address
  a.sin_port=htons(80);                                        // port 80
  memcpy(&a.sin_addr.s_addr, h->h_addr, h->h_length);          // of host
  if (connect(s,(struct sockaddr*)&a,sizeof(a))<0)             // connect
    return ERROR("can't connect");                             // handle error
  return s;                                                    // return socket
}

inline FILE* fetch_page(int s, char*w, char* p){
  //:returns open file handle for page p from site w connected to socket s

  FILE*f=fopen("/tmp/wcache","w+");                   // create cache file
  size_t n; char*b=malloc(SZ);                        // allocate temp buffer
  if (!s||!f||!b) return VP(ERROR("Resource Error")); // check for errors
  sprintf(b,                                          // compose request
          "GET / HTTP/1.0\r\nHost:%s\r\nAccept:*/*\r\nConnection:close\r\n\r\n",
          w);
  send(s,b,strlen(b),0);                         // send request
  while ((n=recv(s,b,SZ,0))>0)                   // receive response
    fwrite(b,1,n,f);                             // write it to cache file
  fseek(f,n=0,SEEK_SET);                         // read from beginnng
  fgets(b,SZ,f);                                 // look at first line
  if (!f||strncmp(strtok(b," "),"HTTP/",5))      // is it http?
    return VP(ERROR("Invalid Response"));        // error if not
  if (atoi(strtok(0," "))!=200)                  // check good status code
    return VP(ERROR("Bad Status Code"));         // error if not
  while (getline(&b,&n,f)>=0 && *b!='\r');       // skip headers upto blank line
  free(b);                                       // cleanup
  return f;                                      // return open handle
}

inline void list_matches(FILE* f, char* regx){
  //prints all strings from f which match regx
  regex_t r;
  size_t n = SZ; char*b=malloc(n);               // temp buffer
  if (regcomp(&r,regx,REG_NOSUB|REG_EXTENDED))   // compile regex
    puts("invalid regex");                       // handle error
  else while (getline(&b,&n,f)>0)                // fetch line
         if (!regexec(&r,b,0,0,0))               // check match
           puts(b);                              // show match
  regfree(&r); free(b);                          // cleanup
}

It compiles without warnings on out-of-the-box gcc (version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) )

Answer 47

Swift (2.2)

What? Please write the Swift version. That square bracket nonsense is hurting my eyes :)

Mister Smith on the Objective-C answer.

let x = try!NSString(contentsOfURL:NSURL(string:"http://www.stroustrup.com/C++.html")!,encoding:4)
for y in try!NSRegularExpression(pattern:"\"((http)s?://.*?)\"",options:[]).matchesInString(x as String,options:[],range:NSMakeRange(0,x.length)){print(x.substringWithRange(y.range))}

Ungolfed:

let url = NSURL(string: "http://www.stroustrup.com/C++.html")!
let html = try! NSString(contentsOfURL: url, encoding: NSUTF8StringEncoding)
let regex = try! NSRegularExpression(pattern: "\"((http)s?://.*?)\"", options: [])
let results = regex.matchesInString(html as String, options: [], range: NSMakeRange(0, html.length))
for result in results {
    print(html.substringWithRange(result.range))
}

Assumes Foundation has been implicitly imported.

I can't access the original page for some reason, so this was tested with the Google Cached version: http://webcache.googleusercontent.com/search?q=cache:USk4BseSofcJ:www.stroustrup.com/C%2B%2B.html+&cd=1&hl=en&ct=clnk&gl=us

282 bytes. Slightly shorter than the 292 byte Objective-C answer. I'm falling back onto Foundation APIs, so there may be room for improvement by using pure Swift types. The Cocoa APIs have also changed since the Objective-C answer was posted.

stringWithContentsOfURL: has been deprecated on NSString in favor of stringWithContentsOfURL:encoding:error:. We lose some bytes on the encoding parameter, but gain some back because the ErrorPointer is no longer passed in with Swift. The function now throws its NSError so instead, I'm using try! to force the execution of the NSString and NSRegularExpression initializers. I also save some bytes by passing the raw value 4 as the value of the encoding parameter instead of the constant NSUTF8StringEncoding. 19 bytes saved. But I lose some bytes by having to pass in an empty array ([]) instead of 0 to represent no options. 2 bytes lost there. I also lose two bytes for every variable declaration since Swift requires whitespace characters on either side of the = character.

I lose 10 bytes by having to cast the NSString as a Swift String when calling matchesInString. This is required because I'm using the NSString method contentsOfURL to get the web page HTML, but the NSRegularExpression method matchesInString takes in a Swift String as a parameter. The implicit conversion between NSString and String isn't available here, so I am forced to use as to explicitly convert the types.

Interestingly enough matchesInString has not been completely converted to use Swift types. It still requires its range parameter to be an NSRange struct instead of a Swift Range<String>. I have to fall back and use NSMakeRange to create the range of the string. I could save 4 bytes by using x.characters.indices of type Range<String.CharacterView.Index> instead, but Swift Range structs are not compatible with Foundation NSRange structs. Additionally, if x were a Swift String, I might be able to save a few bytes by replacing substringWithRange with a subscript on String. I haven't found a great way to do that yet, as creating two Index structs is currently longer than using substringWithRange.

Answer 48

ColdFusion

(using the same regex that Stroustrup uses)

<cfhttp url="http://www.stroustrup.com/C++.html" result="response" />
<cfif response.statusCode does not contain "200">
    <cfset writeOutput("Error getting the page: #response.statusCode#") />
<cfelse>
    <cftry>
        <cfset htmlLinks = REMatchNoCase("((http://)?www([./#\+-]w*)+)",response.fileContent) />
        <cfdump var="#htmlLinks#" />
    <cfcatch>
        <cfset writeOutput("There was a problem: #cfcatch.message# #cfcatch.detail#") />
    </cfcatch>
    </cftry>
</cfelse>

Answer 49

XQuery

HTML should be something similar to XML, so why not use langauges designed for this job?

If the page would've been "real" XHTML, we could run a query as beautiful as

doc("http://www.stroustrup.com/C++.html")//a/@href/data()

As this is crappy, broken HTML, let's use the BaseX-specific HTML parser (BaseX is an XQuery implementation):

html:parse(fetch:binary("http://www.stroustrup.com/C++.html"))//a/@href/data()

If limiting to URLs starting with http: is a must, lets do it:

html:parse(fetch:binary("http://www.stroustrup.com/C++.html"))//a/@href[starts-with(., 'http:')]/data()

Disclaimer: I am somewhat affiliated with the BaseX team as I wrote some code during my thesis. This would've been the tool of my choice for that kind of task, anyway. Other XQuery implementations provide similar HTML parsing capabilities, but I don't know their XQuery extensions by heart.

Answer 50

Bash + AWK

wget -q -O http://www.stroustrup.com/C++.html \
|awk '/((http:\/\/)?www([./#\+-]\w*)+)/ {print gensub(/.*((http:\/\/)?www([./#\+-]\w*)+).*/,"\\1","g")}'

I know it probably misses a few URLs, but I chosen to use the same regex than the original Stroustrup's code, so this should returns the same output than the original piece of code.

It may be possible to add some CR to make it more visible but I don't have a Linux available ATM for testing it works... (tested on Windows)

Here is a "clean" version of the AWK part:

/((http:\/\/)?www([./#\+-]\w*)+)/ {
    print gensub(/.*((http:\/\/)?www([./#\+-]\w*)+).*/,"\\1","g")
}

Answer 51

PHP 4.3+ / 5.0+

I know there are 2 different answers regarding PHP, but I'm going to show here a similar aproach, using nothing but standard functions.

For this, you will need to have the following on a file named php.ini ON THE SAME DIRECTORY:

allow_url_fopen= On
allow_url_include= On

THAT PART IS IMPORTANT!
In case you can't change (didn't worked with XAMPP), you have a default php.ini file on the PHP installation folder.
Changing the values will solve it.
Remember to restart apache after.

Since this isn't code-golf, I made my code somewhat readable.

Here it is:

ob_start(); //creates an output buffer

//now we 'include' the file, which will output the source code.
include 'http://www.stroustrup.com/C++.html';

$html = ob_get_clean(); //stores the output buffer and closes it

$offset = 0; //initial offset to search
$links = array(); //will contain all links

//while a link is found
while($pos = strpos($html, 'href="http', $offset))
{
    //look for the closing "
    $end = strpos($html, '"', $pos + 7);
    //take it from the string, store it into the array
    $links[] = substr($html, $pos + 6, ($end - $pos) - 6);
    //increase the offset, so it doesn't find the same link again
    $offset = $end + 1;
}

print_r($links); //spits it out, with the output buffer closed

I've added some comments to try to explain the code.

No regex used or DOM parsers: only pure hard-cold string manipulation.

For this to work in other pages, you must be sure that the values of the property href are between "", or it will fail.

Answer 52

F#

do
  use client = new System.Net.WebClient()
  let html = client.DownloadString "http://www.stroustrup.com/C++.html"
  System.Text.RegularExpressions.Regex.Matches(html, @"https?://[^""]+")
  |> Seq.cast<System.Text.RegularExpressions.Match>
  |> Seq.iter (printfn "%O")

Answer 53

CSS - idea

In Firefox, for example, for any page you are on, you can go to Tools | Web Developer | Style Editor, and use CSS to display anchors only:

* {display:none;}
a {display:block;}

However, the above will not work because display of parent elements overrides children.

Still working on a CSS solution, but suggestions welcome!

Answer 54

C++/U++

Here is the U++ (which is 'another' C++ library, slightly different approach from boost) version:

#include <Core/Core.h>
#include <plugin/pcre/Pcre.h>

using namespace Upp;

CONSOLE_APP_MAIN
{
    String s = ToCharset(CHARSET_UTF8, HttpRequest("http://www.stroustrup.com/C++.html").Execute(),
                         CHARSET_ISO8859_1);
    RegExp x("href *= *\"(.*?)\"");
    while(x.GlobalMatch(s))
        Cout() << x.GetStrings()[0] << "\n";
}

(BTW, funny how most versions here are IMO not implementing the original specification: extract all links. There are many links on that page that do not start with http...)

Answer 55

Lua

More concise, without checking for errors or unique URLs: In readable...

http = require 'socket.http'
page = http.request 'http://www.stroustrup.com/C++.html'
page:gsub('http://www[%w./#\\+-]+', print)

...and less-readable...

require'socket.http'.request'http://www.stroustrup.com/C++.html':gsub('http://www[%w./#\\+-]+',print)

Answer 56

Javascript / DOM

var r = new XMLHttpRequest();
r.open('GET', 'http://www.stroustrup.com/C++.html', false);
r.send(null);
var div = document.createElement('div');
div.innerHTML = r.response;
[].slice.call(div.getElementsByTagName('a')).forEach( function(a) {
  if (a.href && a.getAttribute('href').charAt(0) != '#') console.log(a.href);
} );

(Tested in the Google Chrome javascript console.)

Of course, CORS blocks this by default - but its easy to disable CORS in Google Chrome for development purposes.

Answer 57

Bash

 GET http://www.stroustrup.com/C++.html | grep -o "https\?://[^ \"]\+

needs libwww-perl and grep packages :)