How do I extract all the external links of a web page and save them to a file?

How do I extract all the external links of a web page and save them to a file?

If you have any command line tools that would be great.

5 Answers

You will need 2 tools, lynx and awk, try this:

$ lynx -dump | awk '/http/{print $2}' > links.txt 

If you need numbering lines, use command nl, try this:

$ lynx -dump | awk '/http/{print $2}' | nl > links.txt 

Here's an improvement on lelton's answer: you don't need awk at all for lynx's got some useful options.

lynx -listonly -nonumbers -dump 

if you want numbers

lynx -listonly -dump 

As discussed in other answers, Lynx is a great option, but there are many others in nearly every programming language and environment.

Another choice is xmllint. Sample usage:

$ curl -sS "" \ | xmllint --html --xpath '//a[starts-with(@href, "http")]/@href' 2>/dev/null - \ | sed 's/^ href="\|"$//g' \ | tail -3 

Additionally, Perl offers HTML::Parser:

#!/usr/bin/perl use strict; use warnings; use HTML::Parser; use LWP::Simple; sub start { my $href = shift->{href}; print "$href\n" if $href && $href =~ /^https?:\/\//; } my $url = shift @ARGV or die "No argument URL provided"; my $parser = HTML::Parser->new(api_version => 3, start_h => [\&start, "attr"]); $parser->report_tags(["a"]); $parser->parse(get($url) or die "Failed to GET $url"); 

Sample usage (including writing to file per OP request; usage is the same for any script here with a shebang):

$ ./scrape_links > links.txt \ && cat links.txt | tail -3 

Ruby has the nokogiri gem:

#! /usr/bin/env ruby require 'nokogiri' require 'open-uri' doc = Nokogiri::HTML(URI.open(')) doc.xpath('//a[starts-with(@href, "http")]/@href').each do |link| puts link.content end 

NodeJS has cheerio:

const axios = require("axios"); const cheerio = require("cheerio"); (async () => { const $ = cheerio.load((await axios.get("")).data); $("a").each((i, e) => console.log($(e).attr("href"))); })(); 

Python's BeautifulSoup hasn't been shown yet in this thread:

import requests from bs4 import BeautifulSoup soup = BeautifulSoup(requests.get("").text, "lxml") for x in soup.find_all("a", href=True): if x["href"].startswith("http"): print(x["href"]) 

1. Use Beautiful Soup to retrieve the web pages in question.
2. Use awk to find all URLs that do not point to your domain

I would recommend Beautiful Soup over screen scraping techniques.

if command line is not a force you can use Copy All Links Firefox extension.