Deadly Regular Expressions

What can we learn by using regular expression to do what it cannot do?



"You can't parse [X]HTML with regex."

The only explanation I'll give for the following code it to provide this link to my favorite Stack Overflow answer.

#!/usr/bin/env ruby -w

require "open-uri"

URL    = "" +
PARSER = %r{
    <!DOCTYPE\b (?<doctype> [^>]* ) >
    <!-- .* -->

    < \s* (?<tag_name> script ) \s* (?<attributes> [^>]* > )
      (?<script> .*? )
    < \s* / \s* script \s* >
    < \s* (?<tag_name> \w+ ) \s* (?<attributes> [^>]* / \s* > )
    < \s*
    (?<tag_name> link | meta | br | input | hr | img ) \b
    (?<attributes> [^>]* > )
    < \s* (?<tag_name> \w+ ) \s* (?<attributes> [^>]* > )
    < \s* / \s* (?<tag_name> \w+ ) \s* >

    (?<attribute_name> [-\w]+ )
    (?: \s* = \s* (?<attribute_value> "[^"]*" | '[^']*' | [^>\s]+ ) )? \s*
    (?= [^>]* > \z )  # attributes keep a trailing > to disambiguate from text

    (?! [^<]* /?\s*> \z )  # a guard to prevent this from parsing attributes


def parse(html)
  stack = [{attributes: [ ], contents: [ ], name: :root}]
  loop do
    html.sub!(PARSER, "") or break
    if $~[:doctype_declaration]
      add_to_tree(stack.last, "DOCTYPE", $~[:doctype].strip)
    elsif $~[:script_tag]
      add_to_stack(stack, $~[:tag_name], $~[:attributes], $~[:script])
    elsif $~[:self_closed_tag] || $~[:unclosed_tag] || $~[:open_tag]
      add_to_stack(stack, $~[:tag_name], $~[:attributes], "", $~[:open_tag])
    elsif $~[:close_tag]
    elsif $~[:text]
      stack.last[:contents] << $~[:text]

def add_to_tree(branch, name, value)
  if branch.include?(name)
    branch[name]  = [branch[name]] unless branch[name].is_a?(Array)
    branch[name] << value
    branch[name] = value

def add_to_stack(stack, tag_name, attributes_html, contents, open = false)
  tag = { attributes: parse_attributes(attributes_html),
          contents:   [contents].reject(&:empty?),
          name:       tag_name }
  add_to_tree(stack.last, tag_name, tag)
  stack.last[:contents] << tag
  stack                 << tag if open

def parse_attributes(attributes_html)
  attributes = { }
  loop do
    attributes_html.sub!(PARSER, "") or break
      ($~[:attribute_value] || $~[:attribute_name]).sub(/\A(["'])(.*)\1\z/, '\2')

def convert_to_bbcode(node)
  if node.is_a?(Hash)
    name = node[:name].sub(/\Astrike\z/, "s")
    "[#{name}]#{node[:contents].map { |c| send(__method__, c) }.join}[/#{name}]"

html = open(URL, &:read).strip
ast  = parse(html)
puts ast["html"]["body"]["div"]
  .find { |div| div[:attributes]["class"] == "container"      }["div"]
  .find { |div| div[:attributes]["id"]    == "content"        }["div"]["div"]
  .find { |div| div[:attributes]["id"]    == "mainbar"        }["div"]
  .find { |div| div[:attributes]["id"]    == "answers"        }["div"]
  .find { |div| div[:attributes]["id"]    == "answer-1732454" }["table"]["tr"]
  .find { |div| div[:attributes]["class"] == "answercell"     }["div"]["p"]
  .map(&method(:convert_to_bbcode))  # to reach a wider audience
$ ruby html_parser.rb
You can't parse [X]HTML with regex. Because HTML can't be parsed by regex.
Regex is not a tool that can be used to correctly parse HTML. As I have
answered in HTML-and-regex questions here so many times before, the use of
regex will not allow you to consume HTML. Regular expressions are a tool that
is insufficiently sophisticated to understand the constructs employed by HTML.
HTML is not a regular language and hence cannot be parsed by regular
expressions. Regex queries are not equipped to break down HTML into its
meaningful parts. so many times but it is not getting to me. Even enhanced
irregular regular expressions as used by Perl are not up to the task of parsing
HTML. You will never make me crack. HTML is a language of sufficient
complexity that it cannot be parsed by regular expressions. Even Jon Skeet
cannot parse HTML using regular expressions. Every time you attempt to
parse HTML with regular expressions, the unholy child weeps the blood of
virgins, and Russian hackers pwn your webapp. Parsing HTML with regex
summons tainted souls into the realm of the living. HTML and regex go
together like love, marriage, and ritual infanticide. The &lt;center> cannot hold
it is too late. The force of regex and HTML together in the same conceptual space
will destroy your mind like so much watery putty. If you parse HTML with regex
you are giving in to Them and their blasphemous ways which doom us all to
inhuman toil for the One whose Name cannot be expressed in the Basic
Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the
sentient whilst you observe, your psyche withering in the onslaught of horror.
Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow [i]it is too
late it is too late we cannot be saved[/i]the trangession of a chi͡ld ensures regex
will consume all living tissue (except for HTML which it cannot, as previously
prophesied) [i]dear lord help us how can anyone survive this scourge[/i]using
regex to parse HTML has doomed humanity to an eternity of dread torture and
security holes [i]using rege[/i]x as a tool to process HTML establishes a brea[i]ch
between this world[/i]and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities,
but [i]more corrupt) a mere glimp[/i]se of the world of reg​[b]ex parsers for HTML
will ins[/b]​tantly transport a p[i]rogrammer's consciousness i[/i]nto a w[i]orl[/i]d of
ceaseless screaming, he comes[s], the pestilent sl[/s]ithy regex-infection wil​[b]l
devour your HT[/b]​ML parser, application and existence for all time like Visual
Basic only worse [i]he comes he com[/i]es [i]do not fi[/i]​ght h[b]e com̡e̶s, ̕h̵i[/b]​s
un̨ho͞ly radiańcé de[i]stro҉ying all enli̍̈́̂̈́ghtenment, HTML tags [b]lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur
eye͢s̸ ̛l̕ik͏e liq[/b]​uid p[/i]ain, the song of re̸gular exp​re[s]ssion parsing [/s]will
exti[i]​nguish the voices of mor​[b]tal man from the sp[/b]​here I can see it can you
see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​[/i]he f[code]inal snuf[/code]fing o[i]f the lie​[b]s of Man ALL IS
LOŚ͖̩͇̗̪̏̈́T A[/b][/i][b]LL I​S L[/b]OST th[i]e pon̷y he come[/i]s he c̶̮om[s]es he
co[/s][b][s]me[/s]s t[i]he[/i]ich​[/b]or permeat[i]es al[/i]l MY FAC[i]E MY FACE ᵒh god
n[b]o NO NOO̼[/b][/i][b]O​O N[/b]Θ stop t[i]he an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨ[/i]e̠̅s[code]͎a̧͈͖r̽̾̈́͒͑e[/code]n[b]​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆
ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ T[/b]O͇̹̺ͅƝ̴ȳ̳ TH̘[b]Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝[/b]S̨̥̫͎̭ͯ̿̔̀ͅ
Comments (1)
  1. Aldric Giacomoni
    Aldric Giacomoni September 19th, 2014 Reply Link

    You've forgotten the last two lines of the answer:

    Have you tried using an XML parser instead?

    Also, I'm laughing. I can't stop. Thank you.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
Leave a Comment (using GitHub Flavored Markdown)

Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

Ajax loader