And that's why it was changed to this kind-of ugly function in lispy2:

    def tokenize(s):
        """Separate string s into tokens.  A token can be:
        a comment (which is ignored); a paren or ,@ or , or quote or quasiquote; a non-string atom;
        or a string consisting of quotes around (non-quotes or backslash plus anything)."""
        tokens = re.findall(r"""\s*(;.*|,@|[('`,)]|[^\s('");]+|"(?:[\\].|[^\\"])*")\s*""", s)
        return [t for t in tokens if not t.startswith(';')]