sjf on (How to Write a (Lisp) Interpreter (in Python))

Comment on (How to Write a (Lisp) Interpreter (in Python))parent

It's simple because it doesn't do much. A really tokeniser would have the ability to handle strings, escape characters and comments.

pufuwozu 15y

And that's why it was changed to this kind-of ugly function in lispy2:

    def tokenize(s):
        """Separate string s into tokens.  A token can be:
        a comment (which is ignored); a paren or ,@ or , or quote or quasiquote; a non-string atom;
        or a string consisting of quotes around (non-quotes or backslash plus anything)."""
        tokens = re.findall(r"""\s*(;.*|,@|[('`,)]|[^\s('");]+|"(?:[\\].|[^\\"])*")\s*""", s)
        return [t for t in tokens if not t.startswith(';')]

Comments