1. Sketch out what you want the first iteration of the language to look like. Write some example programs in your language.
2. Write a lexer: a program which turns a string of source code in your language into a list of values like "open parenthesis", "plus sign", "if keyword", "identifier". These values are called tokens. To test your lexer, output each token and see if your example programs tokenize properly.
3. Write a parser: a program which turns a list of tokens into a structured representation of your program's source code. For example, "if keyword" "identifier" "equals sign" "number" "then keyword" "print keyword" "identifier" might turn into an IfStatement with a predicate EqualsExpression (that itself has a left IdentifierExpression and a right LiteralNumberExpression) and a list of Statements for the code to run. You can write the parser yourself (look up recursive descent parsing) or use a parser generator tool to do it.
4. Write a code generator: a program which goes through your structured representation and outputs lower-level code (in this case, C) for each expression and statement.
I'm sure this is vastly oversimplified, but thanks for writing a quick summary of the steps. Do you have a good recommendation for books on compilers other than Dragon?
I don't know of any good books, sorry! There's plenty of helpful stuff on the web, though it can be hard to sift through. Personally I think just trying to write the code is the best way to learn.
Lisps and MLs can be really good parsing languages. I needed to parse some stuff and took the "easy" way of just using clojure's instaparse library and wrote a grammar which the library turned into a parsing function. After that you write your transform functions to manipulate the resulting parse tree. My target text wasn't a programming language but it would have been a nightmare to do with regexes.
I had never tried to write a parser before and it took about a day to go from no idea what I was doing to "I can pretty much see where to go next were I to want to write a toy language". Something you might want to take a look at is http://norvig.com/lispy.html
Comments
1. Sketch out what you want the first iteration of the language to look like. Write some example programs in your language.
2. Write a lexer: a program which turns a string of source code in your language into a list of values like "open parenthesis", "plus sign", "if keyword", "identifier". These values are called tokens. To test your lexer, output each token and see if your example programs tokenize properly.
3. Write a parser: a program which turns a list of tokens into a structured representation of your program's source code. For example, "if keyword" "identifier" "equals sign" "number" "then keyword" "print keyword" "identifier" might turn into an IfStatement with a predicate EqualsExpression (that itself has a left IdentifierExpression and a right LiteralNumberExpression) and a list of Statements for the code to run. You can write the parser yourself (look up recursive descent parsing) or use a parser generator tool to do it.
4. Write a code generator: a program which goes through your structured representation and outputs lower-level code (in this case, C) for each expression and statement.
I'm sure this is vastly oversimplified, but thanks for writing a quick summary of the steps. Do you have a good recommendation for books on compilers other than Dragon?
I don't know of any good books, sorry! There's plenty of helpful stuff on the web, though it can be hard to sift through. Personally I think just trying to write the code is the best way to learn.
Is there a typical preferred language to write lexers and parsers in? I imagine Python would be quite intuitive...
Lisps and MLs can be really good parsing languages. I needed to parse some stuff and took the "easy" way of just using clojure's instaparse library and wrote a grammar which the library turned into a parsing function. After that you write your transform functions to manipulate the resulting parse tree. My target text wasn't a programming language but it would have been a nightmare to do with regexes.
I had never tried to write a parser before and it took about a day to go from no idea what I was doing to "I can pretty much see where to go next were I to want to write a toy language". Something you might want to take a look at is http://norvig.com/lispy.html