Sunlight

{
	//the language's keywords
	keywords: [],
	
	//strings, comments, etc.
	scopes: {},
	
	//i.e. +, &&, >>=, etc.
	operators: [],
	
	//should match a single character
	identFirstLetter: /regex/,
	
	//should match a single character
	identAfterFirstLetter: /regex/,
	
	//rules for coloring named idents, e.g. a class name
	namedIdentRules: {},
	
	//any custom tokens that don't fall under keywords or operators
	customTokens: {},
	
	//any custom parsing rules, e.g. the regex literal in JavaScript
	customParseRules: {},
	
	//custom analyzer for generating the HTML
	analyzer: myCustomAnalyzer,
	
	//whether keywords, tokens, etc. are case insensitive, default is false
	caseInsensitive: false,
	
	//regular expression that matches punctuation
	punctuation: /[^\w\s]/,
	
	//parse rule that parses a number
	numberParser: function(context) {},
	
	//regular expression defining what characters to not parse
	//this will short circuit the parsing process and no further parsing
	//will be done
	doNotParse: /\s/,
	
	//dictionary for storing arbitrary stateful objects during parsing/analysis
	contextItems: {}
};

Keywords should be an array of strings.

//JavaScript's keywords
[ 
	"break", "case", "catch", "continue", "default", "delete", "do", "else",	
	"finally", "for", "function", "if", "in", "instanceof", "new", "return",
	"switch", "this", "throw", "try", "typeof", "var", "void", "while", "with",
	"true", "false", "null"
];

Scopes are defined as a map from the scope name to the scope definition. The scope definition is an array of arrays. In each array, index 0 is the scope opener, index 1 is the scope closer index 2 is an array of escape sequences (optional) and index 3 is a boolean indicating whether the closer is zero width, meaning it should be included with the token’s value (optional, defaults to false).

{
	string: [
		[
			//opened by a double quote
			"\"", 
			//closed by a double quote
			"\"", 
			//double quotes and backslashes are escaped by a backslash
			["\\\"", "\\\\"]
		], [
			//opened by a single quote
			"'", 
			//closed by a single quote
			"'", 
			//single quotes and backslashes are escaped by a backslash
			["\\\'", "\\\\"]
		]
	],
	
	comment: [
		[
			//opened by two forward slashes
			"//",
			//closed by a line break
			"\n",
			//no escape sequences
			null,
			//don't include the line break in the value
			true
		]
	]
};

There is an alternate way of defining the scope closer which allows more flexibility. Instead of just a string, it can be defined as an object with properties regex, which is a regular expression matching the end of the scope, and length, which is the length of the closer.

//Java's annotation scope definition
{
	annotation: [ ["@", { length: 1, regex: /[\s\(]/ }, null, true] ]
}

Idents are variable names, function names, etc. They are defined by two regular expressions.

{
	//the first character of an ident must be a letter or an underscore
	identFirstLetter: /[A-Za-z_]/,
	
	//anything after that can be a letter, number or underscore
	identAfterFirstLetter: /\w/
};

Operators are defined as an array of strings. The only difference between an operator and a keyword (to sunlight) is that operators are an exact regular expression match and keywords end with a word boundary.

The order that operators are defined is important, because the match is greedy. That means + will match before += and stop looking, so += should come before + in the array.

//JavaScript's operators
[
	//arithmetic
	"++", "+=", "+",
	"--", "-=", "-",
		  "*=", "*",
		  "/=", "/",
		  "%=", "%",

	//boolean
	"&&", "||",

	//bitwise
	"|=",   "|",
	"&=",   "&",
	"^=",   "^",
	">>>=", ">>>", ">>=", ">>",
	"<<=", "<<",

	//inequality
	"<=", "<",
	">=", ">",
	"===", "==", "!==", "!=",

	//unary
	"!", "~",

	//other
	"?", ":", ".", "="
];

Some languages (e.g. CSS) are not case sensitive. You can indicate this by setting caseInsensitive: true.

Punctuation can be customized using a regular expression. By default, punctuation is considered anything that is not whitespace and not a letter, number or underscore.

If you want to treat punctuation as just text, you can use a regular expression that doesn’t match anything: /(?!x)x/.

By default, numbers are considered integers (e.g. 12), floats (e.g. 12.25), hex (e.g. 0x1A) and scientific notation (e.g. 1e3). The default number parsing algorithm is pretty loose, i.e. it will consider any number followed by a letter a number. You can customize this by injecting your own number parsing function, which is basically a custom parse rule.

This is the default number parse rule:

function(context) {
	var current = context.reader.current(), 
		number, 
		line = context.reader.getLine(), 
		column = context.reader.getColumn(),
		allowDecimal = true;

	if (!/\d/.test(current)) {
		//is it a decimal followed by a number?
		if (current !== "." || !/\d/.test(context.reader.peek())) {
			return null;
		}

		//decimal without leading zero
		number = current + context.reader.read();
		allowDecimal = false;
	} else {
		number = current;
		if (current === "0" && context.reader.peek() !== ".") {
			//hex or octal
			allowDecimal = false;
		}
	}

	//easy way out: read until it's not a number or letter
	//this will work for hex (0xef), octal (012), decimal and scientific notation (1e3)
	//anything else and you're on your own

	var peek;
	while ((peek = context.reader.peek()) !== context.reader.EOF) {
		if (!/[A-Za-z0-9]/.test(peek)) {
			if (peek === "." && allowDecimal && /\d$/.test(context.reader.peek(2))) {
				number += context.reader.read();
				allowDecimal = false;
				continue;
			}
			
			break;
		}

		number += context.reader.read();
	}

	return context.createToken("number", number, line, column);

}

{
	name: "tokenName",    //e.g. "keyword"
	value: "tokenValue",  //e.g. "if"
	language: "languageName", //e.g. "php"
	line: 1,
	column: 1
};

In addition to idents, sunlight has a built-in way of highlighting what it calls named idents. Basically these are class names in most languages, but they can be anything you want. For example, the XML language definition uses named idents to highlight attribute names.

There are four types of named ident rules. Three of them are for convenience. Those three are:

1. follows rules
2. precedes rules
3. between rules

follows and precedes rules are basically identical in that they both examine the tokens sequentially. The only difference is that follows rules go backward, and precedes rules go forward. Both are defined as an array of objects, where each object has keys token (a string) and values (an array of strings).

between rules are defined as an array of objects. Each object has keys opener and closer, both of which are objects with keys token and values. between rules match any ident that falls between the opener and closer.

{
	namedIdentRules: {
		follows: [
			//this will match "MyClass" in "new MyClass()"
			[{ token: "keyword", values: ["new"] }, sunlight.util.whitespace]
		],
		
		precedes: [
			//this will match "String" in "String[]"
			[
				sunlight.util.whitespace, 
				{ token: "punctuation", values: ["["] }, 
				sunlight.util.whitespace,
				{ token: "punctuation", values: ["]"] }
			]
		],
		
		between: [
			//this will match "Cloneable" and "Kissable" in 
			//"class MyClass implements Cloneable, Kissable {"
			{ 
				opener: { token: "keyword", values: ["implements"] }, 
				closer: { token: "punctuation", values: ["{"] }
			}
		]
	}
};

The last type of ident rule is a custom rule. Custom rules allow the most flexibility. They give you direct access to the tokens, and from there you can do whatever you want. Custom rules are defined as an array of functions. Each function should return true or false and is passed a single argument, the analyzer context:

{
	//array of token objects
	tokens: [],
	
	//the index of the token being analyzed
	index: 0,
	
	//the language definition
	language: {},
	
	//used internally for parsing nested languages
	//you probably shouldn't mess with this
	continuation: someFunction,
	
	//adds the specified DOM node to the collection
	addNode: function(node) {},
	
	//creates a text node from the given token (this handles encoding, tab width, etc.)
	createTextNode: function(token) {},
	
	//returns an array of DOM nodes
	getNodes: function() {},
	
	//clears the node array (not recommended unless you know what you're doing)
	resetNodes: function() {},
	
	//the current highlighter instance's options
	options: {},
	
	//dictionary for arbitrary item storage
	items: {}
};

Below is a custom rule used in the C♯ language definition to detect aliases:

function(context) {
	//previous non-ws token must be "using" and next non-ws token must be "="
	var prevToken = sunlight.util.getPreviousNonWsToken(context.tokens, context.index);
	if (prevToken.name !== "keyword" || prevToken.value !== "using") {
		return false;
	}

	var nextToken = sunlight.util.getNextNonWsToken(context.tokens, context.index);
	if (nextToken.name !== "operator" || nextToken.value !== "=") {
		return false;
	}

	return true;
}

And this is it how it works (MyClass is a named ident):

using System.Linq;
using MyClass = System.Collections.ICollection;

Check the C♯ language definition for more examples of custom named ident rules.

Custom tokens are basically keyword extensions. They behave in exactly the same way, but exist solely to provide a better breakdown between keyword-like tokens. For example, PHP has language constructs, functions and keywords. Language constructs (like echo, isset, etc.) are usually colored as keywords even though they often behave like functions.

Custom tokens are defined as a map from the name of the token to an object with keys values and boundary. boundary is a regex that Sunlight uses to determine when the token ends.

//PHP custom token excerpt
{
	customTokens: {
		languageConstruct: { 
			values: [
				"isset", "array", "unset", "list", "echo", "include_once", "include",
				"require_once", "require", "print", "empty", "return", "die", "eval",
				"exit"
			],
			boundary: "\\b"
		},
		
		constant: {
			values: [
				"__CLASS__", "__DIR__", "__FILE__", "__LINE__", "__FUNCTION__", 
				"__METHOD__", "__NAMESPACE__"
			],
			boundary: "\\b"
		},
		
		openTag: {
			values: ["<?php"],
			boundary: "\\s"
		}
	}
}

Custom parse rules can be used to perform your own language-specific parsing. Sunlight covers most of the basics by parsing keywords, operators, idents, etc. but occasionally you’ll need more control over the tokenizing process.

Custom parse rules are defined as an array of functions. Each function returns either a single token or an array of tokens, or null if the parse rule was not satisfied. The function is passed a single argument which is the parse context, defined as follows:

{
	//wrapper around the raw text
	reader: {
		//returns the next {count} characters without advancing the internal pointer
		peek: function(count) {},
		
		//returns the next {count} characters and advances the internal pointer
		read: function(count) {},
		
		//gets the current line number
		getLine: function() {},
		
		//gets the current column
		getColumn: function() {},
		
		//returns a boolean indicating whether the entire string has been read
		isEof: function() {},
		
		//returns a boolean indicating whether the internal pointer is at the end of a line
		isEol: function() {},
		
		//returns a boolean indicating whether the internal pointer is at the start of a line
		isSol: function() {},
		
		//returns a boolean indicating whether the internal pointer is at the start of a line
		//disregarding whitespace
		isSolWs: function() {},
		
		//the "end of string" constant, called EOF because EOS looks stupid
		EOF: undefined,
		
		//gets a string comprised of the current character to the end of the input string
		substring: function(),
		
		//gets a string comprised of the next character to the end of the input string
		peekSubstring: function(),
		
		//returns the character at the internal pointer's position
		current: function() {}
	},
	
	//the language definition
	language: language,
	
	//gets the token at the specified index, or undefined
	token: function(index) {},
	
	//gets all the tokens that have already been parsed
	getAllTokens: function() {},
	
	//gets the number of parsed tokens
	count: function() {},
	
	//the current Highlighter instance's options
	options: {},
	
	//contiguous unparsed characters (usually whitespace) are aggregated into
	//a single token, and are buffered here
	defaultData: {
		text: "",
		line: 1,
		column: 1
	},
	
	//creates and returns a token
	createToken: function(name, value, line, column) {},
	
	//user-defined dictionary for arbitrary storage
	items: {}
}

Below is an example of a custom parse rule from the PHP language definition. It handles the heredoc/nowdoc special string syntax. Another example is the regex literal handling in the JavaScript language definition. C♯ also contains several examples of how to handle contextual keywords (like get, set and value).

function(context) {
	if (context.reader.current() !== "<" || context.reader.peek(2) !== "<<") {
		return null;
	}
	
	var value = "<<<";
	var line = context.reader.getLine();
	var column = context.reader.getColumn();
	context.reader.read(2);
	
	var ident = "", isNowdoc = false;
	var peek = context.reader.peek();
	while (peek !== context.reader.EOF && peek !== "\n") {
		value += context.reader.read();
		
		if (peek !== "'") {
			//ignore NOWDOC apostophres
			ident += context.reader.current();
		} else {
			isNowdoc = true;
		}
		
		peek = context.reader.peek();
	}
	
	if (peek !== context.reader.EOF) {
		//read the newline
		value += context.reader.read();
		
		//read until "\n{ident};"
		while (context.reader.peek() !== context.reader.EOF) {
			if (context.reader.peek(ident.length + 2) === "\n" + ident + ";") {
				break;
			}
			
			value += context.reader.read();
		}
		
		if (context.reader.peek() !== context.reader.EOF) {
			value += context.reader.read(ident.length + 1); //don't read the semicolon
		}
	}
	
	return context.createToken(isNowdoc ? "nowdoc" : "heredoc", value, line, column);
}

After tokenizing, sunlight analyzes the tokens by iterating over each one and calling the appropriate function on an analyzer. That loop looks like this (simplified):

for (var i = 0, func; i < tokens.length; i++) {
	context.index = i;
	func = "handle_" + context.tokens[i].tokenName;
	
	analyzer[func] ? analyzer[func](context) : analyzer.handleToken(context);
}

The default behavior is to create a DOM node that looks like this: <span class="sunlight-{tokenName}">{tokenValue}</span>. If you don’t like that, you can extend the default analyzer. The PHP analyzer does this so that it can transform function names into hyperlinks:

var addFunctionLink = function(context) {
	var word = context.tokens[context.index].value;
	var suffix = context.tokens[context.index].name;
	var link = document.createElement("a");
	link.className = "sunlight-" + suffix;
	link.setAttribute("href", "http://php.net/" + word);
	link.appendChild(context.createTextNode(word));
	context.addNode(link);
}

var phpAnalyzer = sunlight.createAnalyzer();
phpAnalyzer.handle_languageConstruct = addFunctionLink;
phpAnalyzer.handle_function = addFunctionLink;

//then inject it into the language definition
var langauageDefinition = {
	analyzer: phpAnalyzer
};