Porting Razor to TypeScript

| Tags: razor, typescript, vscode, javascript, es6, html, jasmine, karma | Comments

Having not tackled any major TypeScript projects recently, the language itself has continued to be at the top of my 'to learn' list. I wanted to find a new project that I could use to learn TypeScript, and I decided on tackling a port of the Razor parsing framework to TypeScript.

I've enjoyed significant experience with the Razor parsing framework, initially through my open source project RazorEngine and most recently through my new project FuManchu, a C# implementation of HandlebarsJS using the Razor parsing framework and MVC's metadata framework.

Now, converting any library like this to JavaScript for the browser or NodeJS is quite an undertaking, but given how clean the codebase and any lack of significant dependencies, it's seems to me a perfect fit.

Tools

To tackle this project, I'm going out of my comfort zone - that means dropping Visual Studio in favour of a simpler text editor - and Visual Studio Code is again, a perfect choice. I still gives me intellisense, and provides built-in support for TypeScript. I also considered alternatives such as Sublime, Brackets or Atom, all using their respective OmniSharp plugins.

Razor itself comes with a wealth of unit tests already predefined, so that assists ensuring that my TypeScript port at least conforms to the same set of tests that the main project supports, but additionally we're not building a C# language parser, I'm attempting an ECMAScript 6 language parser, so although there are a great many similarities, there are also differences. My unit test framework of choice is Jasmine, combined with Karma. I hope to implement the majority of the unit tests as proof of the implementation.

Where to start

There is a lot going on in Razor, from the tokenizers, to language parsers, chunk generators and tag helpers, it's important to break down the project into smaller blocks which can be ported and tested. So let's start at the beginning, and tackle some simple concepts - reading text. Razor provides a number of text reader implementations that either operate and implement some abstract contract types - the ITextBuffer and ITextDocument. Most of the text readers implement from .NET's TextReader type also - which we'd need to implement. Then there are a few utility types, such as StringBuilder, IEquatable<T>, IComparable<T> and of course, IDisposable.

IDisposable is an interesting concept as this is very much implemented with a language feature in C# and Visual Basic, the using statement. We can implement something similar in TypeScript, and proide that as an export function as part of library.

Firstly, let's define IDisposable:

namespace Razor
{
  export interface IDisposable
  {
    dispose(): void;
  }
}

And now let's implement our using function.

namespace Razor
{
  export function Using(contextOrDisposable: any|IDisposable, disposableOrAction: IDisposable|Function, action?: (disposable: IDisposable) => void)
  {
    if (arguments.length === 2)
    {
      action = <(disposable: IDisposable|Function) => void>disposableOrAction;
      disposableOrAction = <IDisposable>contextOrDisposable;
      contextOrDisposable = null;
    }
    
    try 
    {
      action.apply(contextOrDisposable, [disposableOrAction]);
    }
    finally
    {
      (<IDisposable>disposableOrAction).dispose();
    }
  }
}

We're taking advantage of TypeScript's ability to support multiple types as arguments - this helps us implement a form of method overloading, but it means our method has to consider and/or test arguments to determine the actual intention of the method calls. This doesn't feel entirely right to me, and it's something I may come back and revisit and refactor. Now, one thing that TypeScript handles really well, is the control of closure scope - that being the meaning of this, so although I've provided arguments for specifying the context of your disposable closure, if you're taking advantage of TypeScript's support for arrow functions (=>), it handles this for you in the generated code.

import using = Razor.Using;

export class MyClass
{
  public get someProp(): string
  {
    return "value";
  }

  public someMethod(): string
  {
    var disposable: IDisposable = // ... some disposable instance
    using (disposable, () =>
    {
      // "this" still means the instance of "MyClass
      var value = this.someProp;
    });
  }
}

Next series post

The next post in the series will deal with implementing our text services - including the SourceLocation type and our text readers.

You can follow my progress at the GitHub rep - https://github.com/Antaris/RazorJS

Meta-programming with Roslyn on ASP.NET vNext

| Tags: aspnet, dotnet, roslyn | Comments

The new ASP.NET vNext platform (ASP.NET 5) takes advantage of the new Roslyn compiler infrastructure. Roslyn is a managed-code implementation of the .NET compiler, but in reality, it is so much more than that. Roslyn is the compiler-as-a-service, realising a set of services that have long been the locked away. With this new technology, we have a new set of APIs, which allows us to understand a lot more of the code we are writing.

In ASP.NET vNext, the whole approach to the project system has changed, creating a leaner project system, built on the new Roslyn compiler services. The team have enabled some new scenarios with this approach, and one quite exciting scenario is meta-programming. That being developing programs that understand other programs, or in our instance, writing code to understand our own code, and update/modify our projects at compile time. Meta-programming with your projects can be achieved using the new ICompileModule (github) interface, which the Roslyn compiler can discover at compile time, and utilise both before and after your compilation:

public interface ICompileModule
{
    BeforeCompile(BeforeCompileContext context);

    AfterCompile(AfterCompileContext context);
}

The interesting thing about how the ICompileModule itself is used, is it is included as part of your own target assembly, and can act on code within that assembly itself.

Example project

Let's look at a project structure:

/project.json
/compiler/preprocess/ImplementGreeterCompileModule.cs
/Greeter.cs

This is a very much simplified project, and what we are going to get it to do, is implement the body of a method on our Greeter class:

public class Greeter
{
    public string GetMessage()
    {
        // Implement this method.
    }
}

So first things first, we need to make sure we have a reference to the required assemblies, so edit the project.json file and add the following:

{
    "version": "1.0.0-*",
    "dependencies": {
        "Microsoft.CodeAnalysis.CSharp": "1.0.0-*",
        "Microsoft.Framework.Runtime.Roslyn.Abstractions": "1.0.0-*"
    },

    "frameworks": {
        "dnx451": {
            "frameworkAssemblies": {
                "System.Runtime": "4.0.10.0",
                "System.Threading.Tasks": "4.0.0.0",
                "System.Text.Encoding": "4.0.0.0"
            }
        },
        "dnxcore50": {
            "dependencies": {
                "System.ComponentModel": "4.0.0-*",
                "System.IO": "4.0.10-*",
                "System.Reflection": "4.0.10-*",
                "System.Runtime": "4.0.10-*",
                "System.Runtime.Extensions": "4.0.10-*",
                "System.Threading.Tasks": "4.0.10-*"
            }
        }
    }
}

I'm not going to talk about the frameworks section and how this works as there is already a great deal already written about this concept.

The packages Microsoft.CodeAnalysis.CSharp brings in the Roslyn APIs for working with C# code, and the Microsoft.Framework.Runtime.Roslyn.Abstractions is a contract assembly for bridging between your code and the Roslyn compiler when using the DNX. The DNX implementation with Roslyn is what gives us the ability to use these techniques, you can check out the implementation here.

How does ICompileModule work?

One of the interest bits about how this all works, is when DNX is building your projects, it actually goes through a couple of stages, and these are very broad descriptions:

  1. Discover all applicable source files
  2. Convert to SyntaxTree instances
  3. Discover all references
  4. Create a compilation

Now at this stage, the RoslynCompiler will go ahead and discover any ICompileModules and if they exist, will create a !preprocess assembly, with the same references as your project. It will perform the same as above (steps 1-4), but with just the code in the compiler/preprocess/..., compile it, and load the assembly. The next step is to create an instance of the BeforeCompileContext class which gives us information on the current main project compilation, its references and syntax trees. When the preprocess assembly types are found, they are instantiated, and the BeforeCompile method is executed. At this stage, it sort of feels like Inception.

Implementing our Compile Module

So, now we have some of an understanding about how compile modules work, lets use one to implement some code. We start off with our basic implementation:

using System.Diagnostics;
using System.Linq;

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.Framework.Runtime.Roslyn;

using T = Microsoft.CodeAnalysis.CSharp.CSharpSyntaxTree;
using F = Microsoft.CodeAnalysis.CSharp.SyntaxFactory;
using K = Microsoft.CodeAnalysis.CSharp.SyntaxKind;

public class ImplementGreeterCompileModule : ICompileModule
{
    public void AfterCompile(AfterCompileContext context)
    {
        // NoOp
    }

    public void BeforeCompile(BeforeCompileContext context)
    {

    }
}

When working with syntax trees, I prefer to use shorthand namespace import aliases, so F is SyntaxFactory, T is the CSharpSyntaxTree type, and K is the SyntaxKind type. This short hand lets me write more terse code which is still quite readable.

So first things first, what do we need to do? Well, firstly, we need to find our Greeter class within our compilation. So we can use the context.Compilation.SyntaxTrees collection for this. So we're first going to do a little digging to find it:

// Get our Greeter class.
var syntaxMatch = context.Compilation.SyntaxTrees
    .Select(s => new
    {
        Tree = s,
        Root = s.GetRoot(),
        Class = s.GetRoot().DescendantNodes()
            .OfType<ClassDeclarationSyntax>()
            .Where(cs => cs.Identifier.ValueText == "Greeter")
            .SingleOrDefault()
    })
    .Where(a => a.Class != null)
    .Single();

We keep a reference to the tree itself, its root and the matching class declaration, as we'll need these later on. Next up, let's find our GetMessage method within our class:

var tree = syntaxMatch.Tree;
var root = syntaxMatch.Root;
var classSyntax = syntaxMatch.Class;

// Get the method declaration.
var methodSyntax = classSyntax.Members
    .OfType<MethodDeclarationSyntax>()
    .Where(ms => ms.Identifier.ValueText == "GetMessage")
    .Single();

Of course, we're ignoring things like overloads, etc. here, these are things you would need to consider in production code, but as an example, this is very naive. Now we have our method, we need to implement our body. What I want to create, is a simple return "Hello World!" statement. Now you could shortcut this by using SyntaxFactory.ParseStatement("return \"Hello World!\"");, but let's try building it from scratch:

// Let's implement the body.
var returnStatement = F.ReturnStatement(
    F.LiteralExpression(
        K.StringLiteralExpression,
        F.Literal(@"""Hello World!""")));

So here we are creating a return statement using the SyntaxFactory type. The body of the statement is implemented as the return keyword + a string literal for "Hello World!".

Next up, we need to start updating the syntax tree. The thing to remember at this point, is that compilations, syntax tree, and its nodes are immutable. That being they are read-only structures, so we can't simply add things to an existing tree, we need to create new trees, and replace nodes. So with the next couple of lines, let's do that.

// Get the body block
var bodyBlock = methodSyntax.Body;

// Create a new body block, with our new statement.
var newBodyBlock = F.Block(new StatementSyntax[] { returnStatement });

// Get the revised root
var newRoot = (CompilationUnitSyntax)root.ReplaceNode(bodyBlock, newBodyBlock);

// Create a new syntax tree.
var newTree = T.Create(newRoot);

We've doing a couple of things here, we've first obtained the body block of the GetMessage method declaration. Next, we create a new block with our returnStatement. We then need to go back to the root node, and tell it to replace the bodyBlock node with the newBodyBlock node It does this, but returns us a new root node. The original root node is left unchanged, so to finish that off, we have to create the new syntax tree from this revised root node.

Lastly, we'll replace the current syntax tree with our new one:

// Replace the compilation.
context.Compilation = context.Compilation.ReplaceSyntaxTree(tree, newTree);

If you build now, even though the Greeter.GetMessage method does not currently have an implementation, it will build fine, because we've now dynamically implemented it using our ImplementGreeterCompileModule.

So our complete implementation looks like this:

using System.Diagnostics;
using System.Linq;

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.Framework.Runtime.Roslyn;

using T = Microsoft.CodeAnalysis.CSharp.CSharpSyntaxTree;
using F = Microsoft.CodeAnalysis.CSharp.SyntaxFactory;
using K = Microsoft.CodeAnalysis.CSharp.SyntaxKind;

public class ImplementGreeterCompileModule : ICompileModule
{
    public void AfterCompile(AfterCompileContext context)
    {
        // NoOp
    }

    public void BeforeCompile(BeforeCompileContext context)
    {
        // Uncomment this to step through the module at compile time:
        //Debugger.Launch();
    
        // Get our Greeter class.
        var syntaxMatch = context.Compilation.SyntaxTrees
            .Select(s => new
            {
                Tree = s,
                Root = s.GetRoot(),
                Class = s.GetRoot().DescendantNodes()
                  .OfType<ClassDeclarationSyntax>()
                  .Where(cs => cs.Identifier.ValueText == "Greeter")
                  .SingleOrDefault()
            })
            .Where(a => a.Class != null)
            .Single();

        var tree = syntaxMatch.Tree;
        var root = syntaxMatch.Root;
        var classSyntax = syntaxMatch.Class;

        // Get the method declaration.
        var methodSyntax = classSyntax.Members
            .OfType<MethodDeclarationSyntax>()
            .Where(ms => ms.Identifier.ValueText == "GetMessage")
            .Single();

        // Let's implement the body.
        var returnStatement = F.ReturnStatement(
            F.LiteralExpression(
                K.StringLiteralExpression,
                F.Literal(@"""Hello World!""")));

        // Get the body block
        var bodyBlock = methodSyntax.Body;

        // Create a new body block, with our new statement.
        var newBodyBlock = F.Block(new StatementSyntax[] { returnStatement });

        // Get the revised root
        var newRoot = (CompilationUnitSyntax)root.ReplaceNode(bodyBlock, newBodyBlock);

        // Create a new syntax tree.
        var newTree = T.Create(newRoot);

        // Replace the compilation.
        context.Compilation = context.Compilation.ReplaceSyntaxTree(tree, newTree);
    }
}

I've added a Debugger.Launch() step in which can be useful if you want to step through the compilation process. Simply hit a CTRL+SHIFT+B build and attach to the IDE instance.

Where do we go from here?

There are a myriad of possibilities with this new technology, and the areas I am very much interested in, is component modularity. In future posts, I'll show you how you can use a compile module to discover modules in your code and/or nuget packages and generate dependency registrations at compile time.

I've added the code as both a Gist and pushed it to a public repo on GitHub