# How you can build your own web framework for Node.js

Follow me on Twitter, happy to take your suggestions on topics or improvements /Chris

TLDR; this article teaches you to implement the framework Express to a certain degree. Great for your own learning but don't use in production unless you got issues with space or bandwidth doing an NPM install. Hope it's helpful

The reason I write these kinds of articles is not that I want people to reinvent the wheel but to learn from the experience. I bet if you search npmjs you would find 100s of implementations looking more or less like one of the big known frameworks, Express, Nest, Koa, or Fastify. So what would creating one more framework do? Isn't that a waste of time? I don't think so and the reason is that you can learn a lot by trying to implement it yourself. You can acquire skills that help you in your everyday web dev life. It can also set you up nicely for OSS work as you now see the Matrix.

## Implementing the Express framework

For this article, I've chosen to try to implement a part of the Express framework. What parts is that exactly?

  • Routes, Express has a way of associating specific routes and have specific code run if a route is hit. You are also able to differentiate routes based on HTTP Verb. So a GET to /products is different from a POST to /products.
  • Middleware, is a piece of code that can run before or after your request and even control what should happen to the request. Middleware is how you could inspect a header for an auth token and if valid return the asked for resources. If the token is not valid then the request stops there and a suitable message can be sent back.
  • Query parameters, this is the end part of the URL and is able to help further filter down what you want the response to look at. Given a URL looking like so /products?page=1&pagesize=20, the query parameters are everything that happens after ?.
  • Sending data with a Body, data can be sent from the client to the server application. It can be sent either over the URL or through a body. The body can contain different things, everything from JSON to simple form fields to even files.

An example express App

Let's look at a few lines of implementing an Express app. There are a lot of things happening even with a few lines:

const express = require('express')
const app = express();
app.get('/products/:id', (req, res) => {
  res.send(`You sent id ${req.params.id}`)
})

app.listen(3000, () => {
  console.log('Server up and running on port 3000')
})
1
2
3
4
5
6
7
8
9

A Vanilla HTTP app

How would we go about implementing that? Well, we have the HTTP module at our disposal. So let's look at a very small implementation to understand what's missing:

const http = require('http');
const PORT = 3000;

const server = http.createServer((req, res) => {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('hello world');
});

server.listen(PORT, () => {
  console.log(`listening on port ${PORT}`)
})
1
2
3
4
5
6
7
8
9
10
11

The HTTP module only has a very basic sense of routing. If you navigate toward such an app with URL http://localhost:3000/products the req.url will contain /products and req.method will contain the string get. That's it, that's all you have.

Implementing routing and HTTP Verbs

We are about to implement the following

  • HTTP Verb methods, we need methods like get(), post() etc.
  • Routing and route parameters, we need to be able to match /products and we need to be able to break out the route parameter id from an expression looking like this /products/:id.
  • Query parameters, we should be able to take a URL like so http://localhost:3000/products?page=1&pageSize=20 and parse out the parameters page and pageSize so that they are easy to work with.

HTTP Verb methods

Let's create a server.js and start implementing our server like so:

// server.js
const http = require('http')

function myServer() {
  let routeTable = {};
  http.createServer((req, res) => {

  });
  
  return {
    get(path, cb) {
      routeTable[path] = { 'get': cb }
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Let's leave the code like that and continue to implement the routing.

Parsing route parameters

Implementing /products is easy, that's just string comparison with or without RegEx. Digging out an id parameter from /products/:id is slightly more tricky. We can do so with a RegEx once we realize that /product/:id can be rewritten as the RegEx /products/:(?<id>\w+). This is a so-called named group that when we run the match() method will return an object containing a groups property with content like so { id: '1' } for a route looking like so /products/1. Let's show such an implementation:

// url-to-regex.js

function parse(url) {
  let str = "";

  for (var i =0; i < url.length; i++) {
    const c = url.charAt(i);
    if (c === ":") {
      // eat all characters
      let param = "";
      for (var j = i + 1; j < url.length; j++) {
        if (/\w/.test(url.charAt(j))) {
          param += url.charAt(j);
        } else {
          break;
        }
      }
      str += `(?<${param}>\\w+)`;
      i = j -1;
    } else {
      str += c;
    }
  }
  return str;
}

module.exports = parse;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

And to use it:

const parse = require('./url-to-regex');

const regex = parse("/products/:id")).toBe("/products/(?<id>\\w+)");
const match = "/products/114".match(new RegExp(regex);
// match.groups is { id: '114' }
1
2
3
4
5

Adding routing to the server

Let's open up our server.js file again and add the route management part.

// server.js
const http = require('http')
const parse = require('./regex-from-url')

function myServer() {
  let routeTable = {};
  http.createServer((req, res) => {
    const routes = Object.keys(routeTable);
    let match = false;
    for(var i =0; i < routes.length; i++) {
       const route = routes[i];
       const parsedRoute = parse(route);
       if (
         new RegExp(parsedRoute).test(req.url) &&
         routeTable[route][req.method.toLowerCase()]
       ) {
         let cb = routeTable[route][req.method.toLowerCase()];
         
         const m = req.url.match(new RegExp(parsedRoute));
         
         req.params = m.groups;
         
         cb(req, res);
         
         match = true;
         break;
       }
    }
    if (!match) {
      res.statusCode = 404;
      res.end("Not found");
    }
  });   
  
  return {
    get(path, cb) {
      routeTable[path] = { 'get': cb }
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

What we are doing is looping through all the routes in our route dictionary until we find a match. The comparison looks like this:

if (
  new RegExp(parsedRoute).test(req.url) &&
  routeTable[route][req.method.toLowerCase()]
)
1
2
3
4

Note also how the router params are parsed and placed on the params property like so:

const m = req.url.match(new RegExp(parsedRoute));
req.params = m.groups;
1
2

Query parameters

We already know that using the HTTP module, the URL will contain our route, like so /products?page=1&pageSize. The next step is to dig out those parameters. That can be accomplished by using a RegEx like and the below code:

// query-params.js
  
function parse(url) {
  const results = url.match(/\?(?<query>.*)/);
  if (!results) {
    return {};
  }
  const { groups: { query } } = results;

  const pairs = query.match(/(?<param>\w+)=(?<value>\w+)/g);
  const params = pairs.reduce((acc, curr) => {
    const [key, value] = curr.split(("="));
    acc[key] = value;
    return acc;
  }, {});
  return params;
}

module.exports = parse;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Now we need to tie that into the server code. That's just a few lines, fortunately:

const queryParse = require('./query-params.js')

// the rest omitted for brevity
ress.query = queryParse(req.url);
1
2
3
4

Sending data with a Body

Reading the body can be done by realizing that the input parameter req is of type stream. It's good to know that data arrives in small pieces, so-called chunks. By listening to the event end the client is letting is now that the transmission is complete and no more data will be sent.

You can listen to incoming data by listening to the event data, like so:

req.on('data', (chunk) => {
  // do something
})

req.on('end', () => {
  // no more data
})
1
2
3
4
5
6
7

To implement listening to data being transmitted from a client we can, therefore, create the following helper method:

function readBody(req) {
    return new Promise((resolve, reject) => {
      let body = "";
      req.on("data", (chunk) => {
        body += "" + chunk;
      });
      req.on("end", () => {
        resolve(body);
      });
      req.on("error", (err) => {
        reject(err);
      });
    });
  }
1
2
3
4
5
6
7
8
9
10
11
12
13
14

and then use it in our server code like so:

res.body = await readBody(req);
1

The full code at this point should look like this:

// server.js

const http = require('http')
const queryParse = require('./query-params.js')
const parse = require('./regex-from-url')

function readBody(req) {
    return new Promise((resolve, reject) => {
      let body = "";
      req.on("data", (chunk) => {
        body += "" + chunk;
      });
      req.on("end", () => {
        resolve(body);
      });
      req.on("error", (err) => {
        reject(err);
      });
    });
  }

function myServer() {
  let routeTable = {};
  http.createServer(async(req, res) => {
    const routes = Object.keys(routeTable);
    let match = false;
    for(var i =0; i < routes.length; i++) {
       const route = routes[i];
       const parsedRoute = parse(route);
       if (
         new RegExp(parsedRoute).test(req.url) &&
         routeTable[route][req.method.toLowerCase()]
       ) {
         let cb = routeTable[route][req.method.toLowerCase()];
         
         const m = req.url.match(new RegExp(parsedRoute));
         
         req.params = m.groups;
         req.query = queryParse(req.url);
         req.body = await readBody(req);
         
         cb(req, res);
         
         match = true;
         break;
       }
    }
    if (!match) {
      res.statusCode = 404;
      res.end("Not found");
    }
  });   
  
  return {
    get(path, cb) {
      routeTable[path] = { 'get': cb }
    },
    post(path, cb) {
      routeTable[path] = { 'post': cb }
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

At this point you should be able to call your code like so:

const server = require('./server')
const app = server();

app.get('/products/:id', (req, res) => {
  // for route /products/1, req.params has value  { id: '1' }

})
app.get('/products/', (req, res) => {
  // for route /products?page=1&pageSize=10, req.query has value  { page: '1', pageSize: '10' }
})
app.post('/products/', (req, res) => {
  // req.body should contain whatever you sent across as client
})
1
2
3
4
5
6
7
8
9
10
11
12
13

Response helpers

At this point, a lot is working. But how do you actually return data back to the client? Because you are implementing the HTTP module the res parameter can be used. By calling its end() you can send data back. Here's an example:

res.end('some data')
1

However, if you look at how Express does it, it has all sorts of helpers for this like send(), json(), html() and so on. You can have that too with a few lines of code:

function createResponse(res) {
  res.send = (message) => res.end(message);
  res.json = (message) => {
    res.setHeader("Content-Type", "application/json");
    res.end(JSON.stringify(message));
  };
  res.html = (message) => {
    res.setHeader("Content-Type", "text/html");
    res.end(message); 
  }
  return res;
}
1
2
3
4
5
6
7
8
9
10
11
12

and make sure you add it in the server code:

res = createResponse(res);
1

Middleware

Having middleware allows us to run code before or after the request, or even control the request itself. Have a look at the following code:

server.get("/protected", (req, res, next) => {
  if (req.headers["authorization"] === "abc123") {
    next();
  } else {
    res.statusCode = 401;
    res.send("Not allowed");
  }
 }, (req, res) => {
   res.send("protected route");
 });
1
2
3
4
5
6
7
8
9
10

The second argument is the middleware. It inspects req.headers for an authorization property and checks its value. If everything is ok it invokes next(). If it's not ok then the request stop here and res.send() is invoked and status code set to 401, not allowed.

The last argument is the route response you want the client to see providing they send you an ok header value.

Let's implement this. Create the following function in server.js:

function processMiddleware(middleware, req, res) {
  if (!middleware) {
    // resolve false
    return new Promise((resolve) => resolve(true));
  }

  return new Promise((resolve) => {
    middleware(req, res, function () {
      resolve(true);
    });
  });
}
1
2
3
4
5
6
7
8
9
10
11
12

Above the middleware param is being called and you can see how the last argument to it is a function that resolves a Promise like so:

middleware(req, res, function () {
  resolve(true);
});
1
2
3

For the server code to use this, there are a few steps we need to take:

  1. Ensure we register the middleware
  2. Get a hold of the middleware when we have a matching request
  3. Call the middleware

Register middleware

We need to change slightly how we register routes by first adding this helper method:

function registerPath(path, cb, method, middleware) {
    if (!routeTable[path]) {
      routeTable[path] = {};
    } 
    routeTable[path] = { ...routeTable[path], [method]: cb, [method + "-middleware"]: middleware };
  }
1
2
3
4
5
6

So trying to register a route like so:

server.get('/products', (req, res, next) => {}, (req, res) => {})
1

leads to the middleware callback being saved on a property get-middleware

Then when we register the route we do something like this instead:

return {
    get: (path, ...rest) => {
      if (rest.length === 1) {
        registerPath(path, rest[0] , "get");
      } else {
        registerPath(path, rest[1], "get", rest[0]);
      }
    },
1
2
3
4
5
6
7
8

Get a reference to the middleware

To get a reference to the middleware we can use this code:

let middleware = routeTable[route][`${req.method.toLowerCase()}-middleware`];
1

Process middleware

Finally, to run the middleware write the below code:

const result = await processMiddleware(middleware, req, createResponse(res));
if (result) {
  cb(req, res);
}
1
2
3
4

Summary

The full code is available at this repo:

https://github.com/softchris/mini-web

and it can also be used via NPM by calling:

npm install quarkhttp
1

That was a lot, routing, routing parameters, query parameters, body parsing, and middleware. Hopefully, you can now understand what's going on. Remember that there are great libraries out there for you to use that are well tested. However, understanding how things are implemented can be really beneficial for your understanding.