Example: Debugging an Application with mirrord

Setup

Prerequisites

Clone Repo

First clone the example repo

git clone https://github.com/metalbear-co/nodejs-example.git && cd nodejs-example

Setup Cluster

For this example you will need to create a few deployments and services

kubectl apply -f app.yaml

This command will create four deployments and relevant services

  • example-pg - Postgres Database
  • example-idp - Dexidp OIDC provider
  • example-blog - NodeJS application located in ./blog
  • example-auditor - NodeJS application located in ./auditor

After first initialization the example-idp and example-blog deployments will not work. To fix this we need to run some migrations.

Migrations

First, establish a connection to the cluster’s database in a separate shell

kubectl port-forward svc/example-pg 5432:5432

And then run the migration script

./run_migration.sh

Note: if you don’t have golang-migrate installed you need to verify that docker can mount the ./migrations folder. If the script fails refer to docker docs for linux/macOS.

Traffic Mirroring

A key feature of mirrord is mirroring requests from a Kubernetes container to a local process for debugging purposes.

For this example, you will first need to build the local auditor service

yarn && yarn workspace auditor build

Next you’ll need to get the pod name from your cluster

kubectl get po -l app=example-auditor

The result should be something like

NAME                            READY   STATUS    RESTARTS   AGE
example-auditor-<hash>-<hash>   1/1     Running   0          5m

Now lets debug the container in the cluster by executing

mirrord exec -p example-auditor-<hash>-<hash> yarn -- workspace auditor start

In a separate terminal establish a connection to the remote blog container by running

kubectl port-forward svc/example-blog 8080:8080

Navigate to http://localhost:8080 and you should start seeing “audit” logs on your local process. The requests that the example-blog is sending to the example-auditor in the cluster are mirrored and sent to your local machine as well.

Command breakdown

mirrord exec-p example-auditor-<hash>-<hash>yarn– workspace auditor start
specify the running pod to mirrorexecutableexecutable args

Outgoing Traffic

Another key feature is outgoing traffic tunneling and remote DNS resolution. For example, let’s say you want to test a request your service makes to a database, because you encountered some unwanted behavior or just want to develop something new and want to see the real results

Let’s make a small change to ./blog/pages/index.tsx that should print the raw response from the database when accessing the root page of the blog

export const getServerSideProps: GetServerSideProps = async ({ req }) => {
  const authorised = req.cookies['oauth-access-token'] ? !!await jwtVerify(req.cookies['oauth-access-token'], await getRemoteJWKSet()) : false;
 
-  const { rows } = await pg.query(listBlogPreviews.compile({ authorised }));
+  const result = await pg.query(listBlogPreviews.compile({ authorised }));
+
+  console.log(result);
 
   return {
     props: {
-      posts: rows
+      posts: result.rows
     },
   }
 }

Now lets see the output

First things first you need to get the pod name from your local cluster

kubectl get po -l app=example-blog

The result should be something like

NAME                            READY   STATUS    RESTARTS   AGE
example-blog-<hash>-<hash>      1/1     Running   0          5m

<hash> should be the deployment and replica-set hashes generated on your cluster

And now lets test it.

mirrord exec --no-fs -x NODE_ENV -p example-blog-<hash>-<hash> yarn -- workspace blog dev

Note: connection to the remote blog container via port-forward is still required for this step, please reference Mirroring.

Your local service accesses the database from the internal network of the Kubernetes cluster. You should be able to navigate to http://localhost:8080, and though you receive the result from the server instance running on the cluster, you should still see the following log in the local machine:

Result {
  command: 'SELECT',
  rowCount: 2,
  oid: null,
  rows: [
    { id: '100', title: 'Lorem ipsum' },
    { id: '101', title: 'de Finibus Bonorum et Malorum 1.10.32' }
  ],
  fields: [
    Field {
      name: 'id',
      tableID: 16407,
      columnID: 1,
      dataTypeID: 20,
      dataTypeSize: 8,
      dataTypeModifier: -1,
      format: 'text'
    },
    Field {
      name: 'title',
      tableID: 16407,
      columnID: 2,
      dataTypeID: 25,
      dataTypeSize: -1,
      dataTypeModifier: -1,
      format: 'text'
    }
  ],
  _parsers: [ [Function: parseBigInteger], [Function: noParse] ],
  _types: TypeOverrides {
    _types: {
      getTypeParser: [Function: getTypeParser],
      setTypeParser: [Function: setTypeParser],
      arrayParser: [Object],
      builtins: [Object]
    },
    text: {},
    binary: {}
  },
  RowCtor: null,
  rowAsArray: false
}

Lets break down the command

mirrord exec–no-fs-x NODE_ENV-p example-blog-<hash>-<hash>yarn– workspace blog dev
disable fs*exclude NODE_ENV enviroment variable**specify the running pod to mirrorexecutableexecutable args

* need to disable remote fs access because of yarn caching
** NODE_ENV is used in webpack for compilation and it can cause conflict when running the app with dev command

Traffic Steal

Traffic Steal lets you handle and respond to incoming requests to the pod from your local machine. In contrast to Mirroring, with Steal the response to the incoming request is sent by the local process rather than the remote pod.

First thing we need to establish a connection to the remote idp. In a separate terminal, run the following command

kubectl port-forward svc/example-idp 5556:5556

Now login into the blog by navigating to http://localhost:8080/api/login with the following credentials

email: [email protected]
pass: password

You should see a third blogpost return after the authentication is finished but if you click on it you will get a Page Not Found message. Lets fix it

Edit ./blog/pages/blogpost/[id].tsx

@@ -1,3 +1,4 @@
+import { jwtVerify } from 'jose';
 import type { QueryResult } from 'iql';
 import type { NextPage, GetServerSideProps } from 'next'
 import Head from 'next/head'
@@ -6,6 +7,7 @@ import Link from 'next/link'
 
 import styles from '../../styles/Home.module.css'
 
+import { getRemoteJWKSet } from '../../lib/oidc';
 import pg from '../../lib/pg';
 
 import { findBlogPreviews } from '../../query/blog';
@@ -45,9 +47,11 @@ const Home: NextPage<{ post?: BlogPost }> = ({ post }) => {
 
 
 export const getServerSideProps: GetServerSideProps = async (context) => {
+  const authorised = context.req.cookies['oauth-access-token'] ? !!await jwtVerify(context.req.cookies['oauth-access-token'], await getRemoteJWKSet()) : false;
+
   let id = (Array.isArray(context.params?.id) ? context.params?.id[0] : context.params?.id) ?? '0';
 
-  const { rows } = await pg.query<QueryResult<typeof findBlogPreviews>>(findBlogPreviews.compile({ id }));
+  const { rows } = await pg.query<QueryResult<typeof findBlogPreviews>>(findBlogPreviews.compile({ authorised, id }));
 
   return {
     props: {

And lets see the results by simply adding --steal to the previous command

mirrord exec --no-fs -x NODE_ENV -p example-blog-<hash>-<hash> --steal yarn -- workspace blog dev

Refresh the page, and you should now be able to see the result.

Note: connection to the blog container is still required for this step, please reference Mirroring.

Summary

mirrord can be used to either mirror incoming traffic without influencing the return values or it can be used to “override” the container, both options with full access to other resources in the cluster I.E databases other services or even mounted filesystem/configmap/secret.
This allows you to debug your process as if it was already deployed as part of the cluster.

Teardown

After you’re done with the example you can teardown the cluster with

kubectl delete -f app.yaml