Waypoint and JVM

What are we doing here?

We’re going to to run Waypoint to build a Clojure web application, deploy it, and release it on a URL. Along the way, we’re going to see how it interacts with a DigitalOcean Kubernetes cluster and a Harbor image registry.

Waypoint in a Paragraph

Waypoint is a new build, deploy, and release platform from Hashicorp – the same great folks who gave us Terraform for Infrastructure as Code and Vault for Secrets Management. Waypoint’s big selling point is that it is simple – you merely add modifications to your waypoint.hcl file and call “waypoint init” to setup your project and “waypoint up” to see it go. There’s a number of cool examples they have made available and I highly recommend that you consider doing their Introductory Tutorial – but don’t worry, I will cover some basics on the way.

Clojure in a Paragraph

Clojure is a general-purpose, functional language with a huge focus on immutability and general lisp-like syntax. However, its relevance here is that it is written in Clojure structure and compiled into a jar file, to be ran on a JVM. However, in this example, we’ll presume that you don’t want to write all the compile time and build and simply want to write Clojure code and end up with a functional, basic website. If you want to learn more about Clojure, I high recommend Daniel Higginbotham’s Clojure for the Brave and True. Knowledge of Clojure is not required for this minor tutorial however!

How does it work?

For those of you, like me, that are impatient – I have the entire Gitlab repo and an automated CI pipeline available here. If you’re curious, I also have a full one using Go on GitHub.

One major assumption of this tutorial is that you have a Waypoint server deployed somewhere that you have a token and a URL for. The Waypoint introduction provides an easy way to do this using Docker Desktop! This also uses Kubernetes for deployment, but you can easily modify this for any deployment target.

Waypoint and Buildpacks

As part of Waypoint’s HCL file, it builds the raw code using a Buildpack. Buildpacks, at a high level, do the necessary steps to build in an atomic, yet automated way. Although Heroku, Google, and Cloud Foundry are the main ones used, someone could write and use any that the HCL file can reference – very useful for internal builds!

 build {
    use "pack" {
    builder="paketobuildpacks/builder:base"
    }
    registry {
      use "docker" {
        image = "harbor.ravegrunt.com/waypointdemos/waypoint-clojure"
        tag  = "latest"
      }
    }
  }

This example leverages Cloud Foundry’s “Paketo” buildpack, as it supplies everything needed for Clojure out of the box. This buildpack simply looks for a type of file, in this case a “project.clj” and determines it to be a Clojure project. Without changing anything, you could copy this HCL into a Go project, Python project, or a multitude of others, and the buildpack can detect it, such as looking for a main.go for Go or a Gemlock file for Ruby. In practice, this looks something like the below for a Clojure project

2020/11/22 09:18:36.343653 INFO: ===> DETECTING
[detector] 8 of 18 buildpacks participating 
[detector] paketo-buildpacks/ca-certificates 1.0.1 
[detector] paketo-buildpacks/bellsoft-liberica 5.2.1 
[detector] paketo-buildpacks/leiningen 1.2.1 
[detector] paketo-buildpacks/executable-jar 3.1.3 
[detector] paketo-buildpacks/apache-tomcat 3.1.0 
[detector] paketo-buildpacks/dist-zip 2.2.2 
[detector] paketo-buildpacks/spring-boot 3.4.0 
[detector] paketo-buildpacks/procfile 3.0.0

While building this image, it runs Leiningen’s uberjar command to pack it into a jar file. The true magic here is the Procfile, which tell the image what to do when it’s ran. For instance, in my web-app, my Procfile indicates to simply run the web-app.core class. This file sits in the root repository, at the same level as the waypoint.hcl file.

web: java web_app.core

That way, when Paketo builds it, it references this function. Running Waypoint up shows, in the build logs:

[builder] Executing lein uberjar
[builder] Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
[builder] Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
[builder] Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
[builder] Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
[builder] Retrieving ring/ring/1.8.2/ring-1.8.2.pom from clojars
[builder] Retrieving ring/ring-core/1.8.2/ring-core-1.8.2.pom from clojars
[builder] Retrieving org/clojure/clojure/1.7.0/clojure-1.7.0.pom from central
[builder] Retrieving org/sonatype/oss/oss-parent/7/oss-parent-7.pom from central
[builder] Retrieving ring/ring-codec/1.1.2/ring-codec-1.1.2.pom from clojars
[builder] Retrieving org/clojure/clojure/1.5.1/clojure-1.5.1.pom from central
[builder] Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
[builder] Retrieving commons-codec/commons-codec/1.11/commons-codec-1.11.pom from central
[builder] Retrieving org/apache/commons/commons-parent/42/commons-parent-42.pom from central
[builder] Retrieving org/apache/apache/18/apache-18.pom from central
[builder] Retrieving commons-io/commons-io/2.6/commons-io-2.6.pom from central
[builder] Retrieving commons-fileupload/commons-fileupload/1.4/commons-fileupload-1.4.pom from central
[builder] Retrieving org/apache/commons/commons-parent/47/commons-parent-47.pom from central
[builder] Retrieving org/apache/apache/19/apache-19.pom from central
[builder] Retrieving commons-io/commons-io/2.2/commons-io-2.2.pom from central
[builder] Retrieving org/apache/commons/commons-parent/24/commons-parent-24.pom from central
[builder] Retrieving org/apache/apache/9/apache-9.pom from central
[builder] Retrieving crypto-random/crypto-random/1.2.0/crypto-random-1.2.0.pom from clojars
[builder] Retrieving org/clojure/clojure/1.2.1/clojure-1.2.1.pom from central
[builder] Retrieving commons-codec/commons-codec/1.6/commons-codec-1.6.pom from central
[builder] Retrieving org/apache/commons/commons-parent/22/commons-parent-22.pom from central
[builder] Retrieving crypto-equality/crypto-equality/1.0.0/crypto-equality-1.0.0.pom from clojars
[builder] Retrieving ring/ring-devel/1.8.2/ring-devel-1.8.2.pom from clojars
[builder] Retrieving hiccup/hiccup/1.0.5/hiccup-1.0.5.pom from clojars
[builder] Retrieving clj-stacktrace/clj-stacktrace/0.2.8/clj-stacktrace-0.2.8.pom from clojars
[builder] Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from central
[builder] Retrieving ns-tracker/ns-tracker/0.4.0/ns-tracker-0.4.0.pom from clojars
[builder] Retrieving org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.pom from central
[builder] Retrieving org/clojure/pom.contrib/0.1.2/pom.contrib-0.1.2.pom from central
[builder] Retrieving org/clojure/java.classpath/0.3.0/java.classpath-0.3.0.pom from central
[builder] Retrieving ring/ring-jetty-adapter/1.8.2/ring-jetty-adapter-1.8.2.pom from clojars
[builder] Retrieving ring/ring-servlet/1.8.2/ring-servlet-1.8.2.pom from clojars
[builder] Retrieving org/eclipse/jetty/jetty-server/9.4.31.v20200723/jetty-server-9.4.31.v20200723.pom from central
[builder] Retrieving org/eclipse/jetty/jetty-project/9.4.31.v20200723/jetty-project-9.4.31.v20200723.pom from central
[builder] Retrieving javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.pom from central
[builder] Retrieving net/java/jvnet-parent/3/jvnet-parent-3.pom from central
[builder] Retrieving org/eclipse/jetty/jetty-http/9.4.31.v20200723/jetty-http-9.4.31.v20200723.pom from central
[builder] Retrieving org/eclipse/jetty/jetty-util/9.4.31.v20200723/jetty-util-9.4.31.v20200723.pom from central
[builder] Retrieving org/eclipse/jetty/jetty-io/9.4.31.v20200723/jetty-io-9.4.31.v20200723.pom from central
[builder] Retrieving org/clojure/tools.logging/1.1.0/tools.logging-1.1.0.pom from central
[builder] Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
[builder] Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
[builder] Retrieving commons-codec/commons-codec/1.11/commons-codec-1.11.jar from central
[builder] Retrieving commons-io/commons-io/2.6/commons-io-2.6.jar from central
[builder] Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
[builder] Retrieving commons-fileupload/commons-fileupload/1.4/commons-fileupload-1.4.jar from central
[builder] Retrieving org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.jar from central
[builder] Retrieving org/clojure/java.classpath/0.3.0/java.classpath-0.3.0.jar from central
[builder] Retrieving javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar from central
[builder] Retrieving org/eclipse/jetty/jetty-http/9.4.31.v20200723/jetty-http-9.4.31.v20200723.jar from central
[builder] Retrieving org/eclipse/jetty/jetty-server/9.4.31.v20200723/jetty-server-9.4.31.v20200723.jar from central
[builder] Retrieving org/eclipse/jetty/jetty-io/9.4.31.v20200723/jetty-io-9.4.31.v20200723.jar from central
[builder] Retrieving org/clojure/tools.logging/1.1.0/tools.logging-1.1.0.jar from central
[builder] Retrieving org/eclipse/jetty/jetty-util/9.4.31.v20200723/jetty-util-9.4.31.v20200723.jar from central
[builder] Retrieving ring/ring/1.8.2/ring-1.8.2.jar from clojars
[builder] Retrieving crypto-equality/crypto-equality/1.0.0/crypto-equality-1.0.0.jar from clojars
[builder] Retrieving ring/ring-devel/1.8.2/ring-devel-1.8.2.jar from clojars
[builder] Retrieving ring/ring-core/1.8.2/ring-core-1.8.2.jar from clojars
[builder] Retrieving ring/ring-codec/1.1.2/ring-codec-1.1.2.jar from clojars
[builder] Retrieving crypto-random/crypto-random/1.2.0/crypto-random-1.2.0.jar from clojars
[builder] Retrieving hiccup/hiccup/1.0.5/hiccup-1.0.5.jar from clojars
[builder] Retrieving ns-tracker/ns-tracker/0.4.0/ns-tracker-0.4.0.jar from clojars
[builder] Retrieving clj-stacktrace/clj-stacktrace/0.2.8/clj-stacktrace-0.2.8.jar from clojars
[builder] Retrieving ring/ring-jetty-adapter/1.8.2/ring-jetty-adapter-1.8.2.jar from clojars
[builder] Retrieving ring/ring-servlet/1.8.2/ring-servlet-1.8.2.jar from clojars
[builder] Compiling web-app.core
[builder] 2020-11-22 15:19:04.409:INFO::main: Logging initialized @4027ms to org.eclipse.jetty.util.log.StdErrLog
[builder] WARNING: reverse already refers to: #'clojure.core/reverse in namespace: web-app.core, being replaced by: #'clojure.string/reverse
[builder] WARNING: replace already refers to: #'clojure.core/replace in namespace: web-app.core, being replaced by: #'clojure.string/replace
[builder] Created /workspace/target/web-app-1.0.jar
[builder] Created /workspace/target/web-app-1.0-standalone.jar
[builder] Removing source code
[builder] 
[builder] Paketo Executable JAR Buildpack 3.1.3 
[builder] https://github.com/paketo-buildpacks/executable-jar
[builder] Process types: 
[builder] web: java web_app.core

Without getting too deep into what Leiningen and Clojure do, you can see it’s grabbing the dependencies and creating the final jar file – without me even needing to have Java on my machine. It does this by using the build dependencies already in the buildpack. It also has tucked the Procfile command into it, so that the container simply needs to run.

Waypoint UI

Waypoint also puts an entrypoint on this container. Although it can do many things, it allows the waypoint UI to access it, the automatic URL service to provide it a link in, and the other Waypoint functions of looking at the logs, or executing within the release itself. To bring up the Waypoint UI, it’s a nice and easy command

waypoint ui -authenticate

(Note: you will get a self-signed TLS error, and that’s okay. The Waypoint team is working on fixing that)
The UI has many handy components for easy visualization, but for now, we’ll stick with the CLI

Using a Container Registry

Waypoint then attempts to put an image into a registry. Although you can use the public registries that you might be logged into, or even Gitlab Container Registry, I use a personal registry using Harbor. This tutorial won’t cover Harbor specifically, but it’s good to recognize that it can work with any registry.

Since it runs the build locally, using the buildpack, it also pushes from where it is built, and therefore a login had been ran via

docker login -u <username> -p <password> harbor.ravegrunt.com

A future post will cover automation, and how one might do this in the Gitlab CI.

Release the Kra- Deployment!

The next steps in the Waypoint.hcl file then tell it how to release and deploy

  deploy {
    use "kubernetes" {
      probe_path = "/"
      namespace = "waypoint-clj"
      service_port = 8080
    }
  }

  release {
    use "kubernetes" {
      load_balancer = true
      port          = 8080
      namespace = "waypoint-clj"
    }
  }
}

There are many plugins for deployment, including Google Cloud Run, Kubernetes, Amazon ECS, Nomad, Azure Container Instance, Docker, and others. For this example, I will use Kubernetes.

Much like before, this assumes that you have a local kubeconfig for it to reference, and that the namespace “waypoint-clj” exists. Since I use DigitalOcean’s managed Kubernetes – which Waypoint can run using the $5 Droplet – I can call my Kubeconfig via doctl. This tutorial doesn’t cover how to install doctl or building your cluster in DigitalOcean, but there are plenty of docs available.

What this does is the deploy onto Kubenetes cluster in the designated namespace and do health checks against the root route (“/”) and sets the service port to 8080. By default, the Heroku buildpack uses port 3000, and that’s the Waypoint buildpack selected if nothing else, but GCR and Paketo both use 8080. If you’re using the Paketo buildpack, make sure to call your service_port at 8080, or it will break! This step will produce a waypoint.run URL to test, but will trigger the deployment unless otherwise told not to.

The last step turns on a load balancer and reports out to port 8080 – thus making it live. In the screenshot above, my load balancer is at 159.203.52.28:8080. And that’s it! The Clojure web app is deployed.

Verifying that it worked

If you want to test it, you can go to either the deployment URL

Or the release URL

If you wanted to let your fellow coworkers test it, they could be given those URLs and never see the code. However, due to COVID-19, my current coworker is primarily my cat, Boudicca.

So when she goes to

https://typically-pure-swine--v41.waypoint.run/Boudicca

She can be pleased by the result

Now is a good time to check the application logs, either through the UI or

waypoint logs

Which shows us, indeed

2020-11-22T09:57:14-06:00: Nov 22, 2020 3:57:14 PM clojure.tools.logging$eval9030$fn__9033 invoke
2020-11-22T09:57:14-06:00: INFO: Request: /Boudicca Remote: <MASKED>
2020-11-22T09:57:26-06:00: Nov 22, 2020 3:57:26 PM clojure.tools.logging$eval9030$fn__9033 invoke
2020-11-22T09:57:27-06:00: INFO: Request: /Boudicca Remote: 127.0.0.1

The application functioned as expected there.

Back to the topic at hand

So, we how does this impact JVM builds – well, it doesn’t, and that’s the point. It still uses a JVM behind the scene after building a jar file, but you never need to see it. You only need your Clojure code, your waypoint.hcl, and a Procfile and Waypoint handles the rest, in building your container, deploying it, and releasing it into the world.

Waypoint is also relatively agnostic to other pieces. A future blog post will show how to automate all of this with Gitlab, but by using a DigitalOcean Kubernetes and a Harbor registry, you can see that it doesn’t have an ecosystem lock in, forcing things such as Nomad to be used, or the entire Amazon ecosystem – not that either are bad, but I’ve never seen an enterprise that didn’t have at least a few different pieces.

Go forth! Try this out yourself with a language of choice, a registry of choice, and even a destination. I sincerely hope this breakdown and explanation showed you that it’s more than the default NodeJS that works – but even compiled languages and JVMs are treated the same.