Improve DX by publishing an API SDK - a CDK Serverless example

This blog demonstrates one of many ways to improve the Developer Experience(DX) for both backend and frontend. The Open API Spec(OAS) file becomes the single source of truth between the projects in the form of an API SDK.

The backend project publishes an API SDK after it is deployed so that the frontend can install it as an NPM package. This publishing happens automatically in the backend pipeline (automation not included in the code example) whenever it sees the Open API Spec(OAS) file version increased.

Follow along with the code on GitHub here: backend repo, frontend repo and the published API repo. The code examples do not include the pipeline and slack automation.

The problem

Example Corp is a consultancy company, where small projects usually only have 1 frontend and 1 backend developer. The development team consists of:

• Bob the Backend developer
• Frank the Frontend developer
  
  

Bob decides to use an Open API Spec(OAS) (commonly known as a Swagger ¹) file as the backend documentation of the API. After he coded the API endpoints, he updates the OAS file and then proceeds to give it to Frank. Frank then takes the OAS file and creates Types for his frontend React TypeScript(TS) project to use. Then he creates functions that correspond to the API calls in the file.

Whenever Bob adds, updates or removes something from the API he gives Frank a new OAS file. Frank hunts for changes and implements them. Communication between Bob and Frank works, until it doesn’t.

Repeating dialog between Bob and Frank:

Bob: It is breaking because you are not using the new endpoint that does X.

Frank: I don’t see it in the OAS file.

Bob: Oh yes, let me update it and send you the new file.

Frank: Why is X undefined on this object?

Bob: It was renamed to Y, but it looks like I forgot to send you an updated spec, let me do it quick.

Frank: Okay, thanks.

Frank: Why is X a string instead of an integer?

Bob: That should not happen, let me check and fix it.

Frank: Okay.

Frank: Why is the API returning this strange error message when the OAS file says it should be object X?

Bob: Oh, that is not supposed to happen, I will check the logs but that is on my side.

Frank: k

Sharing a Postman file is also definitely not the answer:

Frank: X is breaking, did the endpoint change?

Bob: Yes, you should look at the postman docs every day. Go through all 50 endpoints and check what changed. I am not going to tell you every time I make changes.

Frank: I quit

Frank doesn’t sound like he is having a good time, is he?

If you are lucky, all of this is caught within the development environment. Changes can still go unnoticed, even though you are using a strongly typed language like TypeScript(TS) for the frontend. These errors will be noticed by your users if you don’t have tests, forcing you to please explain every time something breaks.

What happens when more developers are onboarded? What will communication look like then?

Let’s summarize the problems:

• Documentation is an afterthought, it is updated after the code has been written.
• Lack of communication and processes between back and frontend.
• Errors are caught at runtime and are seen by your users.
• Resources (time and money) are wasted as these problems can be solved with good processes.
  
  

The solution

Documentation should be written first, the OAS file must become the source of truth for both back and frontend projects. For this to happen, Bob must route and validate both input and output so that it does not break the contract.

The API publishing can be automated. When Bob makes changes to the OAS file in the backend, he has to increase the version number. The backend pipeline will detect the version number is more than the current API version and proceed to publish the new API SDK.

Visibility is a form of communication. After publishing the package, the pipeline sends a Slack message to the build channel that has a reference to the current Commit. Frank can then see a quick difference between the old and new versions, he can even leave comments and discuss the changes. Bob will most likely inform Frank of the changes in a DM or at the next standup. Communication was automated with the Slack message, so Frank knows even if Bob does not tell him directly.

Frank can then install the new version of the NPM package (API SDK). After installation, he can run TypeScript compile (tsc) to detect if there are any breaking changes after upgrading the API SDK. This effectively moves the runtime errors to be discovered at compile time. This will also prevent any frontend builds from completing successfully in the pipeline.

Frank is quite happy at this point. He knows when changes are made on the API and can detect breaking changes easily. He also does not have to create functions for each API call, nor does he have to retype all the types the API functions are returning, everything is available in the API SDK. The integration process between back and frontend is now much more efficient saving both time and money.

To summarize, the solution:

• Documentation is the source of truth/contract between back and frontend.
• Visibility is automated using Slack to improve communication. We have a way to easily visualize changes and discuss them.
• Errors between the back and frontend are discovered at compile time and not runtime.
• Resources are used effectively.
  
  

Example projects walkthrough

Backend

The pets-backend project consists of a single API Lambda function deployed to AWS that can be accessed using the Function URL(FURL). The AWS CDK has been used to define the IaC and the Lambda is written in TypeScript. The OpenAPI Spec file can be seen here, it has two endpoints:

• Get all pets
• Get a pet by ID
  
  

It is just returning static data, deploying this should incur almost no cost. You can use the GULP plugin for your IDE to debug and run the gulpfile.js that is used as the build script. You can also run the scripts via npm run <TASK> for your convenience. Please see the README on each project for more details on the available commands.

There are two pre-build processes, the first is to generate types & validation functions from the OAS file. Types are generated with the swagger-typescript-api package. Then the API TS Types and Interfaces are transformed into JSON Schema using the ts-json-schema-generator package. The JSON Schemas are then passed to AJV and by using the standalone functionality we generate JS validation functions to be used at runtime. I described this process in more detail on this blog: TypeScript Type Safety with AJV Standalone.

It seems involved, but it is currently the fastest way to validate data on Lambda within the JS ecosystem without us having to write validation functions ourselves. There is a watch command so that whenever you make changes to the OAS file, it generates types and validation functions.

The second pre-build process is to transpile the TS code (with ES Build) used in the Lambda functions to JS. We do this externally and do not make use of the CDK construct that does this internally. This is because we need to use the same method of transpiling when testing as when deploying. Another reason is that we can not use async plugins with CDK construct.

Our generateAndPushApiSdk function creates an API folder if it does not exist and initializes GIT. It will fetch the existing repository or prompt you to create one with an empty master branch. We then generate the new API SDK into the existing directory so that, when committed, we can view a git diff. The generateApiSdk function does a few things:

• Creates the correct directories
• Ensures the package.json has correct data
• Version checks and updates the package.json version to be that of the OAS file
• Generates a single TypeScript file representing the API. We use the same package swagger-typescript-api as when we created the types for our backend API. Except it is configured to output an API TS file and not just types.
• Transpiles the TS to JS
• Copies the OAS file into the dist/ folder of the package for manual inspection after installing
  
  

You can see the lengthy generateApiSdk function as described above on the GitHub repo here. All that is left is to git commit, git push and then npm publish.

The API SDK folder is pretty standard except that we only publish the dist/ directory, that is why you see the package.json file duplicated/copied along with the top level .npmrc file.

You can see the published package on the GitHub repo.

Frontend

The pets-frontend project is used to demo the usage of the SDK API NPM packaged that is published by the backend project post-deployment. It has a basic HTML file with a JS file that is built from a TS file using browserify.

You can use the package just like any other package, example:

import {Api} from "@rehanvdm/pets-api"

const apiBaseUrl = "<YOUR URL HERE - NO TRAILING SLASH>";
const api = new Api({
   baseUrl: apiBaseUrl,
   // baseApiParams: {
   //   headers: {
   //     "Authorization": "<YOUR TOKEN HERE>"
   //   }
   // }
});

let pet = await api.pets.getPetById(1);
console.log(pet);

Note that you get autocomplete for the function (aka API arguments, with proper types) right in your IDE.

The raw OpenAPI spec file can be seen at node_modules/@rehanvdm/pets-api/schema.yaml. The api.pets.getPetById method corresponds to the GET /pets/{id} API call.

Conclusion

The integration between backend and frontend does not have to be a pain. By making the documentation the source of truth, it becomes a contract between the backend and frontend. The published NPM API SDK package enforces this contract at compile time with TypeScript compile(tsc) which reduces runtime errors. Automation creates visibility with Slack notifications and opens a channel for communication between the parties. Your frontend developers will also be much better utilized and happier.

The code can be found on GitHub: backend repo, frontend repo and the published API repo. The code examples do not include the pipeline and slack automation.

PS. We are looking for both front and backend developers to work with us at Swipe iX in South Africa. Check out our careers page and see my LinkedIn post on why you should join us.

Footnotes

1. There is a difference between the Open API Spec and Swagger, but people still use the terms interchangeably. See this link for an overview.

---

---