Integrating CI/CD into our AWS Lambda functions

At the Lifetime Value Company (LTV), we love to experiment and try new things. We are also constantly optimizing existing processes through a continuous loop of Try, Fail, Learn, Repeat—a Core Value at LTV.

The primary responsibility of our Customer Support Operations (CSOps) team is to create and maintain software that assists Customer Support (CS) representatives with their day-to-day work, making it more manageable, efficient, and robust so we can give the best experience to our increasing customer base.

As a result, the team creates and maintains proprietary tools and leverages third-party vendors, like Zendesk, Amazon Web Services (AWS) Connect and AWS Lambda. With AWS Connect, we leverage a comprehensive call center solution that our agents use to help customers. Using Lambda, we provide the underlying solutions to deliver data and real time information.

We needed to scale our CS team due to our rapid growth and the volume of inquiries received by our CS team was increasing quickly over time. To assist the situation, we needed to introduce new solutions to automate some of the work done by our CS team in order to reduce their workload. In this context, we experimented with tools like AWS Connect and Lambda in an internal innovation hackathon to determine if they could help alleviate our scaling challenges.

With this, we achieved a Minimum Viable Product (MVP) that solved the team’s needs at the time. But as the code and the team continued to grow, the maintenance of our functions also became more complex in the following ways:

• Specific functions were hard to find within our growing collection
• We lacked version control
• It was difficult to code and deploy our changes
• We lacked an adequate testing process

We set the objective to enhance our Lambda functions and make the development process easier. Specifically, we were looking to centralize our source code so that developers could easily access it in a version-controlled environment, and roll out a deployment process that mirrored what we currently use for our main services. Together, this would give us a continuous integration/deployment (CI/CD) pipeline.

Picking the right tools

We decided to implement Serverless to hold our functions and publish them to Lambda, along with Concourse CI to handle the deployments. We’ve used these tools in other projects, so we had good references to help with the implementation.

Gathering everything in one place

After discussing a proposal for the infrastructure we aimed to implement, we gathered our project needs:

- We obtained a list of all the functions authored and maintained by the CSOps team.

- We obtained a list of all the call center flows that use of these functions.

- We created dedicated Github repositories to store and version-control each function.

- We defined a workflow based on good practices, having development comfort and security while doing a change in mind.

- We installed the tools.

Based on other similar projects, we implemented the following workflow:

• Create a new branch from the main branch, our production branch.
• Work and create atomic working commits.
• Open a PR against the dev branch.
• Ask for reviews.
• Cherry-pick and push to upstream dev branch when we have enough approvals.
• This triggers a deploy to a dev AWS account.
• Create a new Interactive Voice Response (IVR) basic phone flow in the dev Connect account to test the function if it doesn’t exist already.
• With the basic IVR phone flow, we can test it by making phone calls to the contact flow phone number.
• Once it works in dev, and is approved by the Quality Assurance (QA) team, we cherry-pick commits into main and push.
• Triggers a deployment to the AWS prod account.
• We QA/Test. If the function is new, we do a 10% and incremental traffic deviation to the new function. If the function already exists, we QA and monitor.

Setting up the project

The first step was to read and follow Serverless’ official documentation. Then, set up each function, one by one, using the following process:

• Scaffold the project with the commands
• Write the config files
• Adapt the current Lambdas source code

Along the way, we determined which variables were best handled as environment variables, as they could have sensitive information or could be different while testing locally.

We also established which environments we would need: local, staging and production. Once we got help from the DevOps team to set up the environments and their variables, …, we could start testing our functions against our development environments. It took some time to adapt the functions again to work well with Connect, but with some help we were able to integrate everything and test our first function in a sandbox environment.

After we made sure the data was correct, we could start replacing the old functions. Starting with low-risk flows, we only drove some production traffic to the new Lambda functions to detect possible issues. Fortunately, the integration process ran smoothly, and we were able to set all requests to the new Lambda functions.

Shiny new features

We also thought we shouldn’t be limited to the primary function, so we also added unplanned features that we like to call “developer ergonomics.” These would help the team in the development process, with the defined workflow mentioned above as an example.

We wanted to improve developer ergonomics as much as possible. For instance, adding our linter/formatting tools following the company standards. Having a Serverless project allows us to easily install prettier/eslint and automate this important part of having a standard code format. Now any developer that codes in the new Lambda function can adhere to the same standards. These rules are also easy to implement when creating a new Lambda function.

We also want to develop with confidence. For example, adding unit tests to the workflow to make sure we could keep a statement of the basic functionality without changing the desired outcomes. We can also be aware of what inputs the lambda functions expect.

A third and essential feature added was the concept of environments. Before, we just had the production environment to make changes. Now we can develop in local, dev and production, each with its own set of environment variables in place and with its valid data set.

A bumpy road

As always, not everything is as easy as it seems. There were also challenges, delays and issues. For instance, adapting the functions to test them with Connect was a slow process because we had to create and modify sandbox flows and ask for help to make calls to test the toll free test phone number. Along the way, we also realized that we were using deprecated libraries as the simple rest client used. The more robust Axios replaced this. Also, the code was hard to read and maintain because it used the abuse of callback functions that were partially migrated to promises with the async/await syntactic sugar. We say “partially” because AWS Connect needs a callback to be executed when integrating it with the lambda functions.

Conclusion

Although it took longer than anticipated, we successfully implemented a Continuous Integration/Continuous Deployment (CI/CD) process for our Lambdas. Through Github, Serverless and Concourse, we managed to have all our code and functions in their respective repositories. We developed enough configuration to deploy it to our AWS Lambda’s instances with just a “push”. We also got to add improvements to the existing code, such as removing deprecated libraries, adding unit testing and configuring a safe testing environment. Some features that we would also like to implement are migrating the functions to use Typescript, which again adds confidence to the code that we are developing.

It still has room for improvements so it is far from finished, but it is a good first step towards improving our team’s needs and wellbeing. We learned a lot in the process, not only technical knowledge but also professional skills and abilities. We needed to organize our resources, collaborate with other teams and people, do research and improve our critical thinking. Following LTV’s core values, we were all in to work on this project with realistic expectations. Though we had to try, fail, learn and repeat, we did our best to deliver something valuable and we are excited to start working on our next project!