AWS Lambda and Serverless Computing – Impacts on Dev, Infrastructure, Ops and Security

Developing software as event driven programming is not new.  Most of our UI is based on events through libraries such as jQuery and AJAX.  However, AWS Lambda service from Amazon has taken this concept of event driven reactive programming to the backend processing in cloud and made it simple and intuitive to use in a few clicks.  AWS Lambda requires an event and some code associated with the event to be executed – that’s it.  No server provisioning, no auto-scaling, no server monitoring and no server patching and security.  The “events” that can be specified with AWS are growing and includes external events such as HTTPS REST endpoint API call or internal events inside AWS ecosystem such as a new S3 file or new record in a Kinesis stream.  The “code” is simply a Node Javascript or Java “function” that encapsulates the business logic. If I just have “events” and “code”, and no servers, then a number of questions arise:

  • What types of use cases are best suited for serverless apps?
  • How does my app design change due to Lambda?
  • What happens to my DevOps pipelines?
  • How do I do  “infrastructure as code” when I don’t have any infrastructure?  No dynamic infrastructure to manage?
  • Do I still need operations?  And do I still need security scans?

This article answers these key questions by analyzing the impact of AWS Lambda on cloud native app design, devops pipeline tools, cloud lifecycle management tools (eg. CMP), operations and security.  It also discusses how serverless infrastructure could become the next disruption after containerization and virtualization in cloud by fundamentally changing the way we design, build and operate cloud native applications.

Use Cases for Server-less Computing

The key use cases where serverless computing such as AWS Lambda have been used in the last year since AWS announced this service in November of 2014 are categorized below.

  1. Simple event driven business logic that does not justify a continuously running server.  There are many use cases of the form of IFTTT (If-This-Then-That).  For example, I want to build a Git push to my custom social tool integration.  This can be easily done by writing a function in Lambda that listens for a webhook from Git push and does some action in my “social tool”.
  2. Event driven highly scalable transformation, notification, audit or analysis of data such as transforming media files from one format to another, or other such utilities can be easily written as Lambda functions and triggered as data arrives.
  3. Stateless microapplications – These applications are simple self-contained utilities or scripts that can be deployed in cloud but again do not justify a server or PaaS to run.
  4. Extreme scaling such as taking a task and breaking it down into 100’s of subtasks each doing independent invocation of the same business logic coded as Lambda function.
  5. Mobile and IoT backend that requires extreme scaling
  6. Rule based processing – Rules can be designed as Lambda functions to drive business logic.  For example, operational metrics and AWS infrastructure management itself can be done through rules (See Netflix for example).

This shows that a wide variety of applications from simple apps to complex backends can be developed in Lambda when automatic unlimited scaling and minimal operational overhead are key drivers.

Cloud Native Application Design

Cloud native application design is heavily based on microservices that can be scaled easily.  With AWS Lambda paired with AWS API Gateway services, the microservices design becomes even simpler and scalable without writing any code or management.  Each microservice is clearly focused on a single responsibility and business function, that is coded as a Lambda “function”.  The REST endpoint for this is provided by API Gateway and becomes the “event” triggering this microservice Lambda funciton.

Application = Collection of Lambda Functions: A cloud native application can be thought of as a collection of microservices or a collection of Lambda functions each with its REST endpoint or triggered from another event or called from another Lambda function.  The composition and chaining of Lambda functions can easily be done to build more complex microservice orchestration.  Also, Lambda requires a more functional and event driven programming approach when designing and building apps on it.

Hybrid: Except for some simple use cases outlined earlier, most of the complex cloud native apps will contain a mix of IaaS, PaaS, containers and Lambda functions.  This requires that a declarative blueprint of the application allow not just servers and PaaS resources but also Lambda functions as resources.  Cloud Management Products (eg. BMC CLM) should support such blueprints.

Development, DevOps and NoOps

Lambda based applications will require tools and technologies to have new integrations to AWS Lambda.

Local Development:  Developing and testing code locally by developers is currently a challenge with AWS Lambda.  Lambda requires a lot of manual steps such as creating multiple IAM roles, dependent AWS resources; zipping up code including dependencies and uploading it to AWS, retrieving logs from CloudWatch and so on. This is an opportunity for a new breed of tools and integration with existing tools to simplify the developer experience.  Source code systems such as Git,  IDE tools such as Eclipse and CI tools such as Jenkins can have more tighter integration with AWS Lambda. Imagine that I just wrote a Lambda function in Eclipse, and with a right click, I am able to test it out by pushing this new version to Amazon and running jUnit test cases before check-in without using AWS CLI/console and creating the half a dozen or so manual steps in setting up my code on AWS Lambda service.

CI/CD Pipelines: Lambda functions based microservices still require the traditional release pipelines where the code “Lambda function microservice” is built, tested and deployed in test environments.  This will require creating AWS Lambda environments for testing the Lambda function as well as integration with AWS Lambda and after the tests are run, decommissioning the Lambda environments.  Many code delivery and release management products have started to add plug-ins to Lambda in their products to achieve this capability:https://blog.codeship.com/integrating-aws-lambda-with-codeship/ and http://blog.cloudbees.com/2015/07/continuous-delivery-with-cloudbees.htm

Lifecycle Management of Lambda Functions: Cloud Management Product (CMP)

Lambda functions are just microservices that are event triggered, and hence require the complete lifecycle management.  This can be accomplished by Cloud Management Platforms (CMP) such as BMC CLM.  As noted earlier, complex blueprints could have both Lambda resources as well as traditional cloud resource such as AMI machines.  Developers and admins require ability to declaratively specify applications consisting of Lambda and regular cloud resources as well as multiple deployment models such as dev, test, staging and prod.  CMP then takes these blueprints and provisions resources.  In case of Lambda, the resources such as API gateway and Lambda functions will be provisioned together with all the security and configuration needed today to set them up properly (such as IAM roles, connecting API gateway linkages to Lambda and so on).  Orchestration and visualization of a set of Lambda and non-Lambda cloud resources are additional capabilities which CMPs will provide.  End users and operations require capabilities to visualize all the Lambda offerings and Lambda functions that they have provisioned for development or production workloads.  There are also a number of day 2 actions possible on Lambda functions such as increasing the memory or timeout for Lambda function.  Many cloud provisioning tools have started to build provisioning support for Lambda – see https://www.terraform.io/docs/providers/aws/r/lambda_function.html

Security, Logs and Monitoring

Even though Lambda functions abstract the servers, security is still needed.  Vulnerability and web pen test of the Lambda functions will be critical in assessing that there are no application level vulnerabilities.  Compliance rules can also be written for such functions that would require evaluation.

Lambda functions as microservices require log analysis and monitoring and traditional tools will require plug-ins to be added to support this.

What next?

We believe that serverless computing thinking will be a disruptive force similar to containerization.  We also believe that event driven computing is not suitable for all applications and we expect to see microservices that are implemented using a mix of containers, virtual machines or Lambda functions.  AWS Lambda will require our DevOps, application release management, cloud management and security tools to adapt for effective operations as described here.

Advertisements