5

I need to store some files for my CF template (GraphQL schema, Lambda source, etc) into an S3 bucket that will also (hopefully) be defined in the same template, as that seems to be the only way outside of dropping the contents directly into the template to do it. I am also trying to think ahead to CI/CD, and it would be nice to have these files checked in and the CI/CD tool move them as appropriately.

Is there a way to copy files to S3 from the template? How do most people do it with CI/CD?

MLu
  • 23,798
  • 5
  • 54
  • 81
CodeChimp
  • 273
  • 1
  • 6
  • 15
  • Hi CodeChimp, if the response below answered your question please upvote and accept it. That's the ServerFault's way to say *thank you* for the time and effort someone took to help you. Thanks! – MLu Oct 18 '18 at 01:22

3 Answers3

2

I'm afraid it's not possible to upload to a bucket created from the same template. However if the bucket is pre-existing, say an account-wide deployment bucket, or one of those cf-templates-... buckets, you've got some options.

  • Upload the auxiliary files using Ansible playbook and in the same playbook create/update the CloudFormation stack. That's a very popular method for CI/CD pipelines. In your Ansible playbook you'll have something like this:

    tasks:
- name: "Upload files to s3://{{deployment_bucket}}/lambda/"
  s3_sync:
    bucket: "{{deployment_bucket}}"
    file_root: lambda/
    prefix: lambda/
    permission: private

- name: "Create CloudFormation Stack"
  cloudformation:
    stack_name: "some-stack-name"
    state: present
    template: template.yml
    template_parameters:
      DeploymentS3Path: "s3://{{deployment_bucket}}/lambda/"

  • Let aws cloudformation package zip up and upload the files to S3 and then aws cloudformation deploy create and execute a CloudFormation change set. Again quite a popular method that works well with CI/CD.

    In your CloudFormation template you can then refer to local files like this:

    MyLambda:
  Type: AWS::Lambda::Function
  Properties:
    Code: lambda/

    And when aws cloudformation package is run it will output a modified template with the code path expanded:

    MyLambda:
  Properties:
    Code:
      S3Bucket: cf-templates-1a2b3c4d5e6f-ap-southeast-2
      S3Key: e24e45d4f5f2ab3c5d437659fa2246a7

    You then aws cloudformation deploy this expanded template. Or use some other method for deployment - the template is ready to use.

    The beauty of this package / deploy method is that it handles the upload of local files to S3 for you.

In different projects we use both methods in various CI/CD pipelines and both work great.

Hope that helps :)

MLu
  • 23,798
  • 5
  • 54
  • 81
1

Another alternative is to use cfpack tool. It allows you to define what artifacts need to be uploaded to a S3 bucket. Take a look at the Artifacts section for more details, it explains how to deal with Graphql schema and resolvers.

In addition to this, cfpack allows you to split your big CloudFormation template into smaller files. This is especially useful when you have a huge CloudFormation template. So, give it a try.

Eugene Manuilov
  • 113
  • 4
1

Some AWS tools, like λ#, are able to create redeployable CloudFormation templates with assets. Take a look at this example: https://github.com/LambdaSharp/StaticWebsite-Sample

Bjorg
  • 11
  • 1