aws ftp s3 serverless

PoC aws ftp s3 serverless

The goal of this project is to provide a space for users under AWS S3 buckets (shared S3 buckets with a "folder" for every user), with a web interface to upload, browse, download and remove files.

This PoC has been done between asamo7 and davidayalas.

Features:

• It's possible to setup "users" and "admins".

• A user adds its username (email from saml response) to each key (uploading, browsing, deleting...) . An admin can see everything under a bucket or a folder. See permissions.csv.

  • In our PoC, it works with SAML-JWT (https://samltest.id for demo purposes) deployed as a lambda (https://github.com/davidayalas/saml-jwt)
  • You need a Custom Authorizer for your API Gateway to validate JWT Token (sample here backend/custom-auth/index.js).
  • Our custom authorizer queries with S3 Select the CSV and retrieves the permissions for a user and adds the buckets, folders and roles to the context.

• A user can upload folders (drag and drop) and the structure is recreated in S3

• Files can be downloaded (not directories)

• Folders can be deleted (included not empty)

• A user can create a folder

Architecture

Quick deploy

• Install serverless framework: https://www.serverless.com/framework/docs/getting-started/

• Setup AWS credentials: https://www.serverless.com/framework/docs/providers/aws/cli-reference/config-credentials/

• Update "serviceName" with your own in setup.demo.json

• Execute first-deploy.sh (give it executable permissions)

    $ chmod +x first-deploy.sh && sh first-deploy.sh
  

• It will modify some files in order to make all work. Go to https://samltest.id/upload.php and upload manually docs/sp-metadata.xml.

• Run your server and try

    $ npm run serve --prefix=frontend
  

Deployment details

• Install all dependencies

    $ npm install serverless-s3-sync
    $ npm --prefix ./backend/custom-auth install ./backend/custom-auth
    $ npm --prefix ./backend/login install ./backend/login
    $ npm --prefix ./frontend install ./frontend
  

• Update "serviceName" with your own in setup.demo.json

• Basic env variables for SAML (your own, because demo is setup):

  • SAML_CERT: you idp saml certificate as string
  • IDP_HOST: your idp
  • JWT_SECRET: to sign JWT from SAML and validate from custom authorizer

• Generate frontend distribution

    $ cd frontend
    $ npm run build
  

• Deploy demo

    $ sls deploy
    $ sls info | grep GET -m 1 | awk -F[/:] '{printf "const endpoint={get(){return '''https://"$4"/demo/''';}};export default endpoint;"}' > frontend/src/assets/js/endpoint.js
    $ sls s3sync
  

• Update your sp-metadata.xml with:

  • your issuer
  • your api gw endpoint
  • your api gw stage

• Upload this file to https://samltest.id/upload.php

Sample interface

Backend

1. Upload: this lambda generates the signature for valid uploads.

2. Browsing: this lambda retrieves the objects in a path

3. Delete: this lambda deletes recursively all objects in a path

4. Download: this lambda generate presigned urls for objects to be downloaded safely

5. Login: in this case SAML based to generate a JWT token.

   • Default setup points to samltest.id. Metadata for samltest.id is generated with https://www.samltool.com/sp_metadata.php:

   • In "Attribute Consume Service Endpoint (HTTP-POST)" you have to put your api endpoint:

       https://${api gateway id}.execute-api.${region}.amazonaws.com/${stage}/login/callback
     

   • In "entityID" update your issuer (same in environment variable for login handler in serverless.yml)

6. Custom authorizer: validates JWT token and adds extra permissions from CSV

Frontend

• New frontend is Vue based. Then follow the Getting Started

• Create file "frontend/src/assets/js/endpoint.js" with the following content replacing values with aproppiate

  const endpoint={
      get(){
          return 'https://${your api id}/${your stage}/';
      }
  };
      
  export default endpoint;

• Or you can generate it from sls info output:

    $  sls info | grep GET -m 1 | awk -F[/:] '{printf "const endpoint={get(){return '''https://"$4"/demo/''';}};export default endpoint;"}' > frontend/src/assets/js/endpoint.js
  

TODO

• Quotas
• Sharing