The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience.

Guruprasad B.R. (15) [Avatar] Offline
#1
Can you share some views on how to implement a long running process (that exceeds the Lambda limit of 300 secs) in a Serverless approach?

For ex:
1. Unzipping large zip files when uploaded to S3 and storing the unzipped contents back to S3 to trigger lambda functions.
2. Executing a third party tool when a file is uploaded into S3.
472618 (13) [Avatar] Offline
#2
Lambda is not the way given the 5 min limit. Since lambda is aws's only server-less compute, if you stick with aws you'll need a server.

the best aws way to do this maybe;
1) have s3 publish event to sqs
2) trigger a lambda function off of same s3 event to start up a ec2 server (perhaps a stopped instance)
3) have the ec2 instance read the event off sqs and process
4) ec2 deletes sqs message when done and stops itself.

you could autoscale ec2 instances based on sqs queue length if needed....
Guruprasad B.R. (15) [Avatar] Offline
#3
What about AWS Batch/ Fargate/ECS...any feedback on these for such long running processes?
472618 (13) [Avatar] Offline
#4
actually aws datapipeline is perfect for the job you described. It can automatically spin up and shutdown ec2 instances for the processing to be done. Not I'd claim its pretty serverless even though it uses ec2, you dont need to do any ec2 management...