Guruprasad B.R. (15) [Avatar] Offline
#1
Can you share some views on how to implement a long running process (that exceeds the Lambda limit of 300 secs) in a Serverless approach?

For ex:
1. Unzipping large zip files when uploaded to S3 and storing the unzipped contents back to S3 to trigger lambda functions.
2. Executing a third party tool when a file is uploaded into S3.
472618 (13) [Avatar] Offline
#2
Lambda is not the way given the 5 min limit. Since lambda is aws's only server-less compute, if you stick with aws you'll need a server.

the best aws way to do this maybe;
1) have s3 publish event to sqs
2) trigger a lambda function off of same s3 event to start up a ec2 server (perhaps a stopped instance)
3) have the ec2 instance read the event off sqs and process
4) ec2 deletes sqs message when done and stops itself.

you could autoscale ec2 instances based on sqs queue length if needed....
Guruprasad B.R. (15) [Avatar] Offline
#3
What about AWS Batch/ Fargate/ECS...any feedback on these for such long running processes?
472618 (13) [Avatar] Offline
#4
actually aws datapipeline is perfect for the job you described. It can automatically spin up and shutdown ec2 instances for the processing to be done. Not I'd claim its pretty serverless even though it uses ec2, you dont need to do any ec2 management...