Comment on Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filterparentComments−illumin89yYou can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.−otterley9yThis was just introduced last November: http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inven...−hoodoof9ySo my database is up to a week wrong? Errr.....−posterboy9yin that it stores undeleted files, until weekly clean-upthe DB is only incomplete for as long it takes to commit to the SQL layer after storing successfully.
Comments
You can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.
This was just introduced last November: http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inven...
So my database is up to a week wrong? Errr.....
in that it stores undeleted files, until weekly clean-up
the DB is only incomplete for as long it takes to commit to the SQL layer after storing successfully.