Skip to content

S3cmd S3 sync memory usage solution?

1 pointjaddison1 comment
On HN

I'm sure I'm not the only one to encounter OOM (MemoryError) issues using `s3cmd` to sync files to AWS S3. I believe that the culprit is that the local/remote file lists are stored in memory.

Has anyone got any novel solutions to this? As for fixing s3cmd itself, I've suggested the use of sqlite3 as a replacement for in-memory lists: https://github.com/s3tools/s3cmd/issues/408

A bad path? Any better options?

Comments

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.