This is a really excellent exercise for understanding material need for refactoring.
I don't think I share the author's conclusions but it makes a really good test case.
"At idle, Elasticsearch uses:
1GB of RAM (out of 8GB total)
30% utilization of a CPU
1GB of Disk space after a week
This doesn’t sound bad, but considering this server houses a dozen containers and the next CPU intensive container uses only 2% of a CPU – Elasticsearch is too heavy. For a server that gets 15 requests a minute, the resource consumption of Elasticsearch doesn’t justify it’s use. What would happen if I received a spike of traffic? I feel like the server would fall over not because the app couldn’t handle it, but log management couldn’t handle the load."
The thing is 'uses more resources than the thing next to it' is maybe a nice referential starting point, but it doesn't mean much.
What is the actual cost/risk/benefit work out to be? Cost in terms of development, maintenance, hosting, opportunity for future expansion?
Are we going to need to 'do more soon'? Are the hosting costs even material? Can Rust be easily supported?
The bit about 'what if there are more requests' does give pause for thought, but, it could be that there's a threshold/minimum that the service needs to operate, above which there's only marginal, incremental resource consumption. I don't know, I'm only pointing out the possibility.
And the choice of Rust, the author denoted has a few nice attributes ... but is this a personal choice ... or an optimal choice? Would the Python or Java solution be cleaner? Elastisearch is based on Lucene (Java), so it might be possible to do something very fast, powerful and extensible there as well.
Thanks to the author for both 'doing it' and 'writing about it' - but I think the meta issue here hinges is the technical product case.
What is the actual cost/risk/benefit work out to be? Cost in terms of development, maintenance, hosting, opportunity for future expansion?
Right below where you quoted the author he stated his biggest risk - denial of service not due to his site having performance issues but his logging solution not being able to keep up if he had an unanticipated traffic spike.
That would indeed be a rather nasty case of self-ownage; the benefits of avoiding that should be pretty obvious.
I imagine Lucene would have fit the use case better than Elasticsearch, especially if the author found SQLite to work well. Given that the indexing and searching seems very minimal SQLite seems like the right choice, but more people should really know about Lucene. It's a great library and I've used it directly rather than go with ES for some smaller, or local projects.
Comments
Great work.
This is a really excellent exercise for understanding material need for refactoring.
I don't think I share the author's conclusions but it makes a really good test case.
"At idle, Elasticsearch uses:
This doesn’t sound bad, but considering this server houses a dozen containers and the next CPU intensive container uses only 2% of a CPU – Elasticsearch is too heavy. For a server that gets 15 requests a minute, the resource consumption of Elasticsearch doesn’t justify it’s use. What would happen if I received a spike of traffic? I feel like the server would fall over not because the app couldn’t handle it, but log management couldn’t handle the load."The thing is 'uses more resources than the thing next to it' is maybe a nice referential starting point, but it doesn't mean much.
What is the actual cost/risk/benefit work out to be? Cost in terms of development, maintenance, hosting, opportunity for future expansion?
Are we going to need to 'do more soon'? Are the hosting costs even material? Can Rust be easily supported?
The bit about 'what if there are more requests' does give pause for thought, but, it could be that there's a threshold/minimum that the service needs to operate, above which there's only marginal, incremental resource consumption. I don't know, I'm only pointing out the possibility.
And the choice of Rust, the author denoted has a few nice attributes ... but is this a personal choice ... or an optimal choice? Would the Python or Java solution be cleaner? Elastisearch is based on Lucene (Java), so it might be possible to do something very fast, powerful and extensible there as well.
Thanks to the author for both 'doing it' and 'writing about it' - but I think the meta issue here hinges is the technical product case.
Right below where you quoted the author he stated his biggest risk - denial of service not due to his site having performance issues but his logging solution not being able to keep up if he had an unanticipated traffic spike.
That would indeed be a rather nasty case of self-ownage; the benefits of avoiding that should be pretty obvious.
I imagine Lucene would have fit the use case better than Elasticsearch, especially if the author found SQLite to work well. Given that the indexing and searching seems very minimal SQLite seems like the right choice, but more people should really know about Lucene. It's a great library and I've used it directly rather than go with ES for some smaller, or local projects.