Interesting. What's the system's uptime? Close to 12h or rather not? If the latter, this still means 1-2 restarts per day.
Regarding crashes, do you know of any special things to look out for? I do crash dumps and log uncaught exceptions, but sometimes node simply dies without any trace in the log files.
The highest I've seen on a dyno (this is all on Heroku) is over a day, web #13 and #14 are on 22 hours now. I think they're actually eventually running out of memory or being retired and replaced by Heroku rather than crashing but I'm not sure, it's not being caught in the exception catching.
Most of the crashes come down to stupid things, it's so easy to make a mistake when you don't have a compiler watching your back. External dependencies can hurt if they're laggy or unavailable. Unterminated requests are a really easy accident as well.
At this point I just use exception catching and dump the results into Redis unless I'm specifically hunting down a bug and want the crash to occur:
Comments
I do about 300,000 - 500,000 concurrent connections on nodejs but it's all short lived web requests.
It took a while to iron out most cases that can crash, right now I have:
web.1: up for 12h
web.2: up for 12h
web.3: up for 12h
web.4: up for 12h
web.5: up for 12h
web.6: up for 4h
web.7: up for 1h
web.8: up for 12h
web.9: up for 12h
web.10: up for 12h
web.11: up for 32m
web.12: up for 12h
web.13: up for 7h
web.14: up for 7h
Interesting. What's the system's uptime? Close to 12h or rather not? If the latter, this still means 1-2 restarts per day.
Regarding crashes, do you know of any special things to look out for? I do crash dumps and log uncaught exceptions, but sometimes node simply dies without any trace in the log files.
The highest I've seen on a dyno (this is all on Heroku) is over a day, web #13 and #14 are on 22 hours now. I think they're actually eventually running out of memory or being retired and replaced by Heroku rather than crashing but I'm not sure, it's not being caught in the exception catching.
Most of the crashes come down to stupid things, it's so easy to make a mistake when you don't have a compiler watching your back. External dependencies can hurt if they're laggy or unavailable. Unterminated requests are a really easy accident as well.
At this point I just use exception catching and dump the results into Redis unless I'm specifically hunting down a bug and want the crash to occur:
http://api.playtomic.com/load.html
Wow, I like your real-time monitors :) I'm about to build mine, and this gives me some great ideas.
Thanks for sharing.