I've been doing a work-oriented "what I did today" list for ~25 years, and really like it. Originally it started because I needed to bill for my time, but when I went to my current job (over a decade ago) I kept doing it. In my iteration, it is a concise sentence about each thing I've worked on that day. At the end of the month I go back through and review it and write up a "Wins" list.
It's surprisingly useful; I share it with my coworkers and we often consult it if we notice something has been behaving differently starting at a certain date to see what was going on then.
I keep it in a simple text file, running in a tmux on a server, so I have connections to it from my laptop and my desktop. It's currently 19,509 lines.
Well, I haven't lost them in over a decade... So I have a pretty good track record. The system it's hosted on has had at least one, maybe two hardware failures over that time. A system isn't done being set up until it has backups up and running.
However, that said, over ~3 decades I've found that having a successful rsync exit code and alerting when that is not true, along with periodic "full" rsync checksum runs, is effectively a failsafe way of ensuing a good backup.
For our less critical systems, this plus "spot checking" by regularly going in and looking at "what did this file look like a few weeks ago" (something we commonly use backups for), has proven pretty effective while also being low work.
For critical systems, definitely do test recoveries. Our database server, for example, every week recovers the production database into our staging and dev environments, so backup problems tend to get noticed pretty quickly.
It's surprisingly useful; I share it with my coworkers and we often consult it if we notice something has been behaving differently starting at a certain date to see what was going on then.
Not for everything, but things that I do have commit logs for it's a needle in a haystack problem of which repo the commit logs would be in. At work we have ~120 repos, at least a dozen of which I'm likely to have been in over a couple weeks. Other things are likely a ticket rather than a commit (running OS updates, switching to a new haproxy might be a commit from days or weeks earlier when done in staging but the commit log wouldn't show when it was activated in production).
It's very powerful having just a few sentences I can read about what was going on specifically on a given day.
Commit logs are isolated and per repository. In large organizations, you're usually working with numerous services, often with different owners and split across a number of repositories. Figuring out what caused something to happen can be a fairly complicated process, especially when you don't know exactly where to start digging. Having an overview like this can be invaluable and save you a lot of time.
Comments
I've been doing a work-oriented "what I did today" list for ~25 years, and really like it. Originally it started because I needed to bill for my time, but when I went to my current job (over a decade ago) I kept doing it. In my iteration, it is a concise sentence about each thing I've worked on that day. At the end of the month I go back through and review it and write up a "Wins" list.
It's surprisingly useful; I share it with my coworkers and we often consult it if we notice something has been behaving differently starting at a certain date to see what was going on then.
I keep it in a simple text file, running in a tmux on a server, so I have connections to it from my laptop and my desktop. It's currently 19,509 lines.
I’m praying you have backups. That last paragraph gives me anxiety
Well, I haven't lost them in over a decade... So I have a pretty good track record. The system it's hosted on has had at least one, maybe two hardware failures over that time. A system isn't done being set up until it has backups up and running.
When the server goes up in smoke, you won’t be able to restore from your track record.
A simple ‘scp remote local’ once a month will save you from years of “damn… if only I had backed up”
Ah. I see below you’re using rsync. Phew!
...and tested.
I only had to see one machine "that was being backed up" unable to restore from backup. Wasn't mine, but was enough to teach me to test them.
Absolutely! You're preaching to the choir here.
However, that said, over ~3 decades I've found that having a successful rsync exit code and alerting when that is not true, along with periodic "full" rsync checksum runs, is effectively a failsafe way of ensuing a good backup.
For our less critical systems, this plus "spot checking" by regularly going in and looking at "what did this file look like a few weeks ago" (something we commonly use backups for), has proven pretty effective while also being low work.
For critical systems, definitely do test recoveries. Our database server, for example, every week recovers the production database into our staging and dev environments, so backup problems tend to get noticed pretty quickly.
Don't you have commit logs for this??
Not for everything, but things that I do have commit logs for it's a needle in a haystack problem of which repo the commit logs would be in. At work we have ~120 repos, at least a dozen of which I'm likely to have been in over a couple weeks. Other things are likely a ticket rather than a commit (running OS updates, switching to a new haproxy might be a commit from days or weeks earlier when done in staging but the commit log wouldn't show when it was activated in production).
It's very powerful having just a few sentences I can read about what was going on specifically on a given day.
I think git keeps a per-branch history but in practice I have never seen it used
Commit logs are isolated and per repository. In large organizations, you're usually working with numerous services, often with different owners and split across a number of repositories. Figuring out what caused something to happen can be a fairly complicated process, especially when you don't know exactly where to start digging. Having an overview like this can be invaluable and save you a lot of time.
Commit logs typically only show the coding part, not everything else. It can also be spread out over several repos, or lost if branches are deleted.