I'm not sure the rationale behind this. That is, the rationale section of the proposal does a poor just of explaining any case where this is actually a problem.
In every case where I've seen this problem, it's a matter of people either not storing the timezone along with the local time, or not storing in UTC time. A local time with a timezone is a unique time, it does not occur twice. A UTC time additionally does not occur twice. Store a time zone along with the date and time or store in UTC and convert on use.
Note: If there are instances where a second is repeated, it's rare special occurrence that developing a formalized interface for seems like overkill.
A local time with a timezone is a unique time, it does not occur twice.
If you define timezone as an IANA timezone, this is incorrect: a whole slew of local times repeat during a DST fallback event: you'll have a (1:30 AM (dst=True), America/New_York), and then a (1:30 AM (dst=False), America/New_York); that "dst=True|False" bit is the only difference, and that needs to get stored. If you consider "America/New_York" to be the TZ, then storing that bit on the TZ isn't appropriate, as it depends on a particular timestamp.
If you've ever worked with PyTZ, there's a sort of rule of "just call normalize() always"; otherwise, you'll get funny answers to some introspections on the datetime instance: things like the offset being not what a local would say the offset should be. My understanding is that pytz stores the dst flag on the timezone instance itself; things get funny because the timezone instance is not given a chance to update after arithmetic on the datetime instance.
(Really, I feel like the whole thing would work better if there was a separate class for "instant in time" and a function for, "convert this instant in time to Gregorian year/month/day/etc. in this TZ", which then returned a broken-out-type. (And a reverse, of course, for building "Instant" instances.))
UTC datetime + IANA TZ (if relevant) is the way to go. Alas, not all data is so nice.
Really I was thinking of it as a distinct timezone that must be tracked. Whether the DST and non-DST versions label themselves as such, the representation used to track time must distinguish whether DST is active or not to display the correct local time. Really, when I say store local time + timezone, I mean local time plus identifier that gets you to the same unique timezone representation in your medium (python, in this case).
Personally, I just always convert to UTC and store that. It changes the problem from one of data fidelity to display or computation annoyance, and annoyances are easy to reduce or eliminate with tooling.
Really I was thinking of it as a distinct timezone that must be tracked. Whether the DST and non-DST versions label themselves as such, the representation used to track time must distinguish whether DST is active or not to display the correct local time. Really, when I say store local time + timezone, I mean local time plus identifier that gets you to the same unique timezone representation in your medium (python, in this case).
I guess that's my point: the IANA identifier is a well-known way to serialize a TZ, but doesn't include DST flags because they're not relevant. I think if you wanted to store something like a Python tzinfo object, the easiest way is just storing (local time, offset from UTC); (maybe (local time, offset from UTC, IANA TZ ID), if you want to keep the TZ)
tzinfo's don't really have a defining quality in Python, I've found. You can end up — depending on libraries used — with two tzinfos that both conceptually are "UTC", but don't compare equal…
Now that I've thought about it again, I'm not entirely sure that the DST flag + TZ name by itself is sufficient, mostly in the case of a TZ deciding to change their offset.
just always convert to UTC and store that. It changes the problem from one of data fidelity to display
Note that the proposal doesn't fix arithmetic. Adding a 60 minute timedelta to 1:30 on the transition day will give the same result (2:30, first=True) whether first=True or first=False.
I believe the use case is for scheduling systems that need to do something at (say) 1:30am. On every day at 1:30am, please run the accounting job.
In the US, once a year you either have to have some locking system set up to avoid the second run (leave a trace that the job has already been started for the day). I believe the person proposing this thinks that it would make this determination easier.
The problem is that it only solves half the trouble caused by dual timezones. There's another day of the year when 2:30am (in the US) doesn't happen at all. If something is scheduled to occur once per day at 2:30am, then that day it is not going to happen.
There are other workarounds available such as avoiding the magic hours around 2am (in the US). But it seems to be a common problem that everyone seems to keep re-solving.
Storing the timezone with the time doesn't really solve the above issue.
I believe the use case is for scheduling systems that need to do something at (say) 1:30am. On every day at 1:30am, please run the accounting job.
In the instance where someone wants something to happen at 1:30 AM, if they aren't specific in the specification, then they should expect that it may happen twice or not at all at at certain times of the year. This is an imprecise specification problem, not a problem in representing time in structures that can and do contain timezones. That is, it's a failure of cron, or the user specifying the time, take your pick. What it's not is a failure of dates, times and timezones, which specifically address this problem. Timezones or UTC (which is just timezone offset 0), are what we have to deal with this specific problem.
For example, specifying the originating timezone would disambiguate the time, as would specifying it in UTC, (or automatically converting to and using the UTC equivalent on entry).
Storing the timezone with the time doesn't really solve the above issue.
It doesn't address the "do this thing at this local time daily" problem when someone chooses a time that has special behavior, but it does address the "do this thing at this offset from UTC daily" which may be the best you can expect when specifying a time for a recurring action and not taking into account timezones. If you want to use local time, you have to deal with either the actually time the job runs possibly shifting slightly throughout the year in some locations, or possibly running twice or not at all.
My real problem with the proposal is that adding a .first() method doesn't solve anything, and really just makes half the problem (it doesn't work if the time doesn't exist), and in a way that's already easily solved, since you can't get a valid result from .first without knowing the timezone already.
Storing it as an offset from UTC has the effect that the task will run at the wrong time 1/4 of the year.
Now, you'll probably claim that I shouldn't expect time to behave this way. But this is exactly the way everybody expect time to behave, thus, no I won't agree with you.
Anyway, yes, that still does not solve the entire problem. Time intervals are really messy.
Now, you'll probably claim that I shouldn't expect time to behave this way. But this is exactly the way everybody expect time to behave, thus, no I won't agree with you.
You shouldn't expect time to behave that way. Cron-like programs also should try to do the right thing. What's really missing is an up-front understanding of what users are to expect when entering a time into a program that supports recurring time. This is a failure of most programs to account for common misconceptions.
This proposal doesn't really help this, as it only detects the instances where we duplicate time, and only where you already have a time zone applied.
This whole working in UTC and transforming the time 'at the last minute' into the timezone is only the current accepted practise precisely because of this problem of the time changing. Ideally, we should be able to just set tzinfo to the timezone location that we want when we construct the time and then we work with that object. Yes, even if we do arithmetic in local time, it should just work.
As far as I read it this pep should eventually lead to a better pytz api. Currently, we have to use the normalize() and localize() functions in order to handle the ambiguous times that happen twice a year when the clocks change, this is ugly and hard to remember to do. I think that the 'first' flag should eventually allow us (once suitable timezones are created in pytz) to do arithmetic with local times and automatically transform between the summer and winter timezones.
I don't think this is going to fix pytz. There's a difference in philosophy - Python expects a tzinfo object that's intelligent and responds to the datetime that's passed into its calls, while pytz expects to be configured once then provides a fixed offset after that.
Seconds are not repeated AFAIK. Leap seconds are only added and the "seconds" field goes 58, 59, 60, 00, 01...
I also completely second your point. Naive datetime objects should only ever be used to describe UTC. Anything else should have an explicit timezone attached to it. Doing otherwise is asking for trouble.
[POSIX] is neither a linear representation of time nor a true representation of UTC […] The Unix time number increases by exactly 86400 each day […] Observe that when a positive leap second occurs (i.e., when a leap second is inserted) the Unix time numbers repeat themselves.
The upside to this is that days are always 86400 seconds "long": computing the start of the next day is simple. (Computing the length is not so much, and computing to-the-second elapsed time is also harder.)
This is somewhat relevant to Python, as the datetime module "ignores" (which I interpret to mean, "repeats the prior second"; I've never watched to see what really happens) leap seconds.
Comments
I'm not sure the rationale behind this. That is, the rationale section of the proposal does a poor just of explaining any case where this is actually a problem.
In every case where I've seen this problem, it's a matter of people either not storing the timezone along with the local time, or not storing in UTC time. A local time with a timezone is a unique time, it does not occur twice. A UTC time additionally does not occur twice. Store a time zone along with the date and time or store in UTC and convert on use.
Note: If there are instances where a second is repeated, it's rare special occurrence that developing a formalized interface for seems like overkill.
If you define timezone as an IANA timezone, this is incorrect: a whole slew of local times repeat during a DST fallback event: you'll have a (1:30 AM (dst=True), America/New_York), and then a (1:30 AM (dst=False), America/New_York); that "dst=True|False" bit is the only difference, and that needs to get stored. If you consider "America/New_York" to be the TZ, then storing that bit on the TZ isn't appropriate, as it depends on a particular timestamp.
If you've ever worked with PyTZ, there's a sort of rule of "just call normalize() always"; otherwise, you'll get funny answers to some introspections on the datetime instance: things like the offset being not what a local would say the offset should be. My understanding is that pytz stores the dst flag on the timezone instance itself; things get funny because the timezone instance is not given a chance to update after arithmetic on the datetime instance.
(Really, I feel like the whole thing would work better if there was a separate class for "instant in time" and a function for, "convert this instant in time to Gregorian year/month/day/etc. in this TZ", which then returned a broken-out-type. (And a reverse, of course, for building "Instant" instances.))
UTC datetime + IANA TZ (if relevant) is the way to go. Alas, not all data is so nice.
Really I was thinking of it as a distinct timezone that must be tracked. Whether the DST and non-DST versions label themselves as such, the representation used to track time must distinguish whether DST is active or not to display the correct local time. Really, when I say store local time + timezone, I mean local time plus identifier that gets you to the same unique timezone representation in your medium (python, in this case).
Personally, I just always convert to UTC and store that. It changes the problem from one of data fidelity to display or computation annoyance, and annoyances are easy to reduce or eliminate with tooling.
I guess that's my point: the IANA identifier is a well-known way to serialize a TZ, but doesn't include DST flags because they're not relevant. I think if you wanted to store something like a Python tzinfo object, the easiest way is just storing (local time, offset from UTC); (maybe (local time, offset from UTC, IANA TZ ID), if you want to keep the TZ)
tzinfo's don't really have a defining quality in Python, I've found. You can end up — depending on libraries used — with two tzinfos that both conceptually are "UTC", but don't compare equal…
Now that I've thought about it again, I'm not entirely sure that the DST flag + TZ name by itself is sufficient, mostly in the case of a TZ deciding to change their offset.
The right thing to do, and for the right reasons.
Note that the proposal doesn't fix arithmetic. Adding a 60 minute timedelta to 1:30 on the transition day will give the same result (2:30, first=True) whether first=True or first=False.
I believe the use case is for scheduling systems that need to do something at (say) 1:30am. On every day at 1:30am, please run the accounting job.
In the US, once a year you either have to have some locking system set up to avoid the second run (leave a trace that the job has already been started for the day). I believe the person proposing this thinks that it would make this determination easier.
The problem is that it only solves half the trouble caused by dual timezones. There's another day of the year when 2:30am (in the US) doesn't happen at all. If something is scheduled to occur once per day at 2:30am, then that day it is not going to happen.
There are other workarounds available such as avoiding the magic hours around 2am (in the US). But it seems to be a common problem that everyone seems to keep re-solving.
Storing the timezone with the time doesn't really solve the above issue.
In the instance where someone wants something to happen at 1:30 AM, if they aren't specific in the specification, then they should expect that it may happen twice or not at all at at certain times of the year. This is an imprecise specification problem, not a problem in representing time in structures that can and do contain timezones. That is, it's a failure of cron, or the user specifying the time, take your pick. What it's not is a failure of dates, times and timezones, which specifically address this problem. Timezones or UTC (which is just timezone offset 0), are what we have to deal with this specific problem.
For example, specifying the originating timezone would disambiguate the time, as would specifying it in UTC, (or automatically converting to and using the UTC equivalent on entry).
It doesn't address the "do this thing at this local time daily" problem when someone chooses a time that has special behavior, but it does address the "do this thing at this offset from UTC daily" which may be the best you can expect when specifying a time for a recurring action and not taking into account timezones. If you want to use local time, you have to deal with either the actually time the job runs possibly shifting slightly throughout the year in some locations, or possibly running twice or not at all.
My real problem with the proposal is that adding a .first() method doesn't solve anything, and really just makes half the problem (it doesn't work if the time doesn't exist), and in a way that's already easily solved, since you can't get a valid result from .first without knowing the timezone already.
Storing it as an offset from UTC has the effect that the task will run at the wrong time 1/4 of the year.
Now, you'll probably claim that I shouldn't expect time to behave this way. But this is exactly the way everybody expect time to behave, thus, no I won't agree with you.
Anyway, yes, that still does not solve the entire problem. Time intervals are really messy.
You shouldn't expect time to behave that way. Cron-like programs also should try to do the right thing. What's really missing is an up-front understanding of what users are to expect when entering a time into a program that supports recurring time. This is a failure of most programs to account for common misconceptions.
This proposal doesn't really help this, as it only detects the instances where we duplicate time, and only where you already have a time zone applied.
This whole working in UTC and transforming the time 'at the last minute' into the timezone is only the current accepted practise precisely because of this problem of the time changing. Ideally, we should be able to just set tzinfo to the timezone location that we want when we construct the time and then we work with that object. Yes, even if we do arithmetic in local time, it should just work.
As far as I read it this pep should eventually lead to a better pytz api. Currently, we have to use the normalize() and localize() functions in order to handle the ambiguous times that happen twice a year when the clocks change, this is ugly and hard to remember to do. I think that the 'first' flag should eventually allow us (once suitable timezones are created in pytz) to do arithmetic with local times and automatically transform between the summer and winter timezones.
I don't think this is going to fix pytz. There's a difference in philosophy - Python expects a tzinfo object that's intelligent and responds to the datetime that's passed into its calls, while pytz expects to be configured once then provides a fixed offset after that.
Seconds are not repeated AFAIK. Leap seconds are only added and the "seconds" field goes 58, 59, 60, 00, 01...
I also completely second your point. Naive datetime objects should only ever be used to describe UTC. Anything else should have an explicit timezone attached to it. Doing otherwise is asking for trouble.
In POSIX time, seconds can repeat[1],
The upside to this is that days are always 86400 seconds "long": computing the start of the next day is simple. (Computing the length is not so much, and computing to-the-second elapsed time is also harder.)
This is somewhat relevant to Python, as the datetime module "ignores" (which I interpret to mean, "repeats the prior second"; I've never watched to see what really happens) leap seconds.
[1]: https://en.wikipedia.org/wiki/Unix_time#Leap_seconds
Agreed - and more succinctly put than I managed - clearly it's the wrong time for me :-)