Erlang Date and Timezone Handling with qdate

2013-04-30

TL;DR: Use qdate for all your date formatting, converting, and timezone handling.

Both Daylight Saving Time and timezones are abominations and should be burned in the fires of Mount Doom. As I do not happen to have a pair of hobbits nearby, nor am I convinced that Mount Doom even exists in New Zealand, I must deal with these problems myself.

Beef with Erlang Date and Time handling.

Erlang's date handling is convoluted and lacking in the major functionality that's needed for proper date handling. It supports multiple formats, but in no unified manner: For example, you can use Erlang's calendar:now_to_datetime/1 to convert from the now() format to a datetime() format, but there's no uniform way to convert back to the now() format. Instead, converting back to now() format requires converting the datetime() value to gregorian seconds (seconds since Jan 1st, 1900 12am), then subtracting the number of seconds from 1900-01-01 to 1970-01-01, then doing some division and remainder stuff to convert that to the number of megaseconds and seconds, and finally sticking those values into a 3-tuple, with the last value just being 0 (since we don't have subsecond precision with basic dates).

I get why (I think) the Erlang devs chose to do it that way: because converting from now() to datetime() requires a loss of precision - Erlang's now() function uses microseconds, and converting back from datetime() to now() will result in the loss of that microsecond accuracy. But that's something that could simply be stated in the documentation: "Warning: Converting from datetime() format to now() format will result in the loss of sub-second precision".

And that's just one example. Erlang only natively supports datetime() format, now() format, and gregorian seconds, but doesn't provide good symmetry for converting to and from those formats. And of course, Erlang doesn't provide any native date and time parsing or formatting - it's all tuples of integers.

And that doesn't even take into consideration timezones.

No timezone handling

Erlang's timezone handling is primitive at best: you have the choice of the OS-defined Localtime (likely the Timezone Environment variable), and the Univeral Time (GMT).

And while everyone knows PHP sucks, it does have the possibly unintended benefit of being able to deal with timezones in a clever way. Since every PHP page request has it's own OS process, you can change the timezone for that single request by changing the OS-process's environment variable. This OS-process isolation allows us to change timezones per-process without affecting other users in the system. And once the process dies (the page is sent), the environment variable is cleaned up automatically. That provides us the ability to deal with users who might want times to display relative to their local timezones without having to do anything wonky with the system - Times can be stored in GMT and merely converted and displayed in their local timezones.

Erlang, however, cannot really do this because it runs under a VM that is a single OS process, and while it's this amazing VM which allows us to spawn huge number of processes and handle all kinds of neat things, it would be impractical if each new process was an OS thread or (like PHP), an actual OS process. Trying to do this trick of changing the environment variable for Erlang would change it for the whole VM, effectively nullifying the otherwise beautiful process isolation we get with Erlang.

So we need to be a bit more clever to deal with timezones in Erlang.

But let's get to the tools we have available for us for dealing with Dates and Times in Erlang.

Useful Tools for Dealing with Erlang Times

ec_date

ec_date is a part of the Erlware Commons project, and is a fork and of the original dh_date by Dale Harvey. I tend to prefer the ec_date version, mainly because the Erlware Commons project also includes a handful of other utilities. For the sake of brevity, going forward, I will simply refer to ec_date, even though dh_date would probably work just as well.

ec_date provides for Erlang much needed date formatting and parsing similar to PHP's date() and strtotime() functions.

There are two main drawbacks to ec_date:

  • It only parses to and formats from Erlang's now ({MegaSecs, Secs, MicroSecs}) and datetime ({{Year,Month,Day},{Hour,Minute,Second}}) formats.
  • It can't deal with timezones at all

erlang_localtime

erlang_localtime is an application that does deal with timezones, but it doesn't deal with formatting times at all - it just uses the Erlang datetime formatting above, however, it's flexible with regard to what is considered a valid timezone (ie, you could specify "America/Chicago" or "CST").

erlang_locatime will convert one an Erlang datetime value and update the values to be consistent with the provided timezone(s).

And while this is useful, its major drawback is that whenever you're doing dates and times, especially with user-facing interfaces, you'll be parsing dates and times, converting the timezones accordingly, and then probably re-formatting them back. This is far too many steps for something that might very well be frequently performed and should be encapsulated into a single process.

A Better Solution: qdate

So we get to the solution: qdate is a tool that provides a wrapper for both ec_date and erlang_localtime into a single module. It provides backwards API compatibility with ec_date, allowing it to be a drop-in replacement for it, while extending the conversion and formatting enhancements to incorporate timezones, and to also finish off some of ec_date's missing pieces in the formatting department: The Timezone characters in PHP's date function: e, I, O, P, T, Z, c, and r.

What does qdate do?

In short, qdate's functions will take any date format and convert to any date format, while using either an implicit timezone (setting the timezone on a per-process basis), or by setting a specific timezone.

By "any date format", I mean:

  • Erlang now: {MegaSec, Sec, MicroSec}
  • Erlang date: {{Year, Month, Day}, {Hour, Minute, Second}}
  • Parsable Formatted String: "12/31/2013 8:15pm","2012-12-31 16:15", "Dec-15 2013 8:15:23pm"
  • Unix integer timestamp: 13534567335

This can be extended by providing a custom parser for esoteric and non-standard date formats. For example, if you are importing dates from a format that looks like "20131215.161500" (for "Dec 15th, 2013, 8:15pm"), you can register a custom parser with qdate to automatically handle that kind of formatting, eliminating the need to directly call your custom parsing function elsewhere in the code. In short, register it once, use it whenever it's needed - qdate will figure it out for you.

You're also able to register custom formatting strings with the the qdate server. For example, if you commonly have to convert and displays dates in the format above, you could call qdate:register_format(my_format, "Ymd.His"), and then subsequently, you could call qdate:format(my_format, SomeDate) and it would return a string formatted like "20131215.161500".

Handling Timezones

Because ec_date doesn't support timezones for parsing or formatting, qdate will preemptively parse out timezones by checking the end of any string formats for timezone information. For example, if you pass "2012-12-31 16:15 CDT" to qdate, it will be smart enough to extract the timezone and simply pass the date component of the parsing off to ec_date. This works with named timezones like CDT, GMT, HKT, or with GMT-relative timezones like "+0500" or "-1200". In these cases qdate will use the information to accurately display or convert the dates. Further, you can specify target timezones and dates will be converted to those timezones.

Further, instead of tracking timezones manually, you're able to set timezones based on a certain lookup key (The default key is the pid of the current running process, or self()).

Note: Due to the fact that unixtime() and now() are time-zone agnostic, it's recommended that when storing your values in your database, to simply use either of those two formats. That way, there is minimal conversion to and from, and you don't have to store anything else related to timezones. For example, what happens if you store your data in Erlang datetime() format, which doesn't contain any timezone information, and your company moves to another timezone? You'll end up having to update all the records in your database to reflect this change, since the data will no longer be relative to your timezone. Storing your data as a unix timestamp or in now() format, being timezone agnostic, will be far simpler: you won't need to do any conversion in the move, simply changing the timezone with your site will be sufficient.

Conclusion

This brief introduction basically touches on the functionality that qdate can do, but in order to best see how it can help you, check it out on Github, complete with its documentation. For a juicy demo, check out the Demonstration section in the README.