Parquet TIME Logical Type

What is TIME

TIME logical type is used to represent time without date part. It has two parameters:

1. UTC adjustment (boolean type).
2. Unit (or precision). This can be one of three values but also dictates physical storage type:
   1. MILLIS - millisecond precision, with int32 physical type. The number is milliseconds since midnight.
   2. MICROS - microsecond precision, with int64 type. The number is microseconds since midnight.
   3. NANOS - nanosecond precision with int64 type (same as in MICROS). The number is nanoseconds since midnight.

The natural way to represent time duration .NET is using TimeSpan struct, however it has maximum precision of milliseconds, which only covers the first easiest unit. Also, TimeSpan basically just measure time duration, which can be longer than 1 day, unlike what Parquet TIME is designed to be.

1 day has 86'400'000 milliseconds, which is under int32 max value (2'147'483'647).

Or 86'400'000 microseconds, which overflows int32, but fits into int64 range (max int64 is 9'223'372'036'854'775'807).

Or 86'400'000'000'000 nanoseconds, which also fits into int64 range.

Therefore, although millisecond precision can be expressed via TimeSpan, we want some consistency rather than edge cases. For this reason Parquet.Net was originally going to treat TIME logical type simply as long. However, that would be too redundant for millisecond precision, as this would double the amount of memory. Now we have edge case anyways - int for millisecond precision and long for everything else. Logic dictates that it would make sense to use TimeSpan for millisecond precision data, because it feels more native for .NET. However, TimeSpan occupies 8 bytes of memory, which brings us back to doubling the amount of RAM again. Therefore, it would make sense to just use int.

How it’s represented in Parquet.Net

In V6, TIME is broken due to a bug in the library itself, and considering this is also a new major release, it was a good time to fix it. V6 goes simple and correct, and treats TIME as a primitive data type - int or long. You are free to then process the value as you like - whether use TimeSpan, TimeOnly or any other type that is suitable for your use cases. Apache Spark actually follows similar approach - TIME is deserialized as a number, and there is not even a way to write logical time using Spark API at all. If you want to create a test file with all logical types stamped, you need to resort to PyArrow:

 1import pyarrow as pa
 2import pyarrow.parquet as pq
 3
 4# Values representing 10:30:00 in each unit
 5ms = [37800000]  # milliseconds since midnight
 6us = [37800000000]  # microseconds since midnight
 7ns = [37800000000000]  # nanoseconds since midnight
 8
 9table = pa.table(
10    {
11        "time_ms": pa.array(ms, type=pa.time32("ms")),  # INT32, MILLIS
12        "time_us": pa.array(us, type=pa.time64("us")),  # INT64, MICROS
13        "time_ns": pa.array(ns, type=pa.time64("ns")),  # INT64, NANOS
14    }
15)
16
17pq.write_table(table, "output.parquet")

There’s a new DataField derivative now - TimeDataField that allows specifying Unit and IsAdjustedToUtc parameters as well. To create schema field for TIME:

1var field = new TimeDataField("time_in_microseconds", TimeDataField.Unit.Micros);

You can specify nanoseconds or milliseconds as well, and IsAdjustedToUtc flag, which makes no impact on values but only acts as metadata. Also, when reading schema, DataField instances are created as TimeDataField so you can downcast to read appropriate field metadata as well.

How class serializer handles it

Class serializer is a convenience high-level API, and it supports time type for members that are int and long as usual.