Educational Materials

Data Science with Andreas Lauschke (#9)

Andreas Lauschke

Author

Andreas Lauschke

Title

Data Science with Andreas Lauschke (#9)

Description

Date, Time, and Calendar Functionality

Andreas Lauschke, Oct 4 2019

today’s session: date / time / calendar functionality

What is Time?

Time (on planet earth!!!) is the indefinite mono-dimensional physical quantity that expresses past, present and future progress and events.
Time can be dates (June 17, 1953), instants (List[2019,9,14,11,58,24.559106`8.142787536193213]), and durations (3 minutes, 5 days, 7 weeks, 27 years, ...)

DateObjectQ

Object-oriented analogy: DateObject can be used both as a constructor and a container.

DateObject as constructor:

In[]:=

DateObject[]

really does:

In[]:=

{DateObject@Now,Now}

In[]:=

DateObject/@{"Wed 17 Oct 2018 15:24:06",{2018,10,17,15,24,6.370177`7.556726487214414},3.748778646370185`*^9}

In[]:=

DateObject

Wed 17 Oct 2018 15:24:36GMT-5.

,"Day"

In[]:=

Day: Wed 17 Oct 2018

//FullForm

In[]:=

DateObject[Now,#]&/@{"Minute","Day","Week","Month","Year","Decade","Millennium"}

there are other ways to create time instants, but then they are not necessarily DateObjects!

In[]:=

{AbsoluteTime[],AbsoluteTime[{1900,1,1}],AbsoluteTime[{1901,1,1}],AbsoluteTime[{1899,1,1}],AbsoluteTime[{2001,1,1}]-AbsoluteTime[{2000,1,1}]}

In[]:=

{DateList[],DateString[],DateString[{"DayName"," ","Month","/","YearShort"}],DateString[{"DayName","(","Day",")","---","MonthName","<<<>>>","Year"}],DateString[{"12/26/1980",{"Month","/","Day","/","Year"}}]}

for more info, see helpbrowser on DateString, it is a *very* flexible function!

DateObject as container:

In[]:=

Thu 18 Oct 2018 13:00:01CDT

["Properties"]

we can now use DateObjects for many other things:

In[]:=

DateValue

Thu 18 Oct 2018 13:00:01CDT

,"TimeZone"

In[]:=

DateValue

Thu 18 Oct 2018 13:00:01CDT

,#,Quantity&/@{"Minute","Hour","Day","Week","Month","Year"}

In[]:=

DateValue[Now,#,Quantity]&/@{"Minute","Hour","Day","Week","Month","Year"}

In[]:=

{LeapYearQ[Now],LeapYearQ[{2020,5,5}]}

nerd excursion: the kernel uses the proleptic Gregorian calendar for dates before the Gregorian was officially introduced in 1582. Have a look at
https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar

footnote: For these calendars one can distinguish two systems of numbering years BC. Bede and later historians did not use the Latin zero, nulla, as a year (see Year zero), so the year preceding AD 1 is 1 BC. In this system the year 1 BC is a leap year (likewise in the proleptic Julian calendar). Mathematically, it is more convenient to include a year 0 and represent earlier years as negative, for the specific purpose of facilitating the calculation of the number of years between a negative (BC) year and a positive (AD) year. This is the convention used in astronomical year numbering and in the international standard date system, ISO 8601. In these systems, the year 0 is a leap year.

that means this would be improper usage to find the pre-Gregorian leap years:

In[]:=

{#,LeapYearQ[DateObject@{#,1,1}]}&/@Range[-1000,3000,100]

correct is:

In[]:=

{#,LeapYearQ[DateObject@{#,1,1}]}&/@Range[-200,2000]//Cases[{_,True}]

DateObjectQ predicate

As DateObject is both a constructor and a container, it’s possible for an expression to appear with head DateObject, but not represent a valid date. In such cases the DateObjectQ predicate can be used to test the validity of a given expression:

In[]:=

DateObjectQ[Now]

In[]:=

DateObjectQ[DateObject[foo]]

In[]:=

DateObjectQ["Wed 14 Oct 2018 13:16:49"]

Anatomy of DateObject:

DateObject (currently) contains 4 principal elements that constitute a date in the language:

All 4 of these elements are necessary to fully resolve a date to a given moment in time.

DateObject representation:

The first argument of DateObject is the representational expression for the date associated with the calendar of the date object. Typically this is a list of one or more elements, but in principle may take any number of different forms depending on the calendar associated with the date. The representation is fundamentally tied to the calendar associated with the date.

In[]:=

First[DateObject[]]FullForm@DateObject[]

In[]:=

DateList[]

you see a different format for the date part with a different calendar:

In[]:=

DateObject[0,CalendarType"AbsoluteTime"]//InputForm

Date Representations:

◼

DateList

- a 6-element list specifying year, month, day, hour, minute, and second

In[]:=

DateList[]

◼

AbsoluteTime

- a numeric value giving the number of seconds since midnight, January 1, 1900 in your local time zone

In[]:=

AbsoluteTime[]

◼

DateString

- a string containing different date string elements

In[]:=

DateString[]

◼

JulianDate

- a numeric value giving the number of days since noon GMT on November 24, 4714 BCE

In[]:=

JulianDate[]

◼

UnixTime

- a numeric value giving the number of seconds since midnight GMT on January 1, 1970

In[]:=

UnixTime[]

◼

SiderealTime

* - a mixed-radix quantity giving the right ascension of the local meridian for a given date and location

In[]:=

SiderealTime[]

Granularity:

Calendar granularity represents a calendar object including and beyond a single instant in time. We have a flexible framework for describing dates of varying resolution all within the same object(as opposed to having dedicated objects to represent a given year, or month, or day). Granularity is fundamentally tied to the calendar associated with a date, and can be either a canonical (such as ‘year’) or derived (such as ‘quarter’) granularity.

In[]:=

Now[[2]]Now//FullForm

In[]:=

Today[[2]]Today//FullForm

available granularities

Currently supported granularity specifications(based on the Gregorian calendar) include:

In[]:=

Grid[{#,DateObject[Now,#]}&/@{"Millennium","MillenniumBeginning01","Century","CenturyBeginning01","Decade","Year","Quarter","Month","Week","WeekBeginningSunday","Day","Hour","Minute","Second","Instant"},AlignmentLeft]

additional convenience constructors

convenience constructors to generate dates of a given granularity:

In[]:=

CurrentDate[#]&/@{"Day","Week","Month","Week","Year"}

In[]:=

NextDate[#]&/@{"Day","Week","Month","Week","Year"}

These offset functions also accepts a reference date for the offset to use:

In[]:=

PreviousDate

Day: Wed 3 Apr 2013

,"Week"

In[]:=

NextDate[#]&/@{"Year","Weekend","Quarter","Decade",Friday,"MonthLastDay","BusinessDay","Day","Week","Month","Week","Year"}PreviousDate[#]&/@{"Year","Weekend","Quarter","Decade",Friday,"MonthLastDay","BusinessDay","Day","Week","Month","Week","Year"}

but NextDate and PreviousDate are missing leap seconds:

In[]:=

{PreviousDate[{2017,1,1,0,0,0},"Second"],NextDate[{2016,12,31,23,59,59},"Second"]}

we can construct in various other forms:

In[]:=

Yesterday

In[]:=

Tomorrow

In[]:=

LocalTime[]

conversions

In[]:=

DateList["23 Nov, 1992"]

In[]:=

DateString[{1992,11,23,0,0,0.`}]

In[]:=

AbsoluteTime[DateList[]]

In[]:=

AbsoluteTime[DateString[]]

comparisons

DateObject expressions support ordering using comparisons, accounting for granularity:

In[]:=

Today<Tomorrow

In[]:=

NowToday

we saw that earlier: Now is an instant, Today is a day.

containment

DateWithinQ and DateOverlapsQ can be used to check for full or partial containment of one granular date within another:

In[]:=

DateWithinQ[Today,Now]

In[]:=

DateWithinQ[CurrentDate["Week"],NextDate["Sunday"]]

In[]:=

DateWithinQ[CurrentDate["Week"],NextDate["Monday"]]

In[]:=

DateWithinQ

Month: Oct 2018

Week beginning: Mon 29 Oct 2018



In[]:=

DateOverlapsQ

Month: Oct 2018

Week beginning: Mon 29 Oct 2018



In[]:=

DateOverlapsQ[Today,Tomorrow]

Calendar Type

In[]:=

Grid[With[{d=DateObject[CalendarType#]},{#,DateString[d],First[d]}]&/@{"Gregorian","Jewish","Islamic","ISOWeek","ISOOrdinal","Julian","JulianDate","AbsoluteTime","UnixTime"},AlignmentLeft,DividersAll]

The calendar associated with a date provides a scheme by which to interpret the date representation provided in the first argument. A calendar must principally have a standard notation, normalization scheme, and epoch conversion function (a method to convert to/from a common reference date, commonly either the JulianDate epoch, or January 1st 1 CE).
At this time we have six primary calendar types: Gregorian(Default), Jewish(Hebrew), Islamic(Um Al’Qura), ISOWeek(ISO-8601 week day), ISOOrdinal(ISO-8601 year day), and Julian, with more coming in future versions.

Time Zone

Time zones represent a region on earth that observe a uniform synchronized time. We have four different input types for the TimeZone option: Numeric offset, Time Zone ID/Entity, location Entity, and None.

Numeric offset from GMT:

A real or integer, or rational value that’s a number of hours offset from GMT. There are currently 38 official numeric time zone offsets observed across the globe, though arbitrary offsets are supported as well:

In[]:=

DateObject[Now,TimeZone-5]

In[]:=

DateObject[Now,TimeZone6.78]

The kernel initializes its local time zone using information from the operating system, and your $TimeZone is used as the default time zone value when not otherwise specified:

In[]:=

$TimeZone

Time zone ID / Location Entity

Sometimes also called “internet time zones” these are the time zone identifiers associated with a geographic or legislative region that determines the physical and temporal boundaries associated with a time zone’s offset from GMT. There are currently 471 recognized time zone identifiers corresponding to different geographic regions.

In[]:=

DateObject[Now,TimeZone"America/Chicago"]

In[]:=

$TimeZoneEntity

In[]:=

EntityList["TimeZone"]//Short

In[]:=

Length[EntityList["TimeZone"]]

Any entity with a “Position” property may be used as a TimeZone specification, where the underlying time zone ID can be fetched from the Wolfram Knowledgebase.

In[]:=

DateObject[Now,TimeZoneEntity["City",List["Bismarck","NorthDakota","UnitedStates"]]]

In[]:=

LocalTimeZone[Entity["City",List["Bismarck","NorthDakota","UnitedStates"]]]

None

TimeZone -> None does not immediately instantiate a TimeZone value in the DateObject, but instead defers determination of the time zone up to the point where it’s required to resolve the date.

In[]:=

DateObject[Now,TimeZoneNone]

Automatic

TimeZone  Automatic is also supported, which indicates that the time zone should automatically be inherited from its argument, and otherwise default to $TimeZone.

In[]:=

DateObject[Now,TimeZoneAutomatic]

date-based computations: DatePlus, DayCount, DayRange, and DateDifference:

DatePlus: raw integers mean days

In[]:=

DatePlus

Day: Sun 1 Jan 2017

,35

In[]:=

DatePlus["Jan. 1, 2017",35]

In[]:=

DatePlus

Day: Sun 1 Jan 2017

,-35

In[]:=

DatePlus[Today,34]

you need quantities for others (i. e., not days):

In[]:=

DatePlus

Day: Sun 1 Jan 2017

,Quantity[14,"Weeks"]

In[]:=

DatePlus

Day: Sun 1 Jan 2017



In[]:=

DatePlus

Day: Sun 1 Jan 2017



In[]:=

DatePlus

Day: Sun 1 Jan 2017

,{{7,"Week"},{2,"Day"}}

In[]:=

DatePlus

Day: Sun 1 Jan 2017

,{342,"Hour"}

In[]:=

DatePlus

Sat 3 Feb 1990 01:12:00GMT-6.`

,{{1,"Month"},{15,"Day"}}

In[]:=

DatePlus

Sun 1 Jan 2017 06:17:44GMT-5.`

,2.45

In[]:=

DatePlus

Sun 1 Jan 2017 06:17:44GMT-5.`

,{-4.6,"Hour"}

In[]:=

future=DatePlus[Today,{19,"BusinessDay"}]

for the opposite direction: DayCount!

In[]:=

DayCount[Today,future,"BusinessDay"]

In[]:=

DayCount[Today,{2020,1,1},"Holiday"]

In[]:=

DayCount[Today,{2020,1,1},"EndOfMonth"]

In[]:=

DayCount[Today,{2020,1,1},Friday]

number of holidays of the current year, subject to default holiday schedule:

In[]:=

With[{first=Now},DayCount[first,DatePlus[first,{{1,"Year"},{-1,"Day"}}],"Holiday"]]

average number of non-weekend holidays for this century for various countries:

In[]:=

(*thiswilltakesometime,bepatient*){#,N@Mean[Table[DayCount[{i},{i+1},"Holiday",HolidayCalendar#],{i,2001,2100}]]}&/@{"Germany","Sweden","Russia","Japan","Switzerland","China","UnitedStates"}//SortBy@Last//TableForm

holidays, weekends, and business days add up to the total number of days in the range, therefore holidays are not considered weekend days.

In[]:=

DayCount[{2012},{2013},"Holiday"]+DayCount[{2012},{2013},"Weekend"]+DayCount[{2012},{2013},"BusinessDay"]-DayCount[{2012},{2013}]

related to DayCount: DayRange:

all Sundays of the current month in the default calendar:

In[]:=

With[{first=Take[DateList[],2]},DayRange[first,DatePlus[first,{{1,"Month"},{-1,"Day"}}],Sunday]]

all third Friday’s of 2020 (motivation for third Fridays: most equity derivatives exchanges have their main expirations on the third Friday of the month):

In[]:=

thirdfridays=DayRange[DateObject[{2020,1,1}],DateObject[{2020,12,31}],Friday]//GatherBy[#,#[[1,2]]&]&//#[[All,3]]&

sidenotes:a) the above will hopefully refresh what you saw in sessions 1 and 2 about post-fix notation, in particular when used together with pure functions, and the flow is “left to right”, which feels natural for the task: give me all Fridays in a particular day range, gather them by the month, then show me the third column.b) could also have used SplitBy instead of GatherBy in this particular case. Also GroupBy would have been interesting. Homework: explain why SplitBy, GatherBy, and GroupBy all produce the proper split / gathering / grouping here that we want, and how they differ, but produce the same result in this particular case.

comment: the third Friday expirations of the quarter-ending months are called the “triple-witching” days. That’s because the expirations of the stock options, stock index futures, and stock index options all occur on the very same day. Triple-witching days of next year:

In[]:=

Last/@Partition[thirdfridays,3]

all business days until year-end:

In[]:=

DayRange[Today,DateObject[{2019,12,31}],"BusinessDay"]Length@%

all holidays next year (US, that means)

In[]:=

DayRange[DateObject[{2020,1,1}],DateObject[{2020,12,31}],"Holiday"]

and in Russia:

In[]:=

DayRange[DateObject[{2020,1,1}],DateObject[{2020,12,31}],"Holiday",HolidayCalendar"Russia"]

footnote: HolidayCalendar has bugs, for example the holiday calendar for Germany is missing Oct 3 (German Unity Day). I’m told (didn’t check! just hearsay!) that some Chinese holidays are wrong too. So, proceed with caution.

In[]:=

DayRange[DateObject[{2020,1,1}],DateObject[{2020,12,31}],"Holiday",HolidayCalendar"Germany"]

Out[]=



Day: Wed 1 Jan 2020

Day: Fri 10 Apr 2020

Day: Mon 13 Apr 2020

Day: Fri 1 May 2020

Day: Thu 24 Dec 2020

Day: Fri 25 Dec 2020

Day: Thu 31 Dec 2020



missing: Oct 3, a statutory federal holiday since 1990.

extremely important: DateDifference!

In[]:=

DateDifference

Day: Wed 1 Jan 2014

Day: Fri 4 Jul 2014

DateDifference[{2006,8,29},{2007,1,1}]

In[]:=

DateDifference

Day: Sat 1 Jan 2000

Day: Sun 4 Jul 2004

,"Week"

In[]:=

DateDifference

Day: Wed 4 Jul 2012

Day: Sun 1 Jan 2012



In[]:=

DateDifference

Day: Sun 1 Jan 2006

Day: Fri 1 Jun 2007

,"Hour"

In[]:=

DateDifference

Day: Fri 14 Aug 1936

Day: Fri 1 Dec 2000

,"Year"

In[]:=

DateDifference

Day: Thu 14 Aug 2003

Day: Wed 1 Dec 2004

,{"Month","Day"}

In[]:=

DateDifference

Year: 2000

,DateObject[],{"Month","Day","Year"}

inverse relationship between DatePlus and DateDifference:

In[]:=

dp=DatePlus

Month: Jan 2007

,{2,"Month"}

In[]:=

DateDifference

Month: Jan 2007

,dp,"Month"

great way to show timelines: TimelinePlot:

In[]:=

TimelinePlot[{DateObject[{2000,6,1}],DateObject[{2000,8,4}],DateObject[{2000,2,4}],Interval[{DateObject[{2000,9}],DateObject[{2000,10}]}],Interval[{DateObject[{2000,3}],DateObject[{2000,4}]}],Interval[{DateObject[{2000}],DateObject[{2001}]}]}]

please look at TimelinePlot in the help browser, it contains a *plethora* of options and great examples.

Date Visualization: DateListPlot is the “workhorse”. It can take: AbsoluteTime, DateString, DateList, DateObject,TimeSeries

In[]:=

GraphicsRow[{DateListPlot[{{3368649600,10},{3369859200,20},{3371155200,5},{3372969600,10},{3374154000,Pi}}],DateListPlot[{{"June 2006",10},{"August 2006",20},{"November 2006",5},{"January 2007",10}},ScalingFunctions"Reverse"]}]GraphicsRowDateListPlot[{{{2006,6},10},{{2006,8},2000},{{2006,11},5},{{2007,1},100}},ScalingFunctions"Log"],DateListPlot

Jun 2006

,10,

Aug 2006

,20,

Nov 2006

,5,

Jan 2007

,10,

Month: Mar 2007

,Sum[1/n^2,{n,1,Infinity}],ScalingFunctions"Reverse"

how about some labels / legends?

In[]:=

data1=TimeSeries[{1,1,2,3,5,8,11},{"Jan 1, 2015"}];data2=TimeSeries[{5,8,9,6,2,4,7},{"Jan 1, 2015"}];

In[]:=

DateListPlot[{data1,data2},PlotLabels{"first","second"}]DateListPlot[{data1,data2},PlotLegends{"first","second"}]

please look at the helpbrowser for DateListPlot, the possibilities for options and styling are endless!

Time Series

In[]:=

v={10,20,5,10};

In[]:=

t=

Jun 2006

Aug 2006

Nov 2006

Jan 2007

;

In[]:=

ts=TimeSeries[v,{t}]

In[]:=

DateListPlot[ts]

Event Series

In[]:=

v={10,20,E^Pi,Pi^E,Sum[1/n^2,{n,1,Infinity}],5,10};

In[]:=

t=

Jun 2006

Month: Jul 2006

Aug 2006

Month: Sep 2006

Month: Oct 2006

Nov 2006

Jan 2007

;

In[]:=

ts=EventSeries[v,{t}]

In[]:=

DateListPlot[ts,JoinedFalse,FillingAxis]

financial data is returned as a time series:

In[]:=

data=FinancialData["AMZN","Close",{{2015,1,1},Today}]

sidenote: FinancialData contains units for different types of retrieved data:

In[]:=

FinancialData["AMZN","OHLCV",{{2015,1,1},Today}]["Path"]//Short

we can then visualise with various plot types, here for example financial market analysis charts:

In[]:=

GraphicsRow[{RenkoChart@data,PointFigureChart@data}]GraphicsRow[{KagiChart@data,LineBreakChart@data}]

In[]:=

data=FinancialData["AMZN","Volume",{{2015,1,1},Today}]

In[]:=

DateListPlot[data,PlotRangeAll]

In[]:=

nta=TimeSeriesAggregate[data,"Quarter",Mean]

In[]:=

DateListPlot[nta,PlotLabelDateListPlot,Filling->Bottom]

In[]:=

DateListStepPlot[nta,PlotLabel->DateListStepPlot,Filling->Bottom]

we can also extract windows from time series:

In[]:=

data=FinancialData["AMZN","Close",{{2015,1,1},Today}]DateListPlot@%

take the elements for 2017 and 2018:

In[]:=

DateListPlot@TimeSeriesWindow[data,{{2017,1,1},{2018,12,31}}]

extract only Mar 2019:

In[]:=

DateListPlot@TimeSeriesWindow[data,DateObject[{2019,3},"Month"]]

extract only end-of-month figures:

In[]:=

TimeSeriesWindow[data,"EndOfMonth"]//DateListPlot[#,PlotMarkersAutomatic]&

split time series into parts and display:

In[]:=

win1=TimeSeriesWindow[data,{{2015,1,1},{2016,12,31}}];win2=TimeSeriesWindow[data,{{2017,1,1},{2018,12,31}}];win3=TimeSeriesWindow[data,{{2019,1,1},Automatic}];

In[]:=

DateListPlot[{win1,win2,win3},PlotLegends{"win1","win2","win3"}]

time series objects are computable. Let’s center a time series:

In[]:=

centered=data-Mean[data]DateListPlot[{data,centered}]

or double it:

In[]:=

doubled=2dataDateListPlot[{data,doubled}]

Brownian motion: RandomFunction returns a TemporalData function!

In[]:=

td=RandomFunction[WienerProcess[0,0.01],{0,200,1},10]

In[]:=

DateListPlot[td]

In[]:=

BoxWhiskerChart[td,JoinedTrue]

In[]:=

DistributionChart[td]

you can also convert to an Association, that way you automatically get a legend (you may remember from second session):

In[]:=

DateListPlot[Association@MapIndexed["path "<>ToString@First@#2#&,td[[2,1]]],{0,200}]

In[]:=

Association@MapIndexed["path "<>ToString@First@#2#&,td[[2,1]]]

we can also use TimeSeriesWindow on temporal data (doesn’t have to be a TimeSeries object):

In[]:=

Head@tdwin=TimeSeriesWindow[td,{30,170}]

In[]:=

DateListPlot[#,PlotRange{{0,200},{-0.5,0.5}}]&/@{td,win}

with TimeSeriesThread we can combine multiple time series using a function (here Mean, Min, Max, MinMax, StandardDeviation):

In[]:=

TimeSeriesThread[Mean,td]//DateListPlotTimeSeriesThread[Min,td]//DateListPlotTimeSeriesThread[Max,td]//DateListPlotTimeSeriesThread[MinMax,td]//DateListPlotTimeSeriesThread[StandardDeviation,td]//DateListPlot

Other extremely helpful functions: DateValue, CalendarConvert, DateHistogram, TimeSeriesResample. Leaving out due to lack of time.

Acknowledgements

the DateObject[] part of this session included material from the presentation of Nick Lariviere held at the 2018 conference, thankfully acknowledged.

Cite this as: Andreas Lauschke, "Data Science with Andreas Lauschke (#9)" from the Notebook Archive (2020), https://notebookarchive.org/2020-09-4lmobnt