5. Advanced annotation¶
5.1. Creating definitions¶
HED version 8.0.0 introduced the Definition tag to facilitate tag reuse and
to allow implementation of concepts such as temporal scope.
The Definition tag allows researchers to create a name to represent a group of tags and
then use the name in place of these tags when annotating data.
These short-cuts make tagging easier and reduce the chance of errors.
Often laboratories have a standard setup and event codes with particular meanings.
Researchers can define names and reuse them for multiple experiments.
Another important role of definitions is to provide the structure for implementing temporal scope as introduced in Chapter 5.3: Temporal Scope.
A HED definition is a tag group that includes one Definition tag whose required
child value is the definition’s name.
The definition tag group also includes an internal tag-group
specifying the definition’s content.
The following summarizes the syntax of HED definitions.
Syntax summary for HED definitions.
- Short forms:
(Definition/xxx)
(Definition/xxx, (definition-content))
(Definition/xxx/#, (definition-content))
- Long forms:
(Property/Organizational-property/Definition/xxx)
(Property/Organizational-property/Definition/xxx, (definition-content))
(Property/Organizational-property/Definition/xxx/#, (definition-content))
Notes:
xxx is the name of the definition, and (definition-content) is a tag group containing the tags representing the definition’s contents.
If the xxx/# form is used, then the (definition-content) MUST contain a single
#representing a value to be substituted for when the definition is used.A definition without a (definition-content) is sometimes use as a general purpose definition for anchoring
Onsetgroups with varying content.
The following example defines the PlayMovie term.
The next example gives a definition that uses a placeholder representing a presentation
rate, for example, to annotate events in which a presentation rate is varied
at random. Usually the specific value substituted for the # will come from
one of the columns in the events.tsv file.
Example: Use definition with placeholder to annotate a variable presentation rate.
Short form:
(Definition/PresentationRate/#,
(Visual-presentation, Experimental-stimulus, Temporal-rate/# Hz))
Long form:
(Property/Organizational-property/Definition/PresentationRate/#,
(Property/Sensory-property/Sensory-presentation/Visual-presentation,
Property/Task-property/Task-event-role/Experimental-stimulus,
Data-property/Data-value/Spatiotemporal-value/Rate-of-change/Temporal-rate/# Hz))
Definitions may only appear in dummy entries of JSON sidecars and as external dictionaries. Definitions cannot be nested. Further, definitions must appear as top-level tag groups.
The validation checks made by the HED validator when assembling and processing definitions are summarized in Appendix B: HED errors. In addition to syntax checks, which occur in early processing passes, HED validators check that the definition names have unique definitions. Additional checks for temporal scope are discussed in Chapter 5.2: Using definitions and Chapter 5.3: Temporal scope.
5.2. Using definitions¶
This section describes how to use definitions to assist in annotation.
5.2.1. The Def tag¶
When a definition name such as PlayMovie or PresentationRate is used in an annotation,
the name is prefixed by the Def tag to indicate that the name represents a defined name.
In other words, Def/PlayMovie is shorthand for
(Visual-presentation, Movie, Computer-screen).
The following summarizes Def tag syntax rules.
Syntax summary for the Def tag:
- Short forms:
Def/xxx
Def/xxx/yyy
- Long forms:
Property/Organizational-property/Def/xxx
Property/Organizational-property/Def/xxx/yyy
Notes:
xxx is the name of the definition.
yyy is the value that is substituted for the definition’s placeholder if it has one.
If the xxx/yyy form is used, then the corresponding definition’s tag-group MUST contain a single
#representing a value to be substituted for when the definition is used.
The following example shows how Def is used in annotation.
Example: Use PresentationRate to annotate a presentation rate of 1.5 Hz.
- Short form:
Def/PresentationRate/1.5 Hz
- Long form:
Property/Organizational-property/Def/PresentationRate/1.5 Hz
5.2.2. The Def-expand tag¶
The Def-expand tag provides an alternative to Def tag in annotations.
Unlike the Def tag, a Def-expand tag must be in a tag group that includes
an inner tag group with the definition’s contents.
If the definition includes a placeholder, that must be replaced with these
contents by the appropriate value.
The following summarizes Def-expand tag syntax rules.
The following example shows how Def-expand is used in an annotation.
Example: Use PresentationRate to annotate a presentation rate of 1.5 Hz.
Short form:
(Def-expand/PresentationRate/1.5 Hz,
(Visual-presentation, Experimental-stimulus, Temporal-rate/1.5 Hz))
Long form:
(Property/Organizational-property/Def-expand/PresentationRate/1.5 Hz,
(Property/Sensory-property/Sensory-presentation/Visual-presentation,
Property/Task-property/Task-event-role/Experimental-stimulus,
Data-property/Data-value/Spatiotemporal-value/Rate-of-change/Temporal-rate/1.5 Hz))
During analysis, tools may replace Def/PlayMovie with a fully expanded tag string.
Tools sometimes need to retain the association of the expanded tag string with the definition
name for identification during searching and substitution.
5.3. Temporal scope¶
Events are often modeled as instantaneous occurrences that occur at single points in time (i.e., time-marked or point events). In reality, many events unfold over extended time periods. The interval between the initiation of an event and its completion is called the temporal scope of the event. HED events are assumed to be point events unless they are given an explicit temporal scope (i.e., they are “scoped” events).
Some events, such as the setup and initiation of the environmental controls
for an experiment, may have a temporal scope that spans the entire data recording.
Other events, such as the playing of a movie clip or a participant performing an action in
response to a sensory presentation, may last for seconds or minutes.
Temporal scope captures the effects of these extended events in a machine-actionable manner.
HED has two distinct mechanisms for expressing temporal scope: Onset/Offset and Duration/Delay.
Tools can transform between one representation and the other.
However, transform from the Duration/Delay representation to the Onset/Offset
representation may require the addition of additional rows (time markers) in the events file.
The mechanisms are summarized in the following table and discussed in more detail in the following sections.
Tag |
Meaning |
Usage |
|---|---|---|
|
Marks start of event |
Used with a |
|
Marks end of event |
Used with a |
|
Marks event intermediate pt |
New in standard schema 8.2.0. |
|
Marks end of an event. |
Doesn’t use a definition anchor. |
|
Marks delayed onset. |
Doesn’t use a definition anchor. |
|
Context of ongoing events. |
Should only be inserted by tools. |
All of these tags must appear in a topLevelTagGroup, which implies that they can’t be nested.
Delay and Duration will not be fully supported until HED standard schema version 8.2.0.
The Inset tag will also not be included until HED standard schema version 8.2.0,
but is listed here for completeness.
5.3.1. Using Onset and Offset¶
The most direct HED method of specifying scoped events combines
Onset and Offset tags with defined names.
Using this method, an event with temporal scope actually corresponds to two point events.
The initiation event is tagged by a (Def/xxx, Onset) where xxx is a defined name.
The end of the event’s temporal scope is marked either by a (Def/xxx, Offset) or by
another (Def/xxx, Onset). The Def/xxx is said to anchor the Onset
(and similarly for Offset).
By anchor, we mean that tools use the anchor to determine
where each event of temporal extent begins and ends.
A Def-expand tag group can also anchor the Onset and Offset groups.
The Onset tag group may contain an additional internal tag group in addition to the
anchor Def tag. This internal tag group usually contains annotations specific
to this instance of the event. As with all HED tags and groups, order does not matter.
Event initiations identified by definitions with placeholders are handled similarly.
Suppose the initiation event is tagged by a (Def/xxx/yyy, Onset) where xxx
is a defined name and yyy is the value substituted for the # placeholder.
The end of this event’s temporal scope is marked either by (Def/xxx/yyy, Offset) or by
another (Def/xxx/yyy, Onset).
An intervening (Def/xxx/zzz, Onset), where yyy and zzz
are different, is treated as a completely distinct temporal event.
The following table summarizes Onset and Offset usage.
Note: A Def-expand/xxx tag group can be used
interchangeably with the Def/xxx.
Syntax summary for Onset and Offset.
- Short forms:
(Def/xxx, Onset, (tag-group))
(Def/xxx/yyy, Onset, (tag-group))
(Def/xxx, Offset)
(Def/xxx/yyy, Offset)
- Long forms:
(Property/Organizational-property/Def/xxx,
Property/Data-property/Data-marker/Temporal-marker/Onset, (tag-group))(Property/Organizational-property/Def/xxx/#,
Property/Data-property/Data-marker/Temporal-marker/Onset, (tag-group))(Property/Organizational-property/Def/xxx, Property/Data-property/Data-marker/Temporal-marker/Offset)
(Property/Organizational-property/Def/xxx/#, Property/Data-property/Data-marker/Temporal-marker/Offset)
Notes:
xxx is the name of the definition anchoring the scoped event.
yyy is the value substituted for a definition’s placeholder if it has one.
The (tag-group), which is optional, contains tags specific to that temporal event. This tag group is not the tag group specifying the contents of the definition.
The additional tag-group is only in effect for that particular scoped event and not for all events anchored by Def/xxx.
If the Def/xxx/# form is used, the
#must be replaced by an actual value.The entire definition identifier Def/xxx/#, including the value substituted for the
#, is used as the anchor for temporal scope.
For example, the PlayMovie definition of the previous section just defines the playing of a
movie clip on the screen.
The (tag-group) might include tags identifying which clip is playing in this instance.
This syntax allows one definition name to be used to represent the
playing of different clips.
The PlayMovie scoped event type can be reused to annotate the playing of other movie clips.
However, scoped events with the same defined name (e.g., PlayMovie) cannot be nested.
The temporal scope of a PlayMovie event ends with a PlayMovie offset or with the
onset of another PlayMovie event.
In the previous example, the Def/PlayMovie “anchors” the temporal scope,
and the appearance of another Def/PlayMovie indicates the previous movie has ceased.
The Label tag identifies the particular movie but does not affect the Onset/Offset
determination.
If you want to have interleaved movies playing, use definitions with
placeholder values as shown in the next example. The example assumes a definition
Definition/MyPlayMovie/# exists.
Because tools need to have the definitions in hand when fully expanding during validation and analysis, tools must gather applicable definitions before final processing. Library functions in Python, Matlab, and JavaScript are available to support gathering of definitions and the expansion. These definitions may be given in JSON sidecars or provided externally.
5.3.2. Using Inset¶
The Inset tag group marks an intermediate point in an event of temporal extent
defined by Onset and Offset.
Like the Offset, the Inset tag is anchored by a Def tag or Def-expand tag group
that is the anchor of its enclosing Onset.
The Inset tag group may contain an additional internal tag group in addition to the
anchor Def tag. This internal tag group usually contains annotations specific
to this instance of the event. As with all HED tags and groups, order does not matter.
The following table summarizes Inset usage.
Note: A Def-expand/xxx tag group can be used
interchangeably with the Def/xxx.
Syntax summary for Inset.
- Short forms:
(Def/xxx, Inset, (tag-group))
(Def/xxx/yyy, Inset, (tag-group))
- Long forms:
(Property/Organizational-property/Def/xxx,
Property/Data-property/Data-marker/Temporal-marker/Inset, (tag-group))(Property/Organizational-property/Def/xxx/#,
Property/Data-property/Data-marker/Temporal-marker/Inset, (tag-group))
Notes:
xxx is the name of the definition anchoring the scoped event.
yyy is the value substituted for a definition’s placeholder if it has one.
The (tag-group), which is optional, contains information specific to that intermediate. point in the ongoing event. This tag group is not the tag group specifying the contents of the definition..
The additional tag-group is only in effect at that particular point.
If the Def/xxx/# form is used, the
#must be replaced by an actual value that is the same as the value used for itsOnset.
5.3.3. Using Duration¶
The Duration tag is an alternative method for specifying an event with temporal scope.
The start of the temporal scope is the event in which the Duration tag appears.
The end of the temporal scope is implicit and may not coincide with an actual event
appearing in the recording.
Instead, tools calculate when the scope ends (i.e., the event offset) by
adding the value of the duration to the onset of the event marker associated
with that Duration tag. As with all HED tags and groups, order does not matter.
The following table summaries the syntax for Duration.
Syntax summary for Duration.
- Short forms:
(Duration/xxx, (tag-group))
(Duration/xxx, Delay/yyy, (tag-group))
- Long forms:
(Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Duration/xxx,
(tag-group)(Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Duration/xxx, (Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Delay/yyy,
(tag-group))
Notes:
xxx is a time value for the duration.
yyy is a time value for the delay if given.
The (tag-group) contains the additional tags specific to the temporal event whose duration is specified.
Duration tags do not use a definition anchor.
Duration should be grouped with tags representing additional information associated
with the temporal scope of that event.
The Duration tag must appear in a top level tag
group that may include an additional Delay tag.
If the Duration appears with Delay, the end of the temporal event is the onset of the current event plus the delay value plus the duration value.
Several events with temporal-scopes defined by Duration tag groups
may appear in the annotations associated with the same event marker.
The Duration tag has the same effect on event context as the
Onset/Offset mechanism explained in
5.5. Event contexts
The Duration tag is convenient because its use does not require a definition.
However, the ending time point of events whose temporal scope is defined
with Duration is not marked by an explicit event in the data recording.
This has distinct disadvantages for analysis if the offset is expected to elicit a
neural response, which is the case for many events involving visual or auditory presentations.
The use of the Duration tag will not be fully supported by validators until HED
standard schema version 8.2.0.
5.3.4. Using Delay¶
The Delay tag is grouped with an inner tag group to indicate that the associated tag-group is
actually an implicit event that occurs at a time offset from the current event.
Delay tags do not use a definition anchor.
If the tag group containing the Delay also contains a Duration tag,
then the tag group represents an event with temporal extent.
Otherwise, it is considered a point event.
As with all HED tags and groups, order does not matter.
The following table summarizes the syntax for Delay.
Syntax summary for Delay.
- Short forms:
(Delay/xxx, (tag-group))
(Delay/xxx, Duration/yyy, (tag-group))
- Long forms:
(Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Delay/xxx,
(tag-group)(Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Duration/xxx, (Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Delay/yyy,
(tag-group))
Notes:
xxx is a time value for the duration.
yyy is a time value for the delay if given.
The (tag-group) contains the additional tags specific to the temporal event whose duration is specified.
A typical use case for Delay is when a secondary stimulus appears offset from
the first. A typical use case for Delay combined with Duration is the encoding
of a participant response, where the reaction time is measured relative to
a secondary stimulus (such as a ‘go’).
In the following example, a trial consists of the presentation of a cross in the center of the screen. The participant responds with a button press upon seeing the cross. The response time of the button push is recorded relative to the stimulus presentation as part of the stimulus event.
Example: Use the delay mechanism for a participant response.
Short form:
(Sensory-event, (Experimental-stimulus, Visual-presentation, Cross))
(Delay/2.83 ms, (Agent-action, Participant-response, (Press, Mouse-button)))
Long form:
(Event/Sensory-event,
Property/Task-property/Task-event-role/Experimental-stimulus,
Property/Sensory-property/Sensory-presentation/Visual-presentation,
(Item/Object/Geometric-object/2D-shape/Cross)),
(Property/Data-property/Data-value/Spatiotemporal-value/Temporal-value/Delay/2.83 ms, (Event/Agent-action,
(Property/Task-property/Task-event-role/Participant-response,
(Action/Move/Move-body-part/Move-upper-extremity/Press/,
Item/Object/Man-made-object/Device/IO-device/Input-device/Computer-mouse/Mouse-button))),
Notice that the Agent-action tag from the Event subtree is
included in the Delay tag-group.
This allows tools to identify this tag group as a distinct event.
For BIDS datasets, such response delays would be recorded in a column of the events.tsv
event files. The HED annotation for the JSON sidecar corresponding to these files would
contain a #. At HED expansion time, tools replace the # with the column value (2.83)
corresponding to each event.
The Delay tag can also be used in the same top level tag group as the Duration tag to
define an event with temporal extent.
HED tools are being developed to support the expansion of delayed events to have their
own event markers without the delay tag.
However, use of the Delay tag will not be fully supported by validators until HED
standard schema version 8.2.0.
5.4. Event streams¶
An event stream is a sequence of events in a data recording. The most obvious event stream is the sequence consisting of all the events in the recording, but there are many other possible streams such as the stream consisting o f all sensory events or the stream consisting of all participant response events.
Event streams can be identified and tagged using the Event-stream tag, allowing annotators
to more easily identify subsets of events and interrelationships of events within those event
sequences.
An event having the tag Event-stream/xxx indicates that event or marker is part of event stream xxx.
Example: Tag a face event as part of the Face-stream event stream.
Short form:
Sensory-event, Event-stream/Face-stream, Visual-presentation, (Image, Face)
Long form:
Event/Sensory-event,
Property/Organizational-property/Event-stream/Face-stream,
Property/Sensory-property/Sensory-presentation/Visual-presentation,
(Item/Object/Man-made-object/Media/Visualization/Image,
Item/Biological-item/Anatomical-item/Body-part/Head/Face)
Using a tag to identify an event stream makes it easier for downstream tools to compute relationships among subsets of events.
Note: Event streams are still under development.
5.5. Event contexts¶
Event annotations generally focus on describing what happened at the instant an event was
initiated. However, the details of the setting in which the event occurs also influence neural
responses. For the PlayMovie example of the previous section,
events that occur between the Onset and Offset pairs for PlayMovie should
inherit the information that a particular movie is playing without requiring
the user to explicitly enter those tags for every intervening event.
The process of event context mapping should be deferred until analysis time because other
events might be added to the event file after the initial annotation of the recording.
For example, a user might run a tool to mark blink or other features as events prior
to doing other analyses.
HED uses the Event-context tag to accomplish the required context mapping.
In normal usage, the Event-context tag is not used directly by annotators.
Rather, tools insert the Event-context tag at analysis time to
handle the implicit context created by enduring or scoped events.
However, annotators may use the tag when an event has explicit context information
that must be accounted for.
Tools are available to insert the appropriate Event-context at analysis time.
The Event-context has the unique attribute,
implying that only one Event-context tag group may appear in the assembled
HED annotation corresponding to each time-marker value.
Syntax summary for Event-context.
- Short form:
(Event-context, other-tag-groups)
- Long form:
(Property/Organizational-property/Event-context, other-tag-groups)
Notes:
The
Event-contextmay only appear in a top-level tag group of an assembled HED string.An event can have at most one
Event-contexttag group in its assembled HED annotation.HED-compliant analysis tools should insert the annotations describing each temporally scoped event into the
Event-contexttag group of the events within its temporal scope during final assembly before analysis of the event.Each of these internal annotations should be in a group, indicating that they represent a distinct event process.
5.6. Experimental design¶
Most experiments are conducted by varying certain aspects of the experiment and measuring the
resulting responses while carefully controlling other aspects.
The intention of the experiment is annotated using the HED Condition-variable,
Control-variable, and Indicator-variable tags.
The Condition-variable tag is used to mark the independent variables of the experiment
– those aspects of an experiment that are explicitly varied in order to observe an effect
or to control bias.
Contrasts, a term that appears in the neuroscience and statistical literature,
are examples of experimental conditions as are factors in experimental designs.
The Indicator-variable tag is used to mark quantities that are explicitly measured or
calculated to evaluate the effect of varying the experimental conditions.
Indicator variables often fall into the Event/Data-feature category.
Sometimes the values of these data features are explicitly annotated as events.
Researchers should provide a sufficiently detailed
description of how to compute these data features so that they can be reproduced.
The Control-variable tag represents an aspect of the experiment that is held constant
throughout the experiment, often to remove variability.
Researchers should use Condition-variable, Control-variable,
and Indicator-variable tags to capture the experiment intent and
organization in as much detail as possible.
Consistent and detailed description allows tools to extract the experiment design
from the data in a machine-actionable form.
Good tagging processes suggest creating definitions with understandable
names to define these aspects of the dataset.
This promotes easy searching and extraction for
analyses such as regression or other modeling of the experimental design.
To illustrate the use of condition-variables to document experiment design, consider an experiment in which one of the conditions is the rate of presentation of images displayed on the screen. The experiment design compares responses under slow and fast image presentation rate conditions. To avoid unfortunate resonances due to a poor choice of rates, the “slow” and “fast” rate conditions each consist of three possible rates. Selection among the three eligible rates for the given condition is done randomly.
In analysis, the researcher would typically combine the “slow presentation” trials into
one group and the “fast presentation” trials into another group even though the exact task
condition varies within the group varies according This type of grouping structure is very
common in experiment design and can be captured by HED tags in a straightforward manner by
defining condition variables for each group and using the # to capture variability within
the group.
Example: Condition variables for slow and fast visual presentation rates.
Short form:
(Definition/SlowPresentation/#,
(Condition-variable/Presentation, Visual-presentation, Computer-screen, Temporal-rate/#))(Definition/FastPresentation/#,
(Condition-variable/Presentation, Visual-presentation, Computer-screen, Temporal-rate/#))
Long form:
(Property/Informational-property/Definition/SlowPresentation/#,
(Property/Organizational-property/Condition-variable/Presentation,
Property/Sensory-property/Sensory-presentation/Visual-presentation,
Item/Object/Man-made-object/Device/IO-device/Output-device/Display-device/Computer-screen,
Property/Data-property/Data-value/Spatiotemporal-value/Rate-of-change/Temporal-rate/#))(Property/Informational-property/Definition/FastPresentation/#,
(Property/Organizational-property/Condition-variable/Presentation,
Property/Sensory-property/Sensory-presentation/Visual-presentation,
Item/Object/Man-made-object/Device/IO-device/Output-device/Display-device/Computer-screen,
Property/Data-property/Data-value/Spatiotemporal-value/Rate-of-change/Temporal-rate/#))
Organizational-property tags such as Condition-variable are often
used in the tag-groups of temporally scoped events.
The Onset of such an event represents the start of theCondition-variable.
The corresponding Offset marks the end of the period during which this condition is in effect.
This type of annotation makes it straightforward to extract
the experimental design from the events.
Example: Annotation using SlowPresentation condition.
Short form:
Sensory-event, (Def/SlowPresentation/1 Hz, Onset)
Long form:
Event/Sensory-event,
(Property/Organizational-property/Def/SlowPresentation/1 Hz,
Property/Data-property/Data-marker/Temporal-marker/Onset)
During analysis, the Def tags may be replaced with the actual definition’s tag group
with an included Def-expand tag giving the definition’s name.
Note: expansion is done by tools at analysis time.
Properly annotated condition variables and response variables can allow researchers to
understand the details of the experiment design and perform analyses such as
ANOVA (ANalysis Of VAriance) or regression to extract the dependence of responses on the
condition variables.
The time-organization of an experiment can be annotated with the
Organizational tags Time-block and Task-trial and used for
visualizations of experimental layout.
A typical experiment usually consists of a sequence of subject task-related activities
interspersed with rest periods and/or off-line activities such as filling in a survey.
The Time-block tag is used to mark a contiguous portion of the data recording during
which some aspect of the experiment conditions is fixed.
Time-block tags can be used to represent temporal organization
in a manner similar to the way Condition-variable
tags are used to represent factors in an experiment design.