Tools and services

1. CTagger for annotation

The CTagger tool for annotating data provides a graphical user interface (GUI) to assist HED users in the annotation process. CTagger can be run as a standalone application (https://github.com/hed-standard/hed-java/raw/master/ctagger.jar) or using the HEDTools plug-in in EEGLAB.

The tool is designed to ease the process of constructing HED strings, with features including tag search, an expandable schema-browser view, and free-form formatting. The interchangeability between long-short forms introduced in HED-3G is fully supported. CTagger is also compatible with BIDS, allowing users to import BIDS events.tsv and events.json files to extract the event structure. Once finished tagging, users can export their HED annotation into a json file compatible with BIDS events.json. See CTagger GitHub repository for more details, guides, and tutorials.

2. HED online tools

The web-based tools are summarized in this section. All tools are available from the main access point https://hedtools.ucsd.edu/hed. The services are implemented in a Docker module and can be deployed locally provided that Docker is installed.

Event files are BIDS style tab-separated value files. The first line is always a header line giving the names of the columns, which are used as keys to metadata in accompanying JSON sidecars. The online tools for BIDS events file are designed to help users debug their HED annotations for BIDS datasets before using the BIDS validator.

Web tools for BIDS-style events.tsv files

Assemble events: Assemble HED annotation of a BIDS-style events file.

  • Upload the event file and an optional JSON sidecar file.

  • Specify the HED version.

  • Select the Assemble processing option and whether to expand definitions.

  • Click the Process button.

  • The tool assembles a new two-column event file.

  • If there are errors, a downloadable file of error messages is returned.

  • If no errors, the new event file is returned.

Validate events: Validate a BIDS-style events file with optional JSON sidecar.

  • Upload the event file and an optional JSON sidecar file.

  • Specify the HED version.

  • Select the Validate processing option.

  • Click the Process button.

  • The tool first validates the sidecar if present.

  • If the sidecar contains no errors, the tool validates the events file with the sidecar.

  • If there are any errors, the tool returns a downloadable file of error messages.

Notes:

  1. If the HED version number is given, the tool downloads a standard version from GitHub. Otherwise, the user must upload a local HED schema.

  2. The Assemble creates a two-column event file. The first column is the event onset time and the second column is the full event-level HED annotation.

HED schema tools are designed to assist HED schema developers and library schema developers in making sure that the schema has the correct form and to provide easy conversion between schema formats.

Web tools for HED schema files.

Convert schema: Convert a HED schema between XML and MEDIAWIKI format.

  • Upload a HED schema file or gives a URL pointing to a schema file.

  • Select the Convert option.

  • Press the Process button.

  • The tool returns a downloadable converted file (XML input is converted to MEDIAWIKI and vice versa).

  • Errors are reported as message at the bottom of the screen.

Validate schema: Validate a HED schema between XML and MEDIAWIKI format.

  • Upload a HED schema file or gives a URL pointing to a schema file.

  • Selects the Validate option.

  • Press the Process button.

  • The tool returns a downloadable file of error messages if the schema is invalid.

BIDS JSON sidecars have file names ending in _events.json. These JSON files contain metadata and HED tags applicable to associated events files.

Web tools for BIDS style JSON sidecar.

Convert to long: Convert the HED tags in a BIDS-style events JSON sidecar to long form.

  • Upload the JSON sidecar file.

  • Specify the HED version of HED.

  • Select the Convert to long option.

  • Press the Process button.

  • The tool first validates the sidecar and returns an error file if errors.

  • Otherwise, the tool returns a JSON sidecar with the HED tags converted to full-paths.

Convert to short: Convert the HED tags in a BIDS-style events JSON sidecar to short form.

  • Upload the JSON sidecar file.

  • Specify the HED version.

  • Select the Convert to short option.

  • Press the Process button.

  • The tool first validates the sidecar and returns an error file if errors.

  • Otherwise, the tool returns a JSON sidecar with the HED tags converted to short-form.

Validate sidecar: Validate a single BIDS-style events JSON sidecar.

  • Upload the JSON sidecar file.

  • Specify the HED version.

  • Select the Validate option.

  • Press the Process button.

  • The tool validates the sidecar and returns an error file if there are errors.

Notes:

  1. If the HED version number is given, the tool downloads a standard version from GitHub. Otherwise, the user must upload a local HED schema.

Spreadsheets (either in Excel or tab-separated-value format) are convenient for organizing tags.

Web tools for spreadsheets of HED tags.

Convert to long: Convert the HED tags in tag spreadsheet to long form.

  • Upload the spreadsheet file, indicating whether the first row is a header.

  • Select a worksheet if the spreadsheet is an Excel file with multiple worksheets.

  • Specify the HED version.

  • Select the columns containing HED tags or are prefix columns.

  • Select the Convert to long option.

  • Press the Process button.

  • The tool first validates the spreadsheet and returns an error file if errors.

  • Otherwise, the tool returns a spreadsheet with the HED tags converted to full-paths.

Convert to short: Convert the HED tags in a spreadsheet to short form.

  • Upload the spreadsheet file, indicating whether the first row is a header.

  • Select a worksheet if the spreadsheet is an Excel file with multiple worksheets.

  • Specify the HED version.

  • Select the columns containing HED tags or are prefix columns.

  • Select the Convert to short option.
    _ Press the Process button.

  • The tool first validates the spreadsheet and returns an error file if errors.

  • Otherwise, the tool returns a spreadsheet with the HED tags converted to short-form.

Validate: Validate the HED tags in a spreadsheet.

  • Upload the spreadsheet file, indicating whether the first row is a header.

  • Selects a worksheet if the spreadsheet is an Excel file with multiple worksheets.

  • Specify the HED version.

  • Select the columns containing HED tags or are prefix columns.

  • Select the Validate option.

  • Press the Process button.

  • The tool validates the spreadsheet and returns an error file if the spreadsheet is invalid.

Notes:

  1. If the HED version number is given, the tool downloads a standard version from GitHub. Otherwise, the user must upload a local HED schema.

Online validation of HED strings.

Web tools for processing strings

Convert to long: Convert a HED string to long form.

  • Paste a string into the input text box.

  • Specify the HED version.

  • Select the Convert to long option.

  • Press the Process button.

  • The tool first validates the string and returns errors in the lower text box if any.

  • Otherwise the tool returns the full path version of the strings in the lower text box.

Convert to short: Convert a HED tag string to short form.

  • Paste a string into the input text box.

  • Specify the HED version.

  • Selects the Convert to short option.

  • Press the Process button.

  • The tool first validates the string and returns errors in the lower text box if any.

  • Otherwise the tool returns the short-form versions of the tag strings in the lower text box.

Validate: Validate a HED string.

  • Paste a string into the input text box.

  • Specify the HED version.

  • Select the Validate option and presses the Process button.

  • The tool validates the string and returns any error messages in the lower text box.

Notes:

  1. If the HED version number is given, the tool downloads a standard version from GitHub. Otherwise, the user must upload a local HED schema.

3. HED REST services

HED supports a number of RESTful web services in support of HED including schema conversion and validation, JSON sidecar validation, spreadsheet validation, and validation of a single BIDS event file with supporting JSON sidecar. Short-to-long and long-to-short conversion of HED tags are supported for HED strings, JSON sidecars, BIDS-style events files and spreadsheets in .tsv format. Support is also included for assembling the annotations for a BIDS-style event file with a JSON sidecar. There is also support for extracting a template of a JSON sidecar from a BIDS events file.

The hedexamples/matlab directory of the hed-python repository gives running MATLAB examples of how to call these services in MATLAB.

3.1 Service setup

The HED web services are accessed by making a request to the HED web server to obtain a CSRF access token for the session and then making subsequent requests as designed. The steps are:

  1. Send an HTTP get request to the HED CSRF token access URL.

  2. Extract the CSRF token and returned cookie from the response to use in the headers of future post requests.

  3. Send an HTTP post request in the format as described below and read the response.

The following table summarizes the location of the relevant URLs for online deployments of HED web-based tools and services.

URLs for HED online services.

Service

URL

Online HED tools

https://hedtools.ucsd.edu/hed

CSRF token access

https://hedtools.ucsd.edu/hed/services

Service request

https://hedtools.ucsd.edu/hed/services_submit

3.2 Request format

HED services are accessed by passing a JSON dictionary of parameters in a request to the online server. All requests are in JSON format and include a service name and additional parameters.

The service names are of the form target_command where target specifies the input data type and commandspecifies the service to be performed. For example, events_validate indicates that a BIDS-style event file is to be validated. The exception to this naming rule isget_services.

All parameter values are passed as strings. The contents of file parameters are read into strings to be passed as part of the request. The following example shows the JSON for a HED service request to validate a JSON sidecar. The contents of the JSON file to be validated are abbreviated as "json file text".

Example: Request parameters for validating a JSON sidecar.

{
    "service": "sidecar_validate",
    "schema_version": "8.0.0", 
    "json_string": "json file text",
    "check_for_warnings": "on"
}

The parameters are explained in the following table. Parameter values listed in square brackets (e,g, [a, b]) indicate that only one of a or bshould be provided.

Summary of HED ReST services

Service

Parameters

Descriptions

get_services

none

Returns a list of available services.

events_assemble

events_string,
json_string,
[schema_version,
schema_string],
check_for_warnings,
defs_expand

Assemble tags for each event in a BIDS-style event file into a single HED string.
Returned data: a file of assembled events as text or an error file as text if errors.

events_extract

events_string

Extract a template JSON sidecar based on the contents of the event file.
Returned data: A JSON sidecar (template) if no errors..

events_validate

events_string,
json_string,
[schema_string,
schema_url,
schema_version],
check_for_warnings

Validate a BIDS-style event file and its JSON sidecar if provided.
Returned data: an error file as text if errors.

sidecar_to_long

json_string,
[schema_string,
schema_url,
schema_version],

Convert a JSON sidecar with all of its HED tags expressed in long form.
Returned data: a converted JSON sidecar as text or an error file as text if errors.

sidecar_to_short

json_string,
[schema_string,
schema_url,
schema_version]

Convert a JSON sidecar with all of its HED tags expressed in short form.
Returned data: a converted JSON sidecar as text or an error file as text if errors.

sidecar_validate

json_string,
[schema_string,
schema_url,
schema_version],
check_for_warnings

Validate a BIDS-style JSON sidecar.
Returned data: an error file as text if errors.

spreadsheet_to_long

spreadsheet_string,
[schema_string,
schema_url,
schema_version],
check_for_warnings,
column_x_check,
column_x_input,
has_column_names

Convert a tag spreadsheet (tsv only) to long form.
Returned data: a converted tag spreadsheet as text or an error file as text if errors.

spreadsheet_to_short

spreadsheet_string,
[schema_string,
schema_url,
schema_version],
check_for_warnings,
column_x_check,
column_x_input,
has_column_names

Convert a tag spreadsheet (tsv only) to short form.
Returned data: a converted tag spreadsheet as text or an error file as text if errors.

spreadsheet_validate

spreadsheet_string,
[schema_string,
schema_url,
schema_version],
check_for_warnings,
column_x_check,
column_x_input,
has_column_names,

Validate a tag spreadsheet (tab-separated format only).
Returned data: an error file as text if errors.

strings_to_long

string_list,
[schema_string,
schema_url,
schema_version]

Returns errors or a list of strings to long form.

strings_to_short

string_list,
[schema_string,
schema_url,
schema_version]

Convert errors or a list of short-form strings.

strings_validate

hed_strings,
[schema_string,
schema_url,
schema_version]

Validates a list of hed strings and returns a list of errors.

The following table gives an explanation of the parameters used for various services.

Parameters for web services.

Key value

Type

Description

check_for_warnings

boolean

If true, check for warnings when processing.

column_x_check:

boolean

If present with value ‘on’, column x has HED tags.”.

column_x_input:

string

Contains the prefix prepended to column x if column x has HED tags.

defs_expand

boolean

If true replaces def/XXX with def-expand/XXX grouped with the definition content.

events_string

string

A BIDS events file as a string..

hed_columns

list of numbers

A list of HED string column numbers (starting with 1).

hed_schema_string

string

HED schema in XML format as a string.

hed_strings

list of strings

A list containing HED strings.

json_string

string

BIDS-style JSON events sidecar as a string.

json_strings

string

A list of BIDS-style JSON sidecars as strings.

schema_string

string

A HED schema file as a string.

schema_version

string

Version of HED to be accessed if relevant.

service

string

The name of the requested service.

spreadsheet_string

string

A spreadsheet tsv as a string.

3.3 Service responses

The web-services always return a JSON dictionary with four keys: service, results, error_type, and error_msg. If error_type and error_msg are not empty, the operation failed, while if these fields are empty, the operation completed. Completed operations always return their results in the results dictionary. Keys in the results dictionary return as part of a HED web service response.

The results dictionary.

Key

Type

Description

command

string

Command executed in response to the service request.

command_target

string

The type of data on which the command was executed..

data

string

A list of errors or the processed result.

schema_version

string

The version of the HED schema used in the processing.

msg_category

string

One of success, warning, or failure depending on the result.

msg

string

Explanation of the result of service processing.

The msg and msg_category pertain to contents of the response information. For example a msg_category of warning in response to a validation request indicates that the validation completed and that the object that was validated had validation errors. In contrast, the error_type, and error_msg values are only for web service requests. These keys indicate whether validation was able to take place. Examples of failures that would cause errors include the service timing out or the service request parameters were incorrect.

4. Python tools

The python code for validation is in the hedtools project located in the hed-python repository https://github.com/hed-standard/hed-python. You can install the tools using pip if you have downloaded the hed-python repository:

pip install <hedtools-local-path>

The validation functions are in the hed.validator module. The data representations for various items such as dictionaries or event files can be found in the hed.models module. The hed_input.py module reads in a spreadsheet and possibly a dictionary and creates a HedInput object representing the spreadsheet. The hed-validator.py module creates a HedValidator object that takes a HedSchema object to use in subsequent validation. The validate_input method of HedValidator validates HED input in various formats and returns a list of issues.

5. JavaScript tools

The JavaScript code for HED validation is in the validation directory of the hed-javascript repository located at https://github.com/hed-standard/hed-javascript.

5.1 Installation

You can install the validator using npm:

npm install hed-validator

5.2 Package organization

This package contains two sub-packages.

hedValidator.validator validates HED strings and contains the functions:

buildSchema imports a HED schema and returns a JavaScript Promise object.
validateHedString validates a single HED string using the returned schema object.

hedValidator.converter converts HED strings between short and long forms and contains the following functions:

buildSchema behaves similarly to the buildSchema function in hedValidator.validator except that it does not work with attributes.

convertHedStringToShort converts HED strings from long form to short form.

convertHedStringToLong converts HED strings from short form to long form.

5.3 Programmatic interface

The programmatic interface to the HED JavaScript buildSchema must be modified to accommodate a base HED schema and arbitrary library schemas. The BIDS validator will require additional changes to locate the relevant HED schemas from the specification given by "HEDVersion" in dataset_description.json.

The programmatic interface is similar to the JSON specification of the proposed BIDS implementation except that the "fileName" key has been replaced by a "path" key to emphasize that callers must replace filenames with full paths before calling buildSchema.

Example: JSON passed to buildSchema.

{
    "path": "/data/wonderful/code/mylocal.xml",
    "libraries": {
        "la": {
            "libraryName": "libraryA",
            "version": "1.0.2"
        },
        "lb": {
            "libraryName": "libraryB",
            "path": "/data/wonderful/code/HED_libraryB_0.5.3.xml"
        }
    }
}

NOTE: This interface is proposed and is awaiting resolution of BIDS PR #820 on file passing to BIDS.

6. MATLAB tools

HED validation can be done using the online web-services from MATLAB as shown in the ./examples/matlab directory of the hedweb project in the hed-python repository.