Tokenizers and n-grams

FANTASTIC m-type tokenization and n-gram counting, plus MUST pitch / interval / duration distributions. FANTASTIC pipeline: FANTASTIC workflow in Usage. MUST is documented separately under “MUST tokenization” on that page.

Melody tokenizer

class MType(pitch_interval, ioi_ratio)[source]

Bases: object

A class representing a melody token based on pitch interval and IOI ratio classifications.

Parameters:

pitch_interval (int)
ioi_ratio (float)

__init__(pitch_interval, ioi_ratio)[source]

Initialize an M-Type token.

Parameters:

pitch_interval (int) – The pitch interval classification
ioi_ratio (float) – The IOI ratio classification

class MelodyTokenizer[source]

Bases: object

Base class for melody tokenization strategies.

class FantasticTokenizer(scheme='FANTASTIC')[source]

Bases: MelodyTokenizer

FANTASTIC melody tokenization using classified interval and IOI-ratio m-types.

Parameters:: scheme (str)

__init__(scheme='FANTASTIC')[source]

Initialize the tokenizer with a specific interval classification scheme.

Parameters:: scheme (str, optional) – The scheme to use for pitch interval classification, by default “FANTASTIC” Options: “FANTASTIC”, “SIMILE”

tokenize_melody(pitches, starts, ends)[source]

Tokenize a melody into M-Type tokens.

Parameters:

pitches (List[int])
starts (List[float])
ends (List[float])

Return type:

List[MType]

ngram_counts(n=None)[source]

Get n-gram counts for the current melody.

Parameters:: n (int | None)
Return type:: Dict

segment_melody(melody, phrase_gap=1.5, units='quarters')[source]

Segment melody into phrases based on IOI gaps.

Parameters:

melody (Melody)
phrase_gap (float)
units (str)

Return type:

List[Melody]

class MustDistribution(values, weights)[source]

Bases: object

Normalized MUST distribution weights with their category labels.

Parameters:

values (ndarray)
weights (ndarray)

values: ndarray

weights: ndarray

as_dict()[source]

Map each category label to its normalized weight.

Return type:: dict[Hashable, float]

entropy()[source]

Shannon entropy (natural log) of the weight vector.

Return type:: float

class MustTokenizer[source]

Bases: MelodyTokenizer

MUST distribution tokenization (Clemente et al., 2020).

Implements pdist*, idist*, and ddist* on notematrix-style timing: onsets and durations in beats.

pitch_tokens(melody)[source]

Raw MIDI pitch values (MUST notematrix column 4).

Parameters:: melody (Melody)
Return type:: ndarray

duration_tokens(melody)[source]

Beat durations for all notes except the last, rounded to 2 dp.

Parameters:: melody (Melody)
Return type:: ndarray

static pitch_distribution(pitches)[source]

Marginal pitch distribution (pdist1 on a pitch vector).

Parameters:: pitches (ndarray)
Return type:: MustDistribution

pdist1(melody)[source]

Pitch distribution (MUST pdist1.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

pdist2(melody)[source]

2-tuple pitch distribution (MUST pdist2.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

pdist3(melody)[source]

3-tuple pitch distribution (MUST pdist3.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

idist1(melody)[source]

Interval distribution marginalized from pdist2 (MUST idist1.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

idist2(melody)[source]

2-interval distribution marginalized from pdist3 (MUST idist2.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

ddist1(melody)[source]

Duration distribution in beats (MUST ddist1.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

ddist2(melody)[source]

2-tuple duration distribution (MUST ddist2.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

ddist3(melody)[source]

3-tuple duration distribution (MUST ddist3.m).

Parameters:: melody (Melody)
Return type:: MustDistribution

N-gram counter

class NGramCounter[source]

Bases: object

A stateful n-gram counter that accumulates counts across multiple sequences.

__init__()[source]: Initialize an empty n-gram counter.

count_ngrams(tokens, max_order=5)[source]

Count n-grams in the token sequence up to max_order length.

Parameters:

tokens (list) – List of tokens to count n-grams from
max_order (int, optional) – Maximum n-gram length to count (default: 5)

Return type:

None

reset()[source]

Reset the n-gram counter to empty.

Return type:: None

get_counts(n=None)[source]

Get the current n-gram counts.

Parameters:: n (int, optional) – If provided, only return counts for n-grams of this length. If None, return counts for all n-gram lengths.
Returns:: Dictionary mapping each n-gram to its count
Return type:: dict

property total_tokens: int: Underlying unigram token count of the sequence.

property freq_spec: dict: Frequency spectrum of n-gram counts.

property count_values: list: List of all n-gram counts.

property yules_k: float

Yule’s K measure of m-type repetitiveness.

This lexical-diversity feature is calculated from the frequency spectrum of m-types in the melody. Higher values indicate that a smaller set of m-types is repeated more often, whereas lower values indicate a more even or varied m-type vocabulary.

Citation

Yule (1944)

property simpsons_d: float

Simpson’s D measure of m-type concentration.

Simpson’s D is calculated from squared m-type frequencies. Higher values indicate a greater probability that two sampled tokens belong to the same m-type, and therefore a more concentrated or repetitive m-type vocabulary.

Citation

Simpson (1949)

property sichels_s: float

The proportion of m-types that occur exactly twice.

Sichel’s S is the number of distinct m-types with frequency two divided by the total number of distinct m-types. Higher values indicate that more of the melody’s m-type vocabulary consists of types that recur once.

Citation

Sichel (1975)

property honores_h: float

Honoré’s H measure of m-type lexical richness.

Honoré’s H relates the total number of m-type tokens to the proportion of distinct m-types that occur exactly once (hapax legomena). It increases when a sequence contains many single-occurrence m-types relative to its overall m-type vocabulary.

Citation

Honoré (1979)

property mean_entropy: float

Mean zeroth-order m-type entropy across counted n-gram orders.

For each n-gram order, this feature treats the m-type counts as a discrete distribution and computes zeroth-order entropy. The returned value is the mean of those entropy values across the counted orders. Higher values indicate more even m-type distributions.

property mean_productivity: float

The proportion of distinct m-types that occur only once.

M-types that occur only once are hapax legomena. This feature divides the number of hapax m-types by the total number of distinct m-types, so higher values indicate that more of the melody’s m-type vocabulary is used only once.