Overview

Module khepri_projection

• Description
• Data Types
• Function Index
• Function Details

Description

Khepri projections

Projections build a replicated ETS table using tree nodes from the store which match a khepri_path:pattern(). When a tree node matching a projection's pattern is changed in the store, the tree node is passed through the projection's projection_fun() to create record(s). These records are then stored in the projection's ETS table for all members in a Khepri cluster.

Projections provide a way to query the store as fast as possible and are appropriate for lookups which require low latency and/or high throughput. Projection tables contain all records matching the pattern, though, so the memory footprint of a projection table grows with the number of tree nodes in the store matching the pattern.

Projection ETS tables are owned by the Khepri cluster and are deleted when the cluster stops.

Updates to projection tables are immediately consistent for the member of the cluster on which the change to the store is made and the leader member but are eventually consistent for all other followers.

Projection functions

1. copy (the literal atom)
2. the simple projection function (identified by its arity of 2)
3. the extended projection function (identified by its arity of 4)

copy

This projection "function" inserts the value of a tree node matching the projection path pattern to the ETS table. The value is expected to be a tuple or a record that can be inserted in the table.

If the tree node was deleted, the old value is deleted from the ETS table.

This is the most efficient projection function as it doesn't rely on an anonymous function.

Simple projection function

This projection function is called to convert the value of a tree node matching the projection path pattern to some arbitrary term. If the tree node is created or updated, the value passed to the projection function is the new value after the creation/update. If the tree node is deleted, the value is the one of the tree node before the deletion. The result of the projection function is then inserted in or deleted from the ETS table.

The projection function is expected to map one path/value to one ETS entry. There is no way for the projection function to indicate that a path/value should not be intserted in ETS for instance.

1. the tree node path
2. the tree node value

Example:

ProjectionName = wood_stocks,
ProjectionFun = fun([stock, wood, Kind] = _Path, Stock) ->
                        {Kind, Stock}
                end,
Options = #{type => set,
            read_concurrency => true},
Projection = khepri_projection:new(ProjectionName, ProjectionFun, Options).

The resulting ETS will look like this:

[
 {<<"oak">>, 100},
 {<<"maple">>, 180}
] = ets:tab2list(wood_stoks).

Extended projection function

This projection function is responsible for managing the content of the ETS table(s), even though the ETS tables are still created by Khepri.

This is useful when a tree node value can be mapped to several entries in an ETS table or it can be mapped to entries in several ETS tables.

By default, if the projection is created like a simple projection function, a single ETS table is created, named after the projection. To use several ETS tables, the caller has to specify the list of ETS tables and their per-table ETS options.

1. the table ID, or the map of table names/IDs if multiple tables where configured
2. the tree node path
3. the tree node properies before the update/deletion
4. the tree node properies after the update/deletion

Example:

This extended projection function reproduces the behaviour of the simple projection function from the example above basically.

ProjectionName = wood_stocks,
ProjectionFun = fun
                    %% Stock update.
                    (Tid, [stock, wood, Kind], _OldProps, #{data := Stock}) ->
                        ets:insert(Tid, {Kind, Stock});
 
                    %% Stock deletion.
                    (Tid, [stock, wood, Kind], #{data := Stock}, _NewProps) ->
                        ets:delete(Tid, {Kind, Stock})
                end,
Options = #{type => set,
            read_concurrency => true},
Projection = khepri_projection:new(ProjectionName, ProjectionFun, Options).

The resulting ETS will look like this:

[
 {<<"oak">>, 100},
 {<<"maple">>, 180}
] = ets:tab2list(wood_stoks).

A projection function becomes useful when there are more involved mapping of ETS entries and possibly multiple ETS tables to manage:

ProjectionName = wood_stocks,
ProjectionFun = fun
                    %% Stock update.
                    (#{wood_stocks := WoodStocksTid,
                       wood_needs := WoodNeedsTid},
                     [stock, wood, Kind],
                     _OldProps,
                     #{data := Stock}) ->
                        ets:insert(WoodStocksTid, {Kind, Stock}),
 
                        %% Depending on the stock, we check if we need to order
                        %% new wood.
                        if
                            Stock < 50 ->
                                ets:insert(WoodNeedsTid, {Kind, true});
                            Stock > 1000 ->
                                ets:delete(WoodNeedsTid, {Kind, true});
                            true ->
                                ok
                        end;
 
                    %% Stock deletion.
                    (#{wood_stocks := WoodStocksTid,
                       wood_needs := WoodNeedsTid},
                     [stock, wood, Kind],
                     #{data := Stock},
                     _NewProps) ->
                        ets:delete(WoodStocksTid, {Kind, Stock}),
                        %% We definitely need to order wood of this kind.
                        ets:insert(WoodNeedsTid, {Kind, true});
                end,
Options = #{%% Map of ETS tables and their specific ETS options; here, we don't
            %% need specific ETS options: they will use the globablly defined
            %% options as a fallback.
            tables => #{wood_stocks => #{},
                        wood_needs => #{}},
 
            %% Global ETS options used for tables that do not override them.
            type => set,
            read_concurrency => true},
Projection = khepri_projection:new(ProjectionName, ProjectionFun, Options).

Data Types

ets_options()

abstract datatype: ets_options()

List of ETS options, passed to ets:new/2.

extended_projection_fun()

extended_projection_fun() = fun((Tids::ets:tid() | #{atom() => ets:tid()}, Path::khepri_path:native_path(), OldPayload::khepri:node_props(), NewPayload::khepri:node_props()) -> any())

An extended projection function.

In some cases, a tree node in the store might correspond to many objects in one or more projection tables. Extended projection functions are allowed and expected to call ETS functions directly in order to build the projection table.

OldPayload or NewPayload are empty maps if there is no tree node. For example, a newly created tree node will have an empty map for OldPayload and a khepri:node_props() map with values for NewPayload.

This function is compiled like a transaction function except that calls to the ets module are allowed.

multi_table_options()

multi_table_options() = #{atom() => overridable_ets_options()}

Options which control multiple created ETS tables.

name()

name() = atom()

The name of a projection.

options()

options() = #{tables => multi_table_options(), standalone_fun_options => horus:options(), type => ets:table_type(), keypos => pos_integer(), read_concurrency => boolean(), write_concurrency => boolean() | auto, compressed => boolean()}

overridable_ets_options()

overridable_ets_options() = #{type => ets:table_type(), keypos => pos_integer(), read_concurrency => boolean(), write_concurrency => boolean() | auto, compressed => boolean()}

Overrideable ETS options.

They can be used for single-table and multi-table projections. For the latter, each per-table option takes precedence over the global option if specified twice.

projection()

projection() = #khepri_projection{name = atom(), projection_fun = copy | horus:horus_fun(), ets_options = khepri_projection:ets_options() | #{atom() => khepri_projection:ets_options()}}

A projection resource.

projection_fun()

projection_fun() = copy | simple_projection_fun() | extended_projection_fun()

A function that formats an entry in the tree into a record to be stored in a projection.

• "simple" - a function that takes the path within the tree for a tree node and the payload and returns a record. See simple_projection_fun().
• "copy" - a projection that ignores the path and use the tree node's payload as the record directly. This is a special case that avoids the overhead of executing standalone functions for the sake of performance.
• "extended" - a function that takes the ETS table identifier, path, and old and new tree node properties and executes ETS functions directly. See extended_projection_fun().

simple_projection_fun()

simple_projection_fun() = fun((Path::khepri_path:native_path(), Payload::khepri:data()) -> Record::tuple())

A simple projection function.

Simple projection functions only take the path and payload for a tree node in the store. The record produced by the function is used to create and delete objects in the ETS table.

Function Index

Function Details

new/2

See also: khepri:register_projection/4, khepri_projection:new/3, khepri_projection:new/3.

new/3

Creates a new projection data structure.

name/1

Returns the name of the given Projection.

Overview

Generated by EDoc