Projections 8: internal indexing

Tutorials
,

In the last post we introduced the new concept of fromStreams([]) that will join multiple streams into a single stream for your fold to be run against. We also introduced options and two options that can be used to control the re-ordering behaviour in a distributed environment. In this post we are going to look at how this concept is used internally in the Event Store in order to provide indexing of queries.

Let’s propose a scenario where you have 80,000,000 events in the Event Store. You wish to write a query that looks like:

fromAll().
    when({
        Foo: function(s,e) { ... },
        Bar: function(s,e) { ... },
        Baz: function(s,e) { ... }
    });

In the store there are a total of 500 Foo events, 100 Bar events, and 1000 Baz events. This query would be very very expensive to run as it would need to look through 80,000,000 events in order to run on the 1600 events that you are interested in (the equivalent of a table scan in sql). This query though can (and is) indexed internally!

If you look in options there is another option called “useEventIndexes” that defaults to true. If you enable this option, this query will only look at 1600 events and ignore the other 80,000,000!

This works using the same principles we have been learning in the blog post series. If you look there is a special projection in the Event Store called $by_event_type. This projection is at this point implemented internally but if it were written in JavaScript it would look something like:

fromAll().
    when({
        $any : function(s,e) { linkTo("$et-" + e.type, e); }
    });

In other words the projection will create a stream per event type named $et-{eventtype} that contains links to all events of that type. This standard projection can then be used in conjunction with other projections to provide indexing. Consider our original projection:

fromAll().
    when({
        Foo: function(s,e) { ... },
        Bar: function(s,e) { ... },
        Baz: function(s,e) { ... }
    });

This can now be converted into an indexed projection using fromStreams(). We can convert it into:

fromStreams(["$et-Foo", "$et-Bar", "$et-Baz"]).
    when({
        Foo: function(s,e) { ... },
        Bar: function(s,e) { ... },
        Baz: function(s,e) { ... }
    });

This is done by default. It will then read only from the three joined streams listed! not from all of the events in the system, in our example it will only see 1600 events, not the 80,000,000 that exist. This is how you can combine some of the ideas we have been looking at to provide indexing in various forms for your projections.


Related posts

Data ingestion with Logstash and EventStore

Tutorials
,

In this post our challenge is to load a CSV file and ingest it using the [EventStoreDB](/eventstoredb/) HTTP API to be ingested. To be precise, we want to convert this: ```csv Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60 ``` To this (which is an example of an HTTP POST to the Event Store HTTP API): ```shell [ { "eventId": "fbf4b1a1-b4a3-4dfe-a01f-ec52c34e16e4", "eventType": "InboundDataReceived", "data": { "message": "Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60" }, "metadata": { "host": "box-1", "path": "/usr/data/sales.csv" } } ] ``` In this example we...


Playing with Projections

Tutorials
,

Projections are common concept in [Event Sourcing](/blog/what-is-event-sourcing/) that allow you to create queries of your events and streams of events. Last year I attended Michel Grootjans's "[playing with projections](https://github.com/michelgrootjans/playing_with_projections)" workshop that taught attendees how to create projections in a variety of programming languages. I decided to convert the workshop exercises to use [Event Store's internal projections](/docs/projections/index.html) engine and show how to use our projections API. The data set is player interactions with a quiz. Visit...


Easier Development Builds on Unix-like Operating Systems

Tutorials
,

A common complaint we've heard about [Event Store](/event-store/) since it was originally open sourced is that it's complex to make development builds locally - and they were slow - and thus the project was hard to contribute to. As part of our recent work on Event Store v3.1.0 (more on this in another post) we decided to rectify this. Event Store is primarily written in C#, and builds on Mono on Linux and MacOS and...