Projections 5: Indexing

Tutorials
,

Now we can start getting to some of the interesting things in Projections. It was quite odd, as I was leading up to this post last night we got into a very long discussion about indexing inside the Event Store on twitter. Mike Brown noted that it would be really useful if we built lucene like indexing into the system so he could use the Event Store as an Audit Log that was easily searchable by things like correlationId and username. While the indexing is very different than something like lucene it is still quite possible.

In order to get into indexing we will need to introduce a new basic operation linkTo. linkTo works very similarly to emit except when you linkTo(“stream”, event) it does not actually write the event to the stream. It instead emits a pointer to the event to the stream. When you resolve “stream” you will see the event as if it was part of that stream but that is due to the resolution of the pointer (with the TCP API you can say whether or not you want to resolve links). Let’s try an example.

fromAll().when({$any : function(s,e) { linkTo(e.metadata.username, e); });

What this code does is it listens to All events in the system. As they happen it will produce links into streams based upon the user (providing username is in metadata). If we were to have:

Chat1 -> Greg Says hi
Chat2 -> John Says yo
Chat1 -> John Says Hey Greg
Chat2 -> Jill Says donuts!
Chat3 -> Jill Says anyone there?
Chat3 -> Greg Says sure

With this projection running we would be creating indexes. To start with there are three streams here (Chat1,Chat2, and Chat3) They look like:

Stream Chat1
---------
Chat1 -> Greg Says hi
Chat1 -> John Says Hey Greg

Stream Chat2
---------
Chat2 -> John Says yo
Chat2 -> Jill Says donuts!

Stream Chat3
---------
Chat3 -> Jill Says anyone there?
Chat3 -> Greg Says sure

After our index is built we will have six streams (you can build an index at anytime). It will create three additional streams Greg, John, and Jill. They would look at follows.

Stream Greg
---------
Chat1 -> Greg Says hi
Chat3 -> Greg Says sure

Stream John
---------
Chat1 -> John Says Hey Greg
Chat2 -> John Says yo

Stream Jill
---------
Chat2 -> Jill Says donuts!
Chat3 -> Jill Says anyone there?

If I were to point my browser at mydomain.com/streams/jill I would see all of Jill’s chat messages. This is generally how indexes get built up using the Event Store. One nice thing about this methodology is that as the result is a stream, you can listen to that stream like any other stream in the system to get updates for when new things are happening.

You will also notice if you look at an ATOMfeed that the URI to the original event does not change. As such if you are working over ATOM you likely will not pull the event down twice (it will be in cache). In our next post we will look at a quick use case for applying linkTo.


Related posts

Data ingestion with Logstash and EventStore

Tutorials
,

In this post our challenge is to load a CSV file and ingest it using the [EventStoreDB](/eventstoredb/) HTTP API to be ingested. To be precise, we want to convert this: ```csv Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60 ``` To this (which is an example of an HTTP POST to the Event Store HTTP API): ```shell [ { "eventId": "fbf4b1a1-b4a3-4dfe-a01f-ec52c34e16e4", "eventType": "InboundDataReceived", "data": { "message": "Europe,Italy,Clothes,Online,M,12/17/2013,278155219,1/10/2014,1165,109.28,35.84,127311.20,41753.60,85557.60" }, "metadata": { "host": "box-1", "path": "/usr/data/sales.csv" } } ] ``` In this example we...


Playing with Projections

Tutorials
,

Projections are common concept in [Event Sourcing](/blog/what-is-event-sourcing/) that allow you to create queries of your events and streams of events. Last year I attended Michel Grootjans's "[playing with projections](https://github.com/michelgrootjans/playing_with_projections)" workshop that taught attendees how to create projections in a variety of programming languages. I decided to convert the workshop exercises to use [Event Store's internal projections](/docs/projections/index.html) engine and show how to use our projections API. The data set is player interactions with a quiz. Visit...


Easier Development Builds on Unix-like Operating Systems

Tutorials
,

A common complaint we've heard about [Event Store](/event-store/) since it was originally open sourced is that it's complex to make development builds locally - and they were slow - and thus the project was hard to contribute to. As part of our recent work on Event Store v3.1.0 (more on this in another post) we decided to rectify this. Event Store is primarily written in C#, and builds on Mono on Linux and MacOS and...