Day IV of the Evented Github Adventure

EventStoreDB
#1

Moving on (after getting distracted by reviewing a pile more Ludum Dare games)

I’m now setting up my production AWS box, I have mono and the event store working properly - but I’d like access to it from the outside world now (for debug purposes)

Now, by default it’ll listen on 127.0.0.1 and attempts to go visit it from say, localhost will fail due to an invalid host

The problem is, I have an elastic IP ( 23.21.129.105 ) with which to visit my box, but that’s not any of the IP addresses held by the card on that machine

Even listening on 0.0.0.0 will yield an invalid host warning

What I could do is set up haproxy to route this for me, but is it possible to just expose the thing directly by saying “Listen on this IP on this hostname”??

#2

You can work fine with elastic ip. You need to listen on that port then set --prefixes for your elastic ip (so the http server allows them in). EG:

SingleNode.exe --ip {localip} --prefixes {yourelasticip}

#3

Ah, I didn’t see the prefixes option - ta

#4

Are there any docs on how to post these projections in a script?

Obviously I know how to write a quick script to post all the files, but I don’t know what headers to pass (and this is still changing judging by the latest pull on dev!)

Which source file to go look in if I want to see what is available for me?

#5

Ah, found it

var name = $("#name").val();

var params = $.param({

name: name,

emit: $("#emit").attr(“checked”) ? “yes” : “no”,

enabled: $("#enabled").attr(“checked”) ? “yes” : “no”,

});

var url = “/projections/” + $("#mode").val() + “?” + params;

$.ajax(url, {

headers: {

Accept: “application/json”,

},

data: $(’#source’).val(),

type: ‘POST’,

success: on_posted,

error: function (xhr) {

var msg = es.util.formatError(“Couldn’t create new projection.”, xhr);

alert(msg);

}

#6

or see the beginning of src/EventStore/EventStore.Projections.Core/Services/Http/ProjectionsController.cs

#7

when this is stable we should update the projections administration page with some curl outputs.

#8

This’ll do for now

var fs = require(‘fs’)

var http = require(‘http’)

var path = require(‘path’)

fs.readdir(‘projections’, function(err, files) {

for(var i = 0 ; i < files.length; i++) {

processFile(files[i])

}

})

function processFile(file) {

var name = file.substr(0, file.indexOf(’.’))

var fullfile = path.join(‘projections’, file)

fs.readFile(fullfile, ‘utf8’, function(err, data) {

var request = http.request({

host: ‘127.0.0.1’,

port: 2113,

method: ‘POST’,

path: ‘/projections/continuous?emit=yes&enabled=yes&name=’ + name,

headers: {

‘accept’: ‘application/json’,

‘content-length’: data.length

}

}, function(res) {

console.log(name, res.statusCode)

})

request.write(data)

request.end()

})

}

Now to get my little graphs hooked up to this data, hopefully I’ll have a preview up by the end of the day before I write some more interesting projections than “stuff what happened recently”

#9

(I actually only need emit for 2/3 of my projections, is there a downside to having it enabled for those that don’t?)

I’m not even sure what it is, something to do with partitioning as it needs it if I use partitionby.

#10

“Enable emit” enables emit(), linkTo() and emitStateUpdated() to be used in a projection. Currently the only difference is that without emits checkpoints for fromStreams([…]) projections can be handled better. These projections write only to -checkpoint streams from time-to-time.

In general projections running without “enable emit” should be faster, but the only way to get their state is via http GET.

foreachStream() projections need emits internally for now, but it will change.

#11

Setting up the event store on this server, I’m using supervisord for laziness and that’s wonderful and great

Now, if the event store needs restarting for any reason, for example the server reboots or I send one of my scripts into an infinite loop because I use unshift instead of shift or whatever…

What’s the easiest way to ensure the default projections are started? It doesn’t seem to be a command line option?

#12

To make sure to start a projection can be scripted (its a http post). I believe with supervised you can specify a script to run

#13

Okay ,after an evening bumming off watching crap on my iPad I’ve come back and

  1. Set up supervisord to set appropriate env vars (LD_LIBRARY_PATH)

  2. Created a script to ensure the projections are running by stopping them then starting them again

  3. Set up an instance of the event store on my production box

  4. Set up haproxy to point at a node instance that is periodically getting projections and preparing them for the client

  5. Set up a background process to churn github events into the event store

  6. Set up Forever to run both the web server and the background process… welll forever

Theoretically that’s all stable, and event store is pointing at a 150gb partition so that should be good

We’ll see how stable it is, lots of logs and all that

http://githubevents.codeofrob.com/

Isn’t very much to look at as it’s midnight and I just started it running, this is largely a stability check so I’d prefer if this link wasn’t passed around until I know it’s working (and I’ve added some prettification to my charts)

#14

Does it live update?

#15

It will do, I’m pulling streams from the event store and I’ll be exposing them to the client via socket-io

#16

Ah ok was trying to get into it :slight_smile:

#17

Btw socket io? That could be interesting with say a few thousand clients :slight_smile:

#18

I’ve had a thousand clients on my server before with socket-io and it was fiiiine (because node is super scalable because the internet said so)

the only reason it capped out at a 1000 is because my haproxy config tells people to bugger off at that point :slight_smile:

#19

I’m kinda with seb on this though, that a more sensible approach is to do polling with an appropriately configured cache sat in the middle - but phooey, it’s not as hacker-news friendly that way

#20

Lol :slight_smile: