partitionBy

Great tangible example - I’ll give it a whirl later

I’ve pushed back the event store posts a week or so, so I’ve got time to sort them all out!

Do we have an overload that just uses outputTo(string)? This would seem to make she’s for most people.

yes, it works

Okay, I’ve done a pull and see that there has been some work done

If I go home this evening and write

fromCategory(‘pushlanguage’)

.foreachStream()

.when({

$init: function(s, ev) {

return { count: 0 }

},

“PushEvent”: function(s, ev) {

s.count++

}

}).outputTo(“PushByLanguage”, “PushByLanguage-{0}”)

This will work? Where do I find the state if so?

I know this is a really dumb example, but if you couple temporal correlation with the partitioning you can start to make some pretty cool analysis across things :slight_smile:

.outputTo("PushByLanguage", "PushByLanguage-{0}")

is saying where to put results to. Look at the stream "PushByLanguage"
linktos will show up in it pointing back to the
PushByLanguage-{stream} streams.

.outputTo("PushByLanguage", "PushByLanguage-{0}")

is saying where to put results to. Look at the stream "PushByLanguage"
linktos will show up in

This should work. Watch PushByLanguage and "PushByLanguage-JavaScript streams.
OutputTo without transformBy will write your state as a result.

Oh cool, I kinda figured it would be something like that - I look forward to trying it!

So, if I read the entire “PushByLangage” stream, this will tell me what partitions there are and where they are, and then I can read those streams to see the state.

You’re saying that transformBy’s purpose is to transform my state into the exact result shape I might want to consume before it is written to a stream?

or null if you dont want a result :slight_smile:

writeIfResult(transformedBy(f(f(init()) e e)))

Very tidy,

I’ll give it a whirl and see if I can get the blog entries re-written with this functionality - I’ve got about a day’s events on my local machine and I’m pushing up to a week on my amazon server at the moment, by the time the blog entries pop I should have three weeks data which should be enough for some pretty good link baity entries.

I’ve updated to latest ‘dev’

Still not getting much joy with this fromCategory stuff :slight_smile:

So, fromCategory clearly works in as much as it recognises the categories, for example

fromCategory(‘pushlanguage’).when({

$init: function() {

return {count: 0};

},

$any: function(s, e) {

s.count++ ;

}

})

Will give me { count: 33 }

Which is how many categories there are (33 languages!!)

If I make this

fromAll()

.foreachStream()

.when({

$init: function() {

return {count: 0};

},

PushEvent: function(s, e) {

s.count++;

}

})

.outputTo(“test18”, “test18-{0}”)

Then I get a results stream for every stream, including the category I’m looking at and every other stream that exists (so that works, I have about 80,000 push events so this takes a bit of time

So if I do

fromCategory(‘pushlanguage’)

.foreachStream()

.when({

$init: function() {

return {count: 0};

},

PushEvent: function(s, e) {

s.count++;

}

})

.outputTo(“test19”, “test19-{0}”)

Nothing of consequence happens (indeed there isn’t even a stream called test19)

If I do

fromCategory(‘pushlanguage’)

.foreachStream()

.when({

$init: function() {

return {count: 0};

},

$any: function(s, e) {

s.count++;

}

})

.outputTo(“test20”, “test20-{0}”)

the last looks like what i would expect… can you a sample event from a pushlanguage stream? they are linktos yes?

I can

fromStream(‘pushlanguage-Ruby’)

.when({

$init: function() {

return {count: 0};

},

PushEvent: function(s, e) {

s.count++ ;

}

})

Gives me

{“count”:3555}

I guess I can always revert to

fromStream(‘github’)

.when({

$init: function() {

return { }

},

PushEvent: function(s, e) {

var language = e.body.repo.language

if(!s[language]) {

s[language] = 0

}

s[language]++

}

})

But that’s pretty much back to the days of partitionBy and non-parallelism, and it doesn’t scale for when I want to do more complicated things(!)

when you do fromcat/foreachstream/when any it counts properly? sry typing on phone…

If I do fromCategory(‘pushlanguage’).foreachStream().when({}) it will run when exactly once for every stream in the category, rather than over the content in the streams.

If I do foreachStream on a fromAlll then it counts properly

If you want the script to plough this data into an event store instance, it’s here:

https://github.com/robashton/githubinfograph

that’s

  • npm install

  • create a config.js with

{

“auth”: "?client_id=MY_CLIENT_ID&client_secret=MY_CLIENT_SECRET

}

Client_id and client_secret can be gotten from Github on https://github.com/settings/applications/new (callback URL is irrelevant)

  • node githubservice.js

That will start ploughing data into the store at :2113 so you can play with me.

sounds like a bug fromcat.foreach.when should run on all in each stream. i will try to repro in london tonight.