Errors on cluster node when writing many events via HTTP

[Pete_Smith]

Hi there,

I have a three node Windows cluster running on Azure VMs.

I’m having trouble trying to write a medium sized amount of events (~24,000) via HTTP. I am using TPL and the following code:

   var tasks = fields.Select(

            async field =>

            {

                var @event = new FieldCreatedEvent(DateTime.UtcNow, field);

                var stream = this.GetStreamName(field.App.Site.Identifier, Context.AppStudioApp,field.App.Identifier);

                var eventData = this.CreateEventData(field.App.Site.Identifier,Context.AppStudioApp,field.App.Identifier,@event);

                var content = new StringContent(JsonConvert.SerializeObject(eventData), new UTF8Encoding(), "application/vnd.eventstore.events+json");

                return

                    client.PostAsync("https://eventstore.softools.net:2113/streams/" + stream, content)

                        .ContinueWith(

                            o =>

                            {

                                if (o.IsFaulted && o.Exception != null)

                                {

                                    throw o.Exception;

                                }

                            });

            });

        await Task.WhenAll(tasks.ToArray());

Here’s an excerpt from the logs from one of the nodes:

[PID:00812:025 2014.10.07 09:29:53.324 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.355 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.417 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.433 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-ibm-Tickets’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-ibm-Tickets’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.433 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.433 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.480 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.495 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.527 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-rbs-Innovation’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-rbs-Innovation’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:53.605 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kt-Dashboard’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kt-Dashboard’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:015 2014.10.07 09:29:53.605 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-Project’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-Project’ (POST): Cannot access a disposed object.

Object name: ‘System.Net.HttpListenerRequest’…

[PID:00812:015 2014.10.07 09:29:53.605 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-Project’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-Project’ (POST): Incorrect function.

[PID:00812:015 2014.10.07 09:29:53.605 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ (POST): Cannot access a disposed object.

Object name: ‘System.Net.HttpListenerRequest’…

[PID:00812:025 2014.10.07 09:29:53.652 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolaglobal-OECertifications’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolaglobal-OECertifications’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-keppel-Issue’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-keppel-Issue’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-keppel-Issue’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-keppel-Issue’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-keppel-Issue’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-keppel-Issue’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-Project’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-Project’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.229 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.260 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-Project’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-Project’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.557 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-afton-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-afton-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.592 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-press-RiskRegister’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-press-RiskRegister’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.620 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-afton-Resource’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-afton-Resource’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:55.620 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:56.011 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:56.027 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:56.205 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kt-CoachingPlan’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kt-CoachingPlan’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:56.768 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:022 2014.10.07 09:29:57.705 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kt-Issue’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kt-Issue’ (POST): Incorrect function.

[PID:00812:025 2014.10.07 09:29:59.404 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kpmg-ExpertResources’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kpmg-ExpertResources’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:29:59.451 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kpmg-ExpertResources’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kpmg-ExpertResources’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kt-People’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kt-People’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-ProjectsApp_copy’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-nutricia-registrations’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-nutricia-registrations’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kpmg-ExpertResources’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kpmg-ExpertResources’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.206 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-forbo-Teams’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-forbo-Teams’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kpmg-ExpertResources’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kpmg-ExpertResources’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolademo-steeringnew’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolademo-steeringnew’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:00.221 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-kpmg-ExpertResources’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-kpmg-ExpertResources’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:022 2014.10.07 09:30:00.440 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-curo-KPI’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-curo-KPI’ (POST): An operation was attempted on a nonexistent network connection.

[PID:00812:025 2014.10.07 09:30:00.628 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-curo-KPI’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-curo-KPI’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:025 2014.10.07 09:30:01.385 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:022 2014.10.07 09:30:01.403 DEBUG AuthenticatedHttpReq] Error occurred while closing timed out connection (http service core): An operation was attempted on a nonexistent network connection.

[PID:00812:025 2014.10.07 09:30:01.403 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:022 2014.10.07 09:30:01.403 DEBUG AuthenticatedHttpReq] Error occurred while closing timed out connection (http service core): An operation was attempted on a nonexistent network connection.

[PID:00812:025 2014.10.07 09:30:01.403 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-ProjectsApp’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:022 2014.10.07 09:30:01.403 DEBUG AuthenticatedHttpReq] Error occurred while closing timed out connection (http service core): An operation was attempted on a nonexistent network connection.

[PID:00812:005 2014.10.07 09:30:02.027 DEBUG AuthenticatedHttpReq] Error occurred while closing timed out connection (http service core): The parameter is incorrect.

[PID:00812:025 2014.10.07 09:30:02.027 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-mainsheet-RAID’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-mainsheet-RAID’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:005 2014.10.07 09:30:02.241 DEBUG AuthenticatedHttpReq] Error occurred while closing timed out connection (http service core): The parameter is incorrect.

[PID:00812:025 2014.10.07 09:30:02.241 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cocacolaglobal-OECertifications’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cocacolaglobal-OECertifications’ (POST): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00812:005 2014.10.07 09:30:03.844 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-ibmdemo-Initiative’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-ibmdemo-Initiative’ (POST): An operation was attempted on a nonexistent network connection.

[PID:00812:005 2014.10.07 09:30:05.589 DEBUG HttpSendService ] Error while forwarding request body from ‘https://eventstore.softools.net:2113/streams/AppStudio.App-cloud9-Project’ to ‘http://192.168.0.4:2112/streams/AppStudio.App-cloud9-Project’ (POST): An operation was attempted on a nonexistent network connection.

Here is an except from the master which intersects with the same timeframe:

[PID:00508:008 2014.10.07 09:29:58.848 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:008 2014.10.07 09:29:58.848 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:008 2014.10.07 09:29:58.848 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:022 2014.10.07 09:29:59.051 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.051 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.051 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.051 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:022 2014.10.07 09:29:59.066 DEBUG AtomController ] Error while reading request (POST entry): The I/O operation has been aborted because of either a thread exit or an application request.

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.145 DEBUG HttpEntityManager ] Error during setting content length on HTTP response: This operation cannot be performed after the response has been submitted…

[PID:00508:035 2014.10.07 09:29:59.160 DEBUG HttpEntityManager ] Close connection error (after crash in read request): An operation was attempted on a nonexistent network connection

[PID:00508:035 2014.10.07 09:29:59.160 DEBUG AtomController ] Error while reading request (POST entry): An operation was attempted on a nonexistent network connection.

Interspersed with this on the master, I am also seeing lots of errors like:

[PID:00508:015 2014.10.07 09:29:27.537 ERROR JsonCodec ] ‘{“EventId”:“79cb6d25-d0ea-42a7-8ba9-c84d8c6838d2”,“EventType”:“FieldCreatedEvent”,“Data”:{“CreatedDate”:“2014-10-07T09:27:09.3264815Z”,“Data”:{“HideGridRowTitle”:false,“IsTitleField”:false,“FormatString”:“yyyy”,“AutoNumber”:false,“DefaultValueIsExpression”:false,“Required”:false,“Orientation”:1,“SubType”:0,“Type”:9,“SystemLabel”:“Steady State Year”,“Identifier”:“SteadyStateYear”,“AppIdentifier”:“project”,“IsUserIdentifier”:false}},“MetaData”:{“Tenant”:“royalmail”,“Username”:“SiteManager”,“ActingTenant”:“royalmail”,“TypeName”:“Softools.Platform.Types.AppStudio.Fields.FieldCreatedEvent-Softools.Platform.Types”,“Identifiers”:{“AppStudioApp”:“project”}}}’ is not a valid serialized EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]

Newtonsoft.Json.JsonSerializationException: Cannot deserialize the current JSON object (e.g. {“name”:“value”}) into type ‘EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]’ because the type requires a JSON array (e.g. [1,2,3]) to deserialize correctly.

To fix this error either change the JSON to a JSON array (e.g. [1,2,3]) or change the deserialized type so that it is a normal .NET type (e.g. not a primitive type like integer, not a collection type like an array or List) that can be deserialized from a JSON object. JsonObjectAttribute can also be added to the type to force it to deserialize from a JSON object.

Path ‘EventId’, line 1, position 11.

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 401

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 258

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 184

at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonSerializer.cs:line 701

at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 833

at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 792

at EventStore.Transport.Http.Codecs.JsonCodec.From[T](String text) in c:\EventStore\src\EventStore.Transport.Http\Codecs\JsonCodec.cs:line 64

There were no problems running this against a single node via TCP. I have also tried this using the JustGiving HTTP Client and get the same result as above.

It does seem that some of the requests get through and succeed, but only a small number of them. It has crossed my mind that perhaps the node is getting overwhelmed with the amount of writes, but surely this should not cause it to fail in this way so I am not entirely sure what could be causing this.

If anyone has any insights it would be much appreciated, please let me know if any more logs or information would be useful.

Thanks, Pete

[Greg_Young1]

What is your http post that you are using?

[PID:00508:015 2014.10.07 09:29:27.537 ERROR JsonCodec ]
'{"EventId":"79cb6d25-d0ea-42a7-8ba9-c84d8c6838d2","EventType":"FieldCreatedEvent","Data":{"CreatedDate":"2014-10-07T09:27:09.3264815Z","Data":{"HideGridRowTitle":false,"IsTitleField":false,"FormatString":"yyyy","AutoNumber":false,"DefaultValueIsExpression":false,"Required":false,"Orientation":1,"SubType":0,"Type":9,"SystemLabel":"Steady
State Year","Identifier":"SteadyStateYear","AppIdentifier":"project","IsUserIdentifier":false}},"MetaData":{"Tenant":"royalmail","Username":"SiteManager","ActingTenant":"royalmail","TypeName":"Softools.Platform.Types.AppStudio.Fields.FieldCreatedEvent-Softools.Platform.Types","Identifiers":{"AppStudioApp":"project"}}}'
is not a valid serialized
EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]
Newtonsoft.Json.JsonSerializationException: Cannot deserialize the
current JSON object (e.g. {"name":"value"}) into type
'EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]'
because the type requires a JSON array (e.g. [1,2,3]) to deserialize
correctly.
To fix this error either change the JSON to a JSON array (e.g.
[1,2,3]) or change the deserialized type so that it is a normal .NET
type (e.g. not a primitive type like integer, not a collection type
like an array or List<T>) that can be deserialized from a JSON object.
JsonObjectAttribute can also be added to the type to force it to
deserialize from a JSON object.
Path 'EventId', line 1, position 11.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader
reader, Type objectType, JsonContract contract, JsonProperty member,
JsonContainerContract containerContract, JsonProperty containerMember,
Object existingValue) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line
401
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader
reader, Type objectType, JsonContract contract, JsonProperty member,
JsonContainerContract containerContract, JsonProperty containerMember,
Object existingValue) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line
258
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader
reader, Type objectType, Boolean checkAdditionalContent) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line
184
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader
reader, Type objectType) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonSerializer.cs:line
701
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type
type, JsonSerializerSettings settings) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line
833
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value,
JsonSerializerSettings settings) in
c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line
792
   at EventStore.Transport.Http.Codecs.JsonCodec.From[T](String text)
in c:\EventStore\src\EventStore.Transport.Http\Codecs\JsonCodec.cs:line
64

[Pete_Smith]

So that question allowed me to see what was causing the JSON issue:

{

Method: POST,

RequestUri: ‘https://eventstore.softools.net:2113/streams/AppStudio.App-royalmail-project’,

Version: 1.1,

Content: System.Net.Http.StringContent,

Headers: {

Content-Type: application/vnd.eventstore.events+json; charset=utf-8

Content-Length: 658

}

}

{“EventId”:“72b3dd2f-ea49-4f81-b2dd-bea4c2217278”,“EventType”:“FieldCreatedEvent”,“Data”:{“CreatedDate”:“2014-10-07T10:50:37.4163777Z”,“Data”:{“HideGridRowTitle”:false,“IsTitleField”:false,“FormatString”:“yyyy”,“AutoNumber”:false,“DefaultValueIsExpression”:false,“Required”:false,“Orientation”:1,“SubType”:0,“Type”:9,“SystemLabel”:“Steady State Year”,“Identifier”:“SteadyStateYear”,“AppIdentifier”:“project”,“IsUserIdentifier”:false}},“MetaData”:{“Tenant”:“royalmail”,“Username”:“SiteManager”,“ActingTenant”:“royalmail”,“TypeName”:“Softools.Platform.Types.AppStudio.Fields.FieldCreatedEvent-Softools.Platform.Types”,“Identifiers”:{“AppStudioApp”:“project”}}}

Evidently there were a few things wrong there (casing, missing array container) that I’d neglected when moving the sample away from the JustGiving client.

With that resolved, I no longer see those JSON convert errors but all the other ones remain.

Pete

[Greg_Young1]

The others look like when forwarding the servers can't talk to each other

Can you access http://192.168.0.4:2112/streams/AppStudio.App-cloud9-ProjectsApp
from the server?

[Pete_Smith]

Yes, I can access that from both of the other nodes although it does require authentication.

[Greg_Young1]

So the error here:

> [PID:00812:022 2014.10.07 09:30:00.440 DEBUG HttpSendService ] Error
> while forwarding request body from
> ‘https://eventstore.softools.net:2113/streams/AppStudio.App-curo-KPI’ to
> ‘http://192.168.0.4:2112/streams/AppStudio.App-curo-KPI’

Is stating that it can't send it to that other address. I am guessing
everything works dandy if you POST to the master node yes?

If you go on another node and do a curl
http://192.168.0.4:2112/streams/AppStudio.App-curo-KPI assuming this
is still your master what do you get?

Greg

[Greg_Young1]

If you run on master it’s ok? And this is from curl or?

[Pete_Smith]

Ok so my addon’s export to Curl was joiked, so ignore those last few messages. Current state of play:

From slave:

C:\EventStore>curl -i “http://192.168.0.4:2112/streams/AppStudio.App-royalmail-project” -X POST -H “Authorization: Basic XXXXXX” -H “Accept-Encoding: gzip,deflate” -H “Content-Type: application/vnd.eventstore.events+json” -d @body.json

HTTP/1.1 201 Created

Content-Length: 0

Content-Type: text/plain; charset=utf-8

Location: http://192.168.0.4:2112/streams/AppStudio.App-royalmail-project/172

Server: Microsoft-HTTPAPI/2.0

Access-Control-Allow-Methods: POST, DELETE, GET, OPTIONS

Access-Control-Allow-Headers: Content-Type, X-Requested-With, X-PINGOTHER, Authorization, ES-LongPoll, ES-ExpectedVersion, ES-EventId, ES-EventType, ES-RequiresMaster, ES-HardDelete, ES-ResolveLinkTo, ES-ExpectedVersion

Access-Control-Allow-Origin: *

Access-Control-Expose-Headers: Location, ES-Position

[Joao_Braganca]

How large is your message

[Joao_Braganca]

How large is your message

[Pete_Smith]

Ok so I have now established that there is nothing wrong with the structure of the requests. It is now definitely load that is now producing the 408 messages. If I manually replay failed requests through fiddler once the batch has finished, I get a 201 success.

Is there a way to configure the EventStore http timeout so that requests are queued up for longer before being thrown out? Or is it the responsibility of the client to react to the 408s and implement some kind of backoff strategy?

@joao - Message length does vary - I have experimented with different batch sizes - but I am seeing for example that sometimes messages ~ 10k are being rejected and others larger than this are succesful.

Pete

The length of messages

[Greg_Young1]

What speed are you sending them at? What level node is this?

[Pete_Smith]

We are sending them as fast as HttpClient will allow using async and TPL… in real terms I’ve measured it and we are sending 1,874 requests over three minutes. Each request contains no more than 10 events.

[Pete_Smith]

Also, we are issuing these to a load balanced endpoint that round-robins between all three nodes in the cluster.

[Joao_Braganca]

are you awaiting the result of the write operation before sending the next one?

[Greg_Young1]

A node should handle 1874 over three minutes without too much issue
depending on network connectivity etc etc. What are you running this
on for network connectivity etc?

"load balanced endpoint that round-robins between all three nodes in
the cluster." why round robin? That an invitation for failure with
non-uniform workload?

[Pete_Smith]

We’ve running the cluster on three Azure VMs, with external access via an azure ‘Load balanced endpoint’. As for the round robin strategy, we’ve just got this set up using a bunch of azure defaults for the moment while we get things up and running.

@joao Nope no waiting… just spamming as fast as TPL will allow.

[Greg_Young1]

What is bandwidth between the nodes and node sizes?

[Pete_Smith]

I don’t have exact figures, but all three nodes are in the same datacenter so I would have thought it would be plenty for these needs.

The nodes themselves are only small - Azure Standard_D1 (1 core, 3.5 GB memory) with SSDs (write cache disabled) We’re not seeing any significant CPU or memory usage spikes on the azure montioring while the batch is being processed however.

[Greg_Young1]

Are you setting requiresmaster header?