Spanish special characters making NewtonSoft bomb

EventStoreDB
1

Hello,

We just noticed something odd happening when one of our team members makes GitHub commits that contain:

ñ

We take the JSON that GitHub sends us, like this:

“commits”: [

{

“id”: “something”,

“distinct”: true,

“message”: “Added status configuration to app.config”,

“timestamp”: “2015-10-07T12:49:15-03:00”,

“url”: “https://hidden”,

“author”: {

“name”: “Santiago Aceñolaza”,

“username”: “acenolaza”

},

“committer”: {

“name”: “Santiago Aceñolaza”,

“username”: “acenolaza”

},

“added”: [

],

“removed”: [

],

“modified”: [

“VersionOne.TeamSync.JiraConnector/Config/JiraSettings.cs”,

“VersionOne.TeamSync.Service/App.config”

]

}

],

And then we attempt to persist into EventStore. It actually appears, on the surface to be working because we aren’t getting errors on the initial write, but I have verified that we do not see the events inside the actual target stream, and then we do get this in the log:

[PID:04092:033 2015.10.07 16:13:08.743 ERROR JsonCodec ] ‘[{“eventId”:“25122d37-c84d-4707-ad53-19eed1f15f46”,“eventType”:“GitHubCommitReceived”,“data”:{“sha”:“something”,“commit”:{“author”:{“name”:“Santiago Aceñolaza”,“email”:"[email protected]",“username”:“acenolaza”},“committer”:{“name”:“Santiago Aceñolaza”,“email”:"[email protected]",“date”:“2015-10-07T12:49:15-03:00”},“message”:“Added status configuration to app.config”},“html_url”:“https://github.com/versionone/VersionOne.TeamSync/commit/42bc908f88cfbab3075b700aa4b0806db723104d",“repository”:{“id”:34900311,“name”:“VersionOne.TeamSync”},“branch”:“S-53254_TeamSyncSyncsStatuses”,“originalMessage”:{“id”:“42bc908f88cfbab3075b700aa4b0806db723104d”,“distinct”:true,“message”:"Added status configuration to app.config”,“timestamp”:“2015-10-07T12:49:15-03:00”,“url”:“https://github.com/versionone/VersionOne.TeamSync/commit/something",“author”:{“name”:"Santiago Aceñolaza”,“email”:"[email protected]",“username”:“acenolaza”},“committer”:{“name”:“Santiago Aceñolaza”,“email”:"[email protected]",“username”:“acenolaza”},“added”:[],“removed”:[],“modified”:[“VersionOne.TeamSync.JiraConnector/Config/JiraSettings.cs”,“VersionOne.TeamSync.Service/App.config”]}},“metadata”:{“instanceId”:“something”,“digestId”:“something”,“inboxId”:"something’ is not a valid serialized EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]

Newtonsoft.Json.JsonReaderException: Unterminated string. Expected delimiter: ". Path ‘[0].metadata.inboxId’, line 1, position 1401.

at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 554

at Newtonsoft.Json.JsonTextReader.ParseString(Char quote) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 118

at Newtonsoft.Json.JsonTextReader.ParseValue() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 925

at Newtonsoft.Json.JsonTextReader.ReadInternal() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 382

at Newtonsoft.Json.JsonTextReader.Read() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 303

at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Int32 initialDepth, Boolean writeChildren, Boolean writeDateConstructorAsDate) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonWriter.cs:line 557

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJObject(JsonReader reader) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 236

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 370

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 258

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 767

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 1794

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 388

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 258

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 1280

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 648

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 260

at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 184

at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonSerializer.cs:line 701

at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 833

at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 792

at EventStore.Transport.Http.Codecs.JsonCodec.From[T](String text) in c:\work\EventStore\src\EventStore.Transport.Http\Codecs\JsonCodec.cs:line 64

I don’t see anything here that indicates any special exclusions: https://github.com/EventStore/EventStore/blob/release-v3.4.0/src/EventStore.Transport.Http/Codecs/JsonCodec.cs

Nor, anything in the NewtonSoft code itself, nor in their issue list at https://github.com/JamesNK/Newtonsoft.Json

Is this a bug?

Thanks,

Josh

2

Can you post what metadata you are sending?

3

I'm just guessing:

"inboxId":"something'

Single quote terminator? Can you send your actual post that you are
doing? We don't do anything on top of newtonsoft.

4

ñ is in the ascii character set, weird that it would cause a problem

5

I obfuscated the meta data, but it’s not actually that sensitive, it’s just some guids:

“metadata”:{“instanceId”:“d80e4bf6-e7fe-420b-8abc-9cfad80f0428”,“digestId”:“4fb047a9-e7fe-420b-8abc-fb7517a3cd0e”,“inboxId”:"d80e4bf6-6f3f-44eb-9e51-2e16381fb06e’ is not a valid serialized EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]

It’s almost like the number of characters is off because it is not recognizing the trailing } on the metadata. It’s fine on all other messages, except when we toss the ñ in there. We tried it where we put the character inside the “message” property of the GitHub commit, as opposed to the author name, and got the same resulting error in the ES logs.

6

Any chance we can try to get a reproduction of this with curl?

7

Sorry about that, lost track of this. I’ll dig it up and post here.