Runtime: System.Text.Json: (De)serialization support for quoted numbers

100



Original proposal by @NickCraver (click to view)

Apologies if this issue exists already...I hunted and couldn't find one.

I've been trying to switch a lot of usages over to System.Text.Json from Newtonsoft and Jil, but a recurring theme is handling external APIs and their not-quite-right returns of JSON.

Unfortunately, an example I'm seeing over and over is:

{ "field":"12345" }

They're numbers, but as strings. int, double, decimal, whatever. It happens _all the time_. And since it's someone's API, it's unlikely we'll get the world to fix this.

Note: this is something Newtonsoft handles _by default_, which is why I don't think most people realize it's an issue.

Is there any chance we can let System.Text.Json handle number types being quoted when deserializing? Something a user opts-into via JsonSerializerOptions?

Small repro example:
```c#
void Main()
{
JsonSerializer.Parse(@"{""Bar"":""1234""}");
}
public class Foo
{
public int Bar { get; set; }
}


cc @ahsonkhan @steveharter 
</details>

---

_Edited by @layomia:_

Users want to be able to deserialize JSON strings into number properties. Scenarios include deserializing "NaN", "Infinity" and "-Infinity" (https://github.com/dotnet/runtime/issues/31024).

We should add an option to enable this scenario, and possibly allow serializing numbers as strings.

These semantics allow better interop with various API endpoints accross the web.

## API Proposal

```cs
namespace System.Text.Json
{
    public partial sealed class JsonSerializerOptions
    {
        public JsonNumberHandling NumberHandling { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    [Flags]
    public enum JsonNumberHandling : byte
    {
        /// <summary>
        /// No specified number handling behavior. Numbers can only be read from <see cref="JsonTokenType.Number"/> and will only be written as JSON numbers (without quotes).
        /// </summary>
        None = 0x0,
        /// <summary>
        /// Numbers can be read from <see cref="JsonTokenType.String"/>. Does not prevent numbers from being read from <see cref="JsonTokenType.Number"/>.
        /// </summary>
        AllowReadingFromString = 0x1,
        /// <summary>
        /// Numbers will be written as JSON strings (with quotes), not as JSON numbers.
        /// </summary>
        WriteAsString = 0x2,
        /// Floating point constants represented as <see cref="JsonTokenType.String"/> tokens
        /// such as "NaN", "Infinity", "-Infinity", can be read when reading, and such CLR values
        /// such as <see cref="float.NaN"/>, <see cref="double.PositiveInfinity"/>, <see cref="float.NegativeInfinity"/> will be written as their corresponding JSON string representations.
        AllowNamedFloatingPointLiterals = 0x4
    }

    public partial sealed class JsonNumberHandlingAttribute : JsonAttribute
    {
        public JsonNumberHandling Handling { get; }

        public JsonNumberHandlingAttribute(JsonNumberHandling handling)
        {
            Handling = handling;
        }
    }
}

Usage

Allow reading numbers from strings; write numbers as strings.

public class ClassWithInts
{
    public int NumberOne { get; set; }
    public int NumberTwo { get; set; }
}

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.WriteAsString 
};

string json = @"{""Number1"":1,""Number2"":""2""}";

ClassWithInts @class = JsonSerializer.Deserializer<ClassWithInts>(json, options);
Console.WriteLine(@class.NumberOne); // 1
Console.WriteLine(@class.NumberTwo); // 2

json = JsonSerializer.Serialize(@class, options);
Console.WriteLine(json); // @"{""Number1"":""1"",""Number2"":""2""}";

Allow reading numbers from floating point constants; write floating point constants

Given a class:

public class ClassWithNumbers
{
    public int IntNumber { get; set; }
    public float FloatNumber { get; set; }
}

Without the new option, reading floating-point-constant representations fails:

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Single. Path: $.FloatNumber | LineNumber: 0 | BytePositionInLine: 34.

Writing also fails:

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj);

// Unhandled exception. System.ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.

With the new option, reading floating-point-constant representations works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing floating-point-constant representations also works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Allow reading numbers from string; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Write numbers as strings; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};
string json = @"{""IntNumber"":""1""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // 0
string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 16.

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":"-1","FloatNumber":"NaN"}

Read semantics are the same as the serializer's default (no quotes)

For example, integers cannot have decimal places:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1.0""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 17.

Similarly, there can't be any leading or trailing trivia:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1a""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Leading or trailing whitespace is also not allowed:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1 ""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Note that formats such as the currency and percentage formats are not allowed. Commas (,), the percent symbol (%), and underscores (_) are not permitted in input strings.

Notes

Newtonsoft compat

  • Newtonsoft.Json allows implicit conversions from strings to numbers on deserialization, i.e, no custom logic or setting is required. We will require that users explictly opt in to this behavior.
  • Newtonsoft.Json does not provide a built-in way to serialize numbers as strings. Custom logic, e.g. a custom converter needs to provided for this behavior. This proposal includes an easy way to specify this behavior.

Read/write semantics

  • The number reading and writing behavior of this feature is exactly the same as if no quotes were specified, except that surrounding quotes can be allowed on deserialization, and can be written on serialization.

    • Reading/writing is not culture aware (i.e InvariantCulture). A custom converter has to be specified for culture-aware handling.

    • The "G17" standard numeric format is used when reading and writing double representations.

    • The "G9" standard format is used when reading and writing float representations.

    • Percentage and currency formats are not supported.

  • Numbers represented as JsonTokenType.String tokens will be unescaped if needed when deserializing. This is in keeping with other string-based parsing for types like DateTime and Guid.
  • Only the literal "NaN", "Infinity", and "-Infinty" values are allowed. This is in keeping with the most common variations on the web. The matching is case-sensitive. We can be more permissive in the future, based on user feedback.
NickCraver picture NickCraver  ·  15 Jul 2019

Most helpful comment

31

From Twitter: a lot of people correctly point out that numbers are intentionally quoted in APIs _so they're not assumed to be floats_ and lose precision. It's an excellent point, and even more reason to support deserializing them I think.

NickCraver picture NickCraver  ·  15 Jul 2019

All comments

31

From Twitter: a lot of people correctly point out that numbers are intentionally quoted in APIs _so they're not assumed to be floats_ and lose precision. It's an excellent point, and even more reason to support deserializing them I think.

NickCraver picture NickCraver  ·  15 Jul 2019
0

numbers are intentionally quoted in APIs so they're not assumed to be floats and lose precision

To me that says that the user should still deserialize them as strings and only turn them into numbers when they're ready to accept the loss of precision.

bartonjs picture bartonjs  ·  15 Jul 2019
14

@bartonjs

That's the intention. The data types for the class properties are what we want the final result to be.

If we have to use string-based properties first and then parse ourselves then there's not much utility for this serialization library in terms of mapping to types at all, especially for APIs.

manigandham picture manigandham  ·  15 Jul 2019
-2

With support for a custom double andor decimal converter you can do this yourself. Perhaps there will be future support for a set of "Json.NET-compat" converters at some point but that is not available now.

Below is an skeleton of such a custom converter. Also see https://github.com/dotnet/corefx/issues/38713 which has a custom object converter that converts to bool if token type is TrueFalse.

public class DoubleConverterWithStringSupport : JsonConverter<double>
{
    public override double Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            return ...;
        }

        // Default behavior; will throw if TokenType != Number
        return reader.GetDouble();
    }

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        // Write as number or string? Could also use [JsonConverter] to specify a different converter for certain properties.
    }
}
steveharter picture steveharter  ·  15 Jul 2019
0

Moving to Future. cc @rynowak

steveharter picture steveharter  ·  15 Jul 2019
0

I'm trying to unwrap all that (thanks for the workaround!) - is that saying we'd never consider adding it to the base serializer and it'd only ever live as an extension point? I'm finding the lack of this to be a very common pain point, so I'm wondering what the bar is for inclusion if so.

NickCraver picture NickCraver  ·  15 Jul 2019
9

@NickCraver I think it's just "there's a whole lot of policy here, and we would need to completely spec it out, and we're trying to be feature complete 30 days ago... so this isn't required for v1 (.NET Core 3.0)"

  • Is there an option to say if deserializing should be strict vs coercive? (Personally, I hate coercive APIs, so I would always want it in "this data better conform to schema" mode)
  • If numbers are deserialized through either JSON Number or JSON String, how are they written?

    • Do we need per property toggles other than the converters?

  • Is only invariant format supported? Or also CurrentCulture? Or CurrentUICulture?
  • How should the deserialization handle it if the property is typed as int but the value is the string "3.0"? Is that 3, or an exception? (Currently (IIRC) if it's a JSON number that says 3.0 and the field is int it throws)
  • Should it ignore trailing spaces if they ended up in the string? Leading spaces?
  • Should it support the percentage format? The currency format?

It's a simple-sounding request, but there are a lot of details and many of them are hard to change once the feature first ships. So the .NET Core 3.0 answer is "deserialization doesn't coerce numbers out of JSON strings; but the converters feature lets you effectively opt-in on a property-by-property basis".

bartonjs picture bartonjs  ·  15 Jul 2019
0

Totally understand, I was mostly curious about where the bar is and if moving to this was a good from a future standpoint (or we shouldn't try to move certain code sections to it). That list helps a lot - I agree this can't be just tossed in.

Thanks for the quick reply!

NickCraver picture NickCraver  ·  15 Jul 2019
3

The converter option mentioned by @steveharter works great now in the .NET Core 3.0 preview-7 release. Confirmed server-side and in Blazor.

public class LongToStringConverter : JsonConverter<long>
{
    public override long Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            ReadOnlySpan<byte> span = reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan;
            if (Utf8Parser.TryParse(span, out long number, out int bytesConsumed) && span.Length == bytesConsumed)
                return number;

            if (Int64.TryParse(reader.GetString(), out number))
                return number;
        }

        return reader.GetInt64();
    }

    public override void Write(Utf8JsonWriter writer, long value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(value.ToString());
    }
}
services.AddControllersWithViews()
   .AddJsonOptions(options =>
   {
        options.JsonSerializerOptions.Converters.Add(new LongToStringSupport());
   });

This was developed in the other thread: https://github.com/dotnet/corefx/issues/36563#issuecomment-519788518

manigandham picture manigandham  ·  26 Jul 2019
0

@AlexPaskhin

It's not every type... this case is about Javascript/JSON not supporting large numeric types so it needs a string conversion, and the reasoning by the team on why this isn't built-in makes sense. The converter code is right there so you copy/paste if you need it, but I guess they can add some default converters for long and double in the package.

What other problems are you having? You don't have to use System.Text.Json if you need more flexibility, Json.NET is still available and easily enabled.

manigandham picture manigandham  ·  26 Jul 2019
0

We have a similar situation: we decided to move our project to .net core 3.0. and use the built-in System.Text.Json. We had to rewrite a small part of the functionality, but in general, the build was successful.
However, in runtime we began to receive json parsing errors(not only from other services, but also from our own). We had to do a commit revert and return to Newtonsoft.Json.
Of course, it is clear that it is necessary to correctly transfer all numeric values, without quotes. But our partners are unlikely to want to redo the already approved integration format.

oleg-varlamov picture oleg-varlamov  ·  28 Aug 2019
1

MS Project Manager has wasted million dollars to develop a technical debt. As a share holder I will send complain about him to the MS board. It should help him to listen.

You are welcome to send your feedback to whomever you please, of course. I'll look forward to hearing their response to you.

Please also keep in mind that trying to threaten folks here is against our code of conduct. Folks here are actively listening to feedback and engaging and trying to make a better library for everyone, but conduct such as yours will only serve for them to start ignoring you.

Thank you.

cc: @terrajobst

stephentoub picture stephentoub  ·  28 Aug 2019
0

@AlexPaskhin

Designs are often trade-off decisions, which means making the product better for someone might make it worse for someone else. Implying that this means we don't care or are not good at our job is missing the point.

We appreciate passion and dissenting views, especially when they help us finding bugs and holes in the design. However, we don't appreciate non-constructive feedback, such as your last responses. Implying we don't do our job or directly threatening us isn't acceptable behavior and against our Code of Conduct. Please keep this in mind when engaging with us or we're forced to block you because it's distracting us from delivering quality by engaging in constructive discourse with our community, which is indeed our job.

terrajobst picture terrajobst  ·  29 Aug 2019
0

@terrajobst
I’ve removed all comments in these thread as it is useless, you like/happy with things that already done.
I wish you luck with design trade-off. I wish that your version of “JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate.”
I wish that System.Text.JSON would have a wide success with other teams, with your JSON/CSON (CSharp Object Notation) design trade off.

AlexPaskhin picture AlexPaskhin  ·  29 Aug 2019
15

For 5.0 we can consider an option to support "looser" deserialization modes depending on how common these scenarios are, and whether we can agree on common semantics.

However I believe all such modes can be supported today by using custom converters and by extending JsonConverterAttribute so that specific properties can be targeted (instead of all properties). For some cases it may be that there is no common consensus for the semantics so leaving it as-is and requiring the use of custom converters and custom attributes is the best solution. Perhaps doc and examples for common scenarios works too, or the community creates extensions for common scenarios.

A 5.0 feature could be to add a options.DeserializeQuotedNumbers == true. Then we need to determine if serializing as numbers is always desired, otherwise we'd need another option like SerializeNumbersAsQuoted (perhaps a flags enum would be easier to extend instead of separate option properties). There are subtleties here like do we account for culture settings? We'd also need a new attribute so that specific properties can be targeted instead of all through the global option.

Similarly, there has been feedback on what primitive types to deserialize when the property is a System.Object. Today that is JsonElement. The requests have been to deserialize truefalse as System.Bool, numbers as System.Double and strings as System.String instead of JsonElement.

Adding additional option(s) for this is also possible, however there are subtleties: precision can be lost (e.g. System.Double has less precision than System.Decimal -- what is the intent?) and when the JSON values belong to an object being deserialized into System.Object, the deserializer would still need to deserialize the object and all of its properties as JsonElement since it doesn't know what type to create and deserialize into (without getting into polymorphic deserialization, which is a separate feature).

steveharter picture steveharter  ·  29 Aug 2019
2

From @dansiegel in dotnet/corefx#42396

There should either be a built in JsonConverter that can handle this, a Serialization option to allow handling this, or it should just be smart enough to try to convert from string to any given numeric type.

ericstj picture ericstj  ·  7 Nov 2019
1

Maybe i'm mistaken, but we shouldn't have to have a custom converter.
Is it possible to mimic what the JsonConvert class does? This would eliminate the burden of migrating to this api, and would be a big QOL improvement. It also would make more sense for people to migrate then.

Int32
Int64
DateTime formats

Going-Gone picture Going-Gone  ·  8 Nov 2019
0

I would hate to think I would have to decorate all of my properties to handle this. Really the built in converter should be smart enough to try to parse the string to the given numeric type.

dansiegel picture dansiegel  ·  8 Nov 2019
1

Really the built in converter should be smart enough to try to parse the string to the given numeric type.

So long as it's an option. I definitely don't want it coercing values in my payloads. If I said I want a number, thou shalt not send me a string.

bartonjs picture bartonjs  ·  8 Nov 2019
5

Just don’t forget that JSON (JavaScript Object Notation) is a lightweight data-interchange format. So, JavaScript doesn’t have types. So, If you are going to communicate with “JavaScript” services, Angular, React, TypeScript, Java (currently ~99%) you have to use Newtonsoft json. Unfortunately System.Text.Json doesn’t like “JavaScript” typeless word , cannot find proper type mapping, … and throws exceptions. Unfortunately, it was not developed the “JavaScript” version of System.Text.Json, therefor you can use it (System.Text.Json) in communication only (System.Text.Json) <-> (System.Text.Json) only.

AlexPaskhin picture AlexPaskhin  ·  9 Nov 2019
30

Please add this option
options.DeserializeQuotedNumbers == true
and allow it to deserialize to decimal type.
"mynum":"0.82" -> decimal type.

leftmostlane picture leftmostlane  ·  28 Nov 2019
0

From @igormenshikov in https://github.com/dotnet/runtime/issues/725

I am migrating to .NET Core 3.1 and have a class with null-able values (decimal, DateTime) and want deserialize JSON to it. I received JSON from a client with property values as strings. Here is the sample code:

public class Range
{
    public decimal? Start { get; set; }
    public decimal? End { get; set; }
}

static void Main(string[] args)
{
    var json = "{\"Start\":\"1\",\"End\":\"2\"}";
    var jsonNumber = "{\"Start\": 1,\"End\": 2}";

    var j1 = Newtonsoft.Json.JsonConvert.DeserializeObject<Range>(json); // works fine
    j1.ToString();

    var jn = System.Text.Json.JsonSerializer.Deserialize<Range>(jsonNumber); // works fine
    jn.ToString();

    var js = System.Text.Json.JsonSerializer.Deserialize<Range>(json); // exception
    js.ToString();
}

This code throws an error:

'The JSON value could not be converted to System.Nullable`1[System.Decimal].

I have tried to use a workarounf at:
dotnet/corefx#41070 (comment)

but it does not help. Can something be some with that? Any workaround? Newtonsoft can handle that without problems.

layomia picture layomia  ·  12 Dec 2019
0

From @IgorMenshikov in dotnet/runtime#725

(De)serialization support for quoted numbers should also work for nullable types.

layomia picture layomia  ·  12 Dec 2019
2

Per https://github.com/dotnet/corefx/issues/41442, this issue should consider the deserialization of "Infinity", "-Infinity" and "NaN" into double types (double.PositiveInfinity, double.NegativeInfinity, double.NaN), and vice-versa.

From @pranavkm in https://github.com/dotnet/corefx/issues/41442:

_From @valeriob on Monday, September 30, 2019 12:49:01 PM_

Hi,
we are upgrading some applications to aspnetcore 3.0, we have been bitten by something unexpected : System.Text.Json refuse to serialize double.NaN values.
This cause really nasty bugs because something that works suddenly does not because some result of some calculation is different, or some data changes in some way.
I see that the recommendation (https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.0&tabs=visual-studio#jsonnet-support) is to use Json.Net.

But how can ppl use this library if such a simple and common use case is not covered ?
I do not know any application that can live without this feature, i understand the fact that there is a specification, but it may very well be unpractical.
https://thefactotum.xyz/post/the-devil-is-in-the-json-details/
Python, Go, Javascript,Rust, Ruby handle it without making much fuss 😄

Thanks
Valerio

_Copied from original issue: aspnet/AspNetCore#14571_

layomia picture layomia  ·  12 Dec 2019
0

Renaming this issue to explicitly mark that serialization of quoted numbers is being considered here as well, per @steveharter's comment above:

A 5.0 feature could be to add a options.DeserializeQuotedNumbers == true. Then we need to determine if serializing as numbers is always desired, otherwise we'd need another option like SerializeNumbersAsQuoted (perhaps a flags enum would be easier to extend instead of separate option properties). There are subtleties here like do we account for culture settings? We'd also need a new attribute so that specific properties can be targeted instead of all through the global option.

layomia picture layomia  ·  12 Dec 2019
0

If added, (de)serialization support for numbers as dictionary keys (https://github.com/dotnet/corefx/issues/40120) will use semantics defined here.

layomia picture layomia  ·  12 Dec 2019
-2

Isn't JSON a spec? I see a lot of requests that are app specific, that would be solved simply by having the app specify the value properly.

For example, Angular wants to put something like:
Amount: "15.50"
unless you make the input type="number", in which it will now put:
Amount: 15.50

As soon as you start converting things you will experience feature bloat and bugs like no other, actually creating pain instead of helping future roadmaps for any apps that use the library. JSON isn't meant to be a text processor.

danchaseCTS picture danchaseCTS  ·  25 Dec 2019
0

I would say it is kind of unexpected that a object like VersionVariables.cs needs a JsonConverter<string> to convert numbers into string

jetersen picture jetersen  ·  23 Jan 2020
0

I've added the example above to JsonSerializerOptions for the AddJsonOptions on mvc but it isn't called when sent a string that should be deserialized as a number. Does this work with .net core 3.1 final?

(incidentally this would be far less bad if we could set the defaults for the serializer/deserializer once instead of every single time it's used)

JohnGalt1717 picture JohnGalt1717  ·  18 Feb 2020
2

Here is an improved code from @manigandham:

    public class StringToLongJsonConverter : JsonConverter<long>
    {
        private readonly bool _writeValueAsString;

        public StringToLongJsonConverter(bool writeValueAsString)
        {
            _writeValueAsString = writeValueAsString;
        }

        /// <summary>
        ///     Reads long value from quoted string
        /// </summary>
        public override long Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
        {
            if (reader.TokenType == JsonTokenType.String)
            {
                var span = reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan;
                if (Utf8Parser.TryParse(span, out long number, out var bytesConsumed) && span.Length == bytesConsumed)
                    return number;

                var strNumber = reader.GetString();
                if (long.TryParse(strNumber, out number))
                    return number;

                throw new InvalidOperationException($"{strNumber} is not a correct long value!")
                {
                    /*
                     * Here is a source from internal const System.Text.Json.ThrowHelper.ExceptionSourceValueToRethrowAsJsonException
                     * to restore a detailed info about current json context (line number and path)
                     */
                    Source = "System.Text.Json.Rethrowable"
                };
            }

            return reader.GetInt64();
        }

        /// <summary>
        ///     Writes a long value as number if <see cref="_writeValueAsString" /> is false
        ///     or as string if <see cref="_writeValueAsString" /> is true
        /// </summary>
        public override void Write(Utf8JsonWriter writer, long value, JsonSerializerOptions options)
        {
            if (_writeValueAsString)
                writer.WriteStringValue(value.ToString());
            else
                writer.WriteNumberValue(value);
        }
    }

And don't forget to add the converter like this: serializerOptions.Converters.Add(new StringToLongJsonConverter(false)).

rodion-m picture rodion-m  ·  10 Mar 2020
0

@rodion-m Someone should toss that in a NuGet package for super easy consumption.

VictorioBerra picture VictorioBerra  ·  7 Apr 2020
0

Also, can we make sure the converter supports nullable numbers as well as decimals, longs, integers?

VictorioBerra picture VictorioBerra  ·  7 Apr 2020
3

This should just be fixed in .NET 5 so that it just works properly like Newtonsoft.json and handles nullables properly too.

And allows us to set default serialization and deserialization options so that it doesn't have to be passed with every use.

JohnGalt1717 picture JohnGalt1717  ·  7 Apr 2020
0

Oh, and bools! That worked before as well.

VictorioBerra picture VictorioBerra  ·  7 Apr 2020
0

Another thing, @rodion-m Utf8Parser.TryParse returns false for "". Which will throw. Using Newtonsoft, sending "" in JSON for a number worked just fine. Your code, and the new .NET 5 fix should account for this.

VictorioBerra picture VictorioBerra  ·  9 Apr 2020
4

Shouldn't need a nuget package. This should have been baked in from day one just like serialization defaults etc. It should never have been released until it had parity with Newtonsoft.json.

But since unwise decisions were made, this should be iterated on quickly and fixed in the core product.

JohnGalt1717 picture JohnGalt1717  ·  17 Apr 2020
0

One more thumbs up for System.Text.Json doing the work native like Newtonsoft.Json. We should not need to decorate every property in our models. We had to replace System.Text.Json with Newtonsoft to avoid the fiasco of code bloat due to custom converters for ever numeric type. Not to mention the communication issue of informing every new developer on our team that they have to tag their properties with our custom converters.

Easy to use, hard to misuse.

logic01 picture logic01  ·  3 Jun 2020
0

@logic01 Not that I disagree, but in ASPNET Core you can make the converters work for your type globally without having to put them on the property.

services.AddControllers().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.Converters.Add(new LongToStringConverter());
});
VictorioBerra picture VictorioBerra  ·  3 Jun 2020
0

@NickCraver I think it's just "there's a whole lot of policy here, and we would need to completely spec it out, and we're trying to be feature complete 30 days ago... so this isn't required for v1 (.NET Core 3.0)"

  • Is there an option to say if deserializing should be strict vs coercive? (Personally, I hate coercive APIs, so I would always want it in "this data better conform to schema" mode)
  • If numbers are deserialized through either JSON Number or JSON String, how are they written?

    • Do we need per property toggles other than the converters?
  • Is only invariant format supported? Or also CurrentCulture? Or CurrentUICulture?
  • How should the deserialization handle it if the property is typed as int but the value is the string "3.0"? Is that 3, or an exception? (Currently (IIRC) if it's a JSON number that says 3.0 and the field is int it throws)
  • Should it ignore trailing spaces if they ended up in the string? Leading spaces?
  • Should it support the percentage format? The currency format?

It's a simple-sounding request, but there are a lot of details and many of them are hard to change once the feature first ships. So the .NET Core 3.0 answer is "deserialization doesn't coerce numbers out of JSON strings; but the converters feature lets you effectively opt-in on a property-by-property basis".

What about escaped characters? Should we allow/emit them or should we stick to the JSON Number spec?

For Dictionary<float/double, > support (https://github.com/dotnet/runtime/issues/30524), the value might include '+' e.g. 3.40282347E+38 which makes me wonder these questions:

  • On serialization, should we serialize the dictionary key as is, given that a number written as a number value is not changed by the JsonSerializerOptions.Encoder OR given that we are writing the number as a property name (string) we should allow the Encoder to take action and escape "+" if needed?
  • On deserialization, using the same example, should we allow the escaped "+" (u002b) only and unescape it or should we allow any amount of escaped characters and unescape them all?
Jozkee picture Jozkee  ·  26 Jun 2020
0

Not to oversimplify a complex set of questions @Jozkee , but how does Newtonsoft do it?

Newtonsoft has been the industry standard for a long time and they have answered these questions. If Microsoft want's to replace them as the industry standard then MS should provide an implementation that is as user friendly as Newtonsoft otherwise why switch?

If I switch now I have to make sure that every developer across dozens of teams, dozens of products, hundreds of microservices all conform to our company standard for serialization and that standard must be backwards compatible with Newtonsoft given that we have already put down so much code. In any organization of scale that is a big ask.

logic01 picture logic01  ·  27 Jun 2020
0

Video

  • Looks good.
  • We should have a discussion around defaults for .NET 5, especially web.
  • We decided to rename JsonNumberHandling.None an JsonNumberHandling.Strict

```C#
namespace System.Text.Json
{
public partial sealed class JsonSerializerOptions
{
public JsonNumberHandling NumberHandling { get; set; }
}
}
namespace System.Text.Json.Serialization
{
[Flags]
public enum JsonNumberHandling
{
Strict = 0x0,
AllowReadingFromString = 0x1,
WriteAsString = 0x2,
AllowNamedFloatingPointLiterals = 0x4
}

[AttributeUsage(
    AttributeTargets.Class |
    AttributeTargets.Struct |
    AttributeTargets.Property |
    AttributeTargets.Field, AllowMultiple = false)]
public partial sealed class JsonNumberHandlingAttribute : JsonAttribute
{
    public JsonNumberHandlingAttribute(JsonNumberHandling handling);
    public JsonNumberHandling Handling { get; }    
}

}
```

terrajobst picture terrajobst  ·  14 Jul 2020
0

I'd like to this proposal include an overload on Utf8JsonWriter.WriteNumberValue to specify if it should be quoted or not. e.g.

int x= 5;
writer.WriteNumberValue(x,true) // writes "5"
writer.WriteNumberValue(x,false) // writes 5

This avoids having to do the extra memory allocation in

writer.WriteStringValue(x.ToString()) // writes "5"

and it allows you to write efficient code when the JSON schema you have to comply with has some numbers quoted but not all of them.

paulhickman-a365 picture paulhickman-a365  ·  17 Jul 2020
0

@paulhickman-a365 this issue addresses support at the serializer level only. Please open a new issue with the "API proposal" format for your suggestion for the writer to be considered.

layomia picture layomia  ·  21 Jul 2020
0

It seems this is a missing feature. Better to add this feature like newtonsoft.json

ironpython2001 picture ironpython2001  ·  15 Sep 2020
1

@ironpython2001 this feature is checked in and coming in .NET 5. You can try it in .NET 5 RC1 - https://devblogs.microsoft.com/dotnet/announcing-net-5-0-rc-1/.

layomia picture layomia  ·  15 Sep 2020