Dealing with messy JSON

I was recently consuming data from a FileMaker oData API, and it was doing some very unexpected things.

Here's a simplified version of the payload it was trying to return:

{
  "result": {
    "code": 0,
    "data": {
      "name": "Bob"
    }
  }
}

But for some reason, it encoded that inner data property to a JSON string:

{
  "result": {
    "code": 0,
    "data": "{\"name\":\"Bob\"}"
  }
}

On top of that, it was encoding whitespace characters into that inner output, so I actually saw this:

{
  "result": {
    "code": 0,
    "data": "{\r  \"name\": \"Bob\"\r}"
  }
}

In my Laravel app, when I tried to json_decode that payload, it failed:

json_decode($messyPayload, true, flags: JSON_THROW_ON_ERROR);

// throws JSONException "Control character error, possibly incorrectly encoded"

Initially, I thought those visible \r characters were what it was complaining about, but in actuality those encoded carriage returns were not the issue. The real culprit was the un-encoded raw tab characters that weren't visible. Fun!

Looking at the JSON spec, any ASCII characters below 32 are not allowed. So I had to strip those out before decoding the JSON. And then I had to do an in-place second json_decode for that inner value which was double-encoded:

// strip control characters asci decimal codes 0 - 31
$cleanedPayload = preg_replace('/[\x00-\x1F]/', '', $messyPayload);

// first pass at decoding
$result = json_decode($cleanedPayload, true, flags: JSON_THROW_ON_ERROR);

// now do an in-place second decode of that inner property
$result['result']['data'] = json_decode($result['result']['data'], true, flags: JSON_THROW_ON_ERROR);

Success!

Here to help,

Joel

P.S. One thing that helped me figure this all out was Tinkerwell. Being able to interactively poke at the data and try different things was a huge time-saver.