Javascript json to most compact QR code
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I have the following example json

https://pastebin.com/2QNjE9hw

Our need is to be able to generate a QR code with all of the data in the json encoded so we can do offline device to device xfer of data. We would like to come up with a json lib that will encode and decode json like this to a more compact format and then encode in a QR code.

For example one of the accounts (there can be many) would go from

    "confirmed": true,
    "value": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3",
    "type": "123"

To this
1:a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3:123

I don’t know a ton about the different levels and formats of QR but I know error correction can add a bit of bloat so let’s fix ecc to be 7% which I think is the lowest.

So the contest task in total is

1- Convert json to a compact format.

2- Convert format to QR - smaller the better.

3- Convert compact format back to a formatted json.

Notes:

-In the full json payload you can see a lot of hashes. It’s fine if needed to make assumptions about trying length for those items.

-Your method should work in the case of multiple “accounts”.

- QR output should be readable by common readers (meaning nothing proprietary).

-$50 to second place winner!

Are there any language requirements/restrictions seeing that you want the implementation to work on "devices"? Are there any restrictions on the compact format (like being human readable)?
dekkard 26 days ago
Nothing that I can think of. As long as in and out are the same you can do any creativity in the middle are you thinking iso 8859-1 ?
Qdev 26 days ago
can you have a sample file of multiple accounts? the qr codes should be saved in what format?(png, jpeg,...)
mashtullah 26 days ago
PNG /gif/svg are all good outputs. I can send an updated json with a couple more accounts.
Qdev 25 days ago
Hey guys really good work on this. its like watching a battle for the lowest byte. my team is reviewing the approaches and we will make the winning call in the next few hours. Also from our side, thanks for the idea around json web tokens.
Qdev 24 days ago
awarded to Wuddrum
Tags
javascript

Crowdsource coding tasks.

5 Solutions


Hi

This is my solution, I have made a Node.js CLI for generating QR code from JSON, the below is a sample usage of the CLI:

Example

node generator.js -j "./data.json" -f -p

And it is the sample Data URL output:

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMQAAADECAYAAADApo5rAAAAAklEQVR4AewaftIAAAimSURBVO3BQY4kSZIEQWFD/P/LvIU52OplFXB4ZFb3rBDhH6mq/zipquukqq6TqrpOquo6qarrpKquk6q6TqrqOqmq66SqrpOquk6q6jqpquukqq6Tqro+eQnIb1KzAfKb1GyAbNRMQN5QswGyUTMB2ajZAPlNat44qarrpKquk6q6PvkyNd8E5A01E5CfBGSjZgKyUTMBmdQ8oWYCMgHZqJmATGo2ar4JyDedVNV1UlXXSVVdn/wwIE+oeQLIBsgbar4JyKRmA2RSMwGZ1ExANmomIBsgk5o3gDyh5iedVNV1UlXXSVVdn9QrQDZqJiAbIJOajZoJyEbNBGSjZgLy3+ykqq6TqrpOqur65L+cmgnIpGYCMgGZ1DwBZFLzBJBJzQRkUvOEmvpfJ1V1nVTVdVJV1yc/TM1vUjMBmdRMQDZqJiCTmgnIE0AmNROQjZoNkCfUPAFkUvOEmn+Sk6q6TqrqOqmq65MvA/JvpmYCMqmZgExqJiA/CcikZqNmArIBMqmZgDwB5J/spKquk6q6Tqrq+uQlNf8kQN4AsgHyhJoJyKRmAjKpmYB8k5o31GzU/JucVNV1UlXXSVVd+EdeADKpmYB8k5ongLyhZgLyhpqfBGSjZgKyUTMBmdRsgHyTmp90UlXXSVVdJ1V1ffJlQCY1E5BJzRNAJjVPqHkCyEbNBGRSswHyTWreUPMEkEnNRs0EZFKzATKp+aaTqrpOquo6qaoL/8gLQCY1E5An1ExAJjVPAHlDzRtA3lCzAbJRMwGZ1GyATGomIBs1E5BJzQRko2YCMql546SqrpOquk6q6vrkJTUTkI2aDZBJzQTkm9RsgLyh5g0gGzUbIBsgGzVPqNmoeULNbzqpquukqq6Tqro++YdRMwGZ1DwB5A01GyCTmgnIpGYDZKNmArJR85uAbNRs1ExANmq+6aSqrpOquk6q6vrky9RMQJ4AMqmZgExq3gAyqZmAbNRMQDZAJjWTmg2QSc0bQH6Smm9SMwGZ1LxxUlXXSVVdJ1V14R/5QUD+JjUTkEnNBGRSMwHZqJmAPKFmA2RS8waQSc0EZKNmAvKb1HzTSVVdJ1V1nVTVhX/ki4BMaiYgGzXfBGRSswHyTWqeAPKT1GyATGomIJOaN4BMap4AMql546SqrpOquk6q6sI/8hcBeUPNBOSb1GyATGomIJOaDZBJzQRkUjMBmdRMQCY1TwDZqHkDyBtq3jipquukqq6Tqro++WVANmqeAPKEmg2QCcikZgNkUvOTgGyATGqeALJRswHyhJoNkJ90UlXXSVVdJ1V1ffISkCfUTEA2QDZqJiAbIJOaN9RsgExqJiCTmo2aN4B8E5CNmieAbNT8pJOquk6q6jqpqgv/yD8IkEnNG0C+Sc0TQDZqNkC+Sc0GyEbNG0A2ajZANmreOKmq66SqrpOquj75MiBPqNkAmdQ8oWYD5Akgk5q/Sc0TQCY1GzVPANmo2QB5Qs03nVTVdVJV10lVXZ+8BGRSMwHZANmo2QCZ1GyATGo2QCY1PwnIpGYCMgGZ1GzUvAFkUrNRMwGZ1ExqJiAbIJOaN06q6jqpquukqi78Iz8IyEbNBKT+b2qeAPKEmieAbNRMQCY1E5CNmp90UlXXSVVdJ1V1ffJlQJ4AMqnZAHlDzQbIpOY3AdkA2ajZqHkCyEbNBsikZqNmAvKbTqrqOqmq66Sqrk9+mJoJyAbIRs0EZKNmA+QNIJOaCcikZqNmAjKp2QB5A8ikZgKyATKp2QCZ1ExqftNJVV0nVXWdVNX1yUtAJjVvqNkAeQLIRs0baiYgGyCTmgnIpGYCslEzAdkAmdQ8AeQNNROQN9S8cVJV10lVXSdVdX3yZUC+CchGzQRkUvMGkG9Ss1GzUbMBMqmZgDwBZFIzAdkA+Tc5qarrpKquk6q6PvmHATKpeULNG0AmNROQjZo3gGzUTEAmNROQjZoJyKTmCTVPANmo2QD5ppOquk6q6jqpquuTL1OzATKpmYBMQJ5QMwGZ1ExAJjUbNROQCchGzRNq3lAzAXkCyBNANmomNRsgGzXfdFJV10lVXSdVdX3yZUAmNRsgGzUTkEnNRs0E5Akgb6iZgGzUfBOQN9T8JCCTmknNbzqpquukqq6Tqrrwj7wAZFLzBJBJzQTkCTVvAJnUPAHkm9RMQCY1E5BJzQbIpGYC8oaaCcikZgIyqZmAbNS8cVJV10lVXSdVdX3yw4C8oWYDZAIyqZmAbNRMQCY1T6iZgExqNkAmNROQSc0EZKNmAjKpeQLIBGRS8wSQSc1POqmq66SqrpOquj75YWomIJOaCchGzUbNBGSjZqPmJwHZqNmomYBMaiYgGzVPAHkCyEbNBsik5ptOquo6qarrpKou/CP/YkAmNROQSc0TQCY1E5A31ExA/iY1E5BJzQRkUvMEkEnNBshGzRsnVXWdVNV1UlXXJy8B+U1qJjUTkEnNBOQJNROQSc0EZFIzAZmAbNRMQCY1E5BJzQbITwIyqdkAeULNN51U1XVSVddJVV2ffJmabwKyATKpmYBMajZAJiBvAJnUPAFkUjMB2QDZqNmoeUPNE2qeADKpeeOkqq6TqrpOqur65IcBeULNG0CeADKp2QDZqHkDyKTmm9RMQL4JyBtA/qaTqrpOquo6qarrk3pEzRNAJjVPAHlDzUbNBshGzX+Tk6q6TqrqOqmq65P/Z4BMar4JyKRmAvKEmgnIpGYDZFIzAdmo2QD5m9R800lVXSdVdZ1U1fXJD1Pzk9RMQL4JyBNqJiCTmg2QCcikZgIyqZnUTEAmNRsg36TmDSA/6aSqrpOquk6q6vrky4D8JiCTmg2QDZBJzQbIRs0E5Ak1E5ANkDeATGo2QCY1GyCTmg2QSc1POqmq66SqrpOquvCPVNV/nFTVdVJV10lVXSdVdZ1U1XVSVddJVV0nVXWdVNV1UlXXSVVdJ1V1nVTVdVJV10lVXf8DvOV90eDW6ggAAAAASUVORK5CYII=

API

-j, --json <s>           The JSON File path or String
-s, --output-as-string   Add this flag to output the QR code as a Data URL string
-f, --output-as-file     Add this flag to output the QR code as a PNG file (Default)
-p, --pack               Add this flag to pack (compact) the JSON data before encoding to QR

Installation

Download the CLI and extract the ZIP and run the generator.js file using Node.js Runtime

Credits

jsonpack - The JSON packing library

qrcode - The QR Code generation library

Any additions and/or questions are welcome!

How small did you get the QR input and can you show the QR?
Qdev 25 days ago
They are all available in the ZIP file, the data.json is the input and the QR.png is the output.
Hasan Bayat 25 days ago
The sample data URL is included in Solution description.
Hasan Bayat 25 days ago
Your solution only compacts a single account (and rather poorly at that). Running jsonpack directly (as in your solution), I'm getting 886 bytes VS 872 bytes (raw JSON with no whitespace). That's a 1.6% increase in size.
CyteBode 25 days ago
Actually it's 655 bytes if not using multi accounts. Updated the solution to include tests and size info.
farolanf 25 days ago
Oh right, I seem to recall one solution had two copies of the account for testing having multiple accounts. I must have forgotten to remove it. I don't have your old solution to make sure, but doing manually what it was doing gives me 677 bytes instead of 750 bytes. I've deleted my old comment as your solution now shows the size. Your updated solution which now flattens the hierarchy indeed achieves a packed size of 655 bytes. That was a clever idea using different value separators for lists and objects to shave off 1 byte! I wish I had thought of that instead of using a list terminator.
CyteBode 25 days ago

Here's another simple solution based on MessagePack for compacting JSON and QR-Code-generator for generating QR codes:

JS

 <script type="text/javascript" src="https://cdn.rawgit.com/ygoe/msgpack.js/199fee6e/msgpack.min.js"></script>
 <script type="text/javascript" src="https://cdn.rawgit.com/nayuki/QR-Code-generator/aa264f5a/javascript/qrcodegen.js"></script>
 <script>
  var compactQr = (function() {
      var QRC = qrcodegen.QrCode;
      var qrBorder = 0;
      var qrScale = 3;
      var qrEcc = QRC.Ecc.LOW;

      var bin = null;

      function getEncoded(json) {
        var obj = JSON.parse(json);
        bin = serializeMsgPack(obj);
        var qr = QRC.encodeBinary(bin, qrEcc);
        return qr;
      }

      return {
        toQrCodeSVG: function(json){
          try {
            var qr = getEncoded(json);
            var result = qr.toSvgString(qrBorder);

            return result;
          } catch (e) {
            console.error(e.message);
            return null;
          }
        },
        toQrCodeDraw: function(json, canvas){
          try {
            var qr = getEncoded(json);
            qr.drawCanvas(qrScale, qrBorder, canvas);

            // For testing purposes
            return bin;
          } catch (e) {
            console.error(e.message);
            return null;
          }
        },
        fromQrCode: function(bin){
          if (bin) {
            var obj = deserializeMsgPack(bin);
            var json = JSON.stringify(obj);
            return json;
          }
          console.error("Nothing to decode!");
          return null;
        },

        // Testing purposes 
        dumpEncoded: function() {
            return bin;
        }
      };
  })();
</script>

Sample usage

<div>
  <h3>Canvas</h3>
  <canvas id="qrcode"></canvas>

  <h3>SVG</h3>
  <div id="svg"></div>
</div>

<script>
  var srcJson = `{
    "accounts": [{
        "confirmed": true,
        "value": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3",
        "type": "123"
    }],
    "encryptKey": "LspSbOsdao2Xx3vk0GcNdQ\u003d\u003d:CIIXpKs5M9ZCWY/9ZZpT+a8sQt+kUvhIzkY0ayWSjSY\u003d",
    "encryptPassword": "a+IJwYkhpbqIWMEiUm39og\u003d\u003d:yX5wpfLNkYEfV5WZHcJSd3mOUINAWjdzpEC0PX+vv5k\u003d:p9yKqvRAUwqWC/jNaF1P5A\u003d\u003d",
    "encryptPrivateKey": "9t+aVC3kZRr8RJH1rY8exw\u003d\u003d:9s4z6/o2e1xglB7mMliLQhPPEI6DQiwv15+OlvpqIoc\u003d:I+1xqssViXcPCYO7AwB4NA\u003d\u003d",
    "encryptSalt": "yVPkKGAVg8fii5BZzmpQ+A\u003d\u003d:UXA7ZCsRO+Mv2um91rz52p7WwxSVRZ6eHk68TWnLmFE\u003d:9i1RxBNdegE+qFfXGFWnVw\u003d\u003d",
    "fingerPrintAuthEnabled": false,
    "invite_code": "Startup2018",
    "isChaseType": false,
    "password": "123456",
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxNWU4ZDkyYi02MTFiLTQ2YTgtYWQ2ZS04ZTAyOTIzMjJkNTEiLCJpYXQiOjE1MzczNTE1ODh9.TdJ3TpAnqVz0jv6AEKDi2jVQEivZ8lc-2_lTgI5cca8",
    "id": "15e8d92b-611b-46a8-ad6e-8e0292322d51"
  }`;

  // 1. Draw on canvas
  compactQr.toQrCodeDraw( srcJson, $("#qrcode").get(0) );

  // 2. Get SVG XML
  var svgXML = compactQr.toQrCodeSVG(srcJson);
  $(svgXML).appendTo("#svg");

</script>

Here's a JSFiddle for your convenience.

This solution reduces the JSON's size from 872 bytes (raw JSON with no whitespace) to 815 bytes, or a 6.5% decrease in size.
CyteBode 25 days ago

Updated version

I've overhauled the code and achieved a much smaller size. The result is now compact enough that using compression actually makes it bigger, so I got rid of it.

Credit where credit is due: Part of the solution is thanks to Wuddrum's great ideas regarding minifying the strings. I don't think I could have come up with it on my own, especially refactoring the JSON Web Token.

Results

With the given example:

  • Raw binary output: 348 bytes (-60.1%).
  • Using Base85 for QR-friendliness: 435 bytes (-50.1%).

This is compared against the size of the raw JSON with no whitespace, which is 872 bytes.

A slightly smaller size for the QR-friendly representation could also be achieved with basE91.

This is the result in Base85 and hex for the raw binary: https://pastebin.com/YxdTWCv7

And here are the resulting QR codes:

Process

The hierarchy is flattened and the keys are discarded. Where applicable, the strings are decoded from hex or Base64 to raw binary and stored as such. The JWT is also refactored to get rid of redundancy. The result, which can be represented with Base85, is finally turned into a QR code.

In order to know how to flatten and unflatten the hierarchy, as well as how the strings are encoded, a schema describing the structure of the JSON data must be provided. Such a schema for the example data is given in the demo below.

Some pretty heavy assumptions had to be made about the nature of the strings (e.g. regarding sizing and encoding). These assumptions are all expressed in the schema, which could easily be modified as needed.

Requirements

json_compactor.js

This is the file containing the library.

String.prototype.replaceAll = function(search, replacement) {
    var target = this;
    return target.split(search).join(replacement);
}

String.prototype.repeat = function(num) {
    return new Array(num + 1).join(this);
}


function hexToBytes(hex) {
    for (var bytes = [], c = 0; c < hex.length; c += 2)
        bytes.push(parseInt(hex.substr(c, 2), 16));
    return bytes;
}

function bytesToHex(bytes) {
    for (var hex = [], i = 0; i < bytes.length; i++) {
        hex.push((bytes[i] >>> 4).toString(16));
        hex.push((bytes[i] & 0xF).toString(16));
    }
    return hex.join("");
}

function typedArrayToArray(typedArray) {
    return [].slice.call(typedArray);
}

function binaryToString(binary) {
    return String.fromCharCode.apply(null, binary);
}

function stringToBinary(string) {
    var output = new Array(string.length);
    for (var i = 0; i < string.length; i++) {
        output[i] = string.charCodeAt(i);
    }
    return output;
}

function b64repad(b64) {
    return b64 + "=".repeat((4 - (b64.length % 4)) % 4);
}

function b64unpad(b64) {
    return b64.replace(/=+$/, "");
}


function BitPacker() {
    var data_store = [];
    var n_bits = 0;

    function addBit(bit) {
        bit = (bit === 0) ? 0 : 1;
        var pos = n_bits % 8;
        if (pos === 0) {
            data_store.push(bit << 7);
        } else {
            var i = data_store.length - 1
            data_store[i] = data_store[i] | (bit << (7 - pos));
        }
        n_bits += 1;
    }

    return {
        "pack": function(data, n) {
            if (typeof n === "undefined") {
                n = data.length * 8;
            }
            var phase = n % 8;
            var mask = (phase === 0) ? 0x80 : 0b1 << (phase - 1);
            var n_bytes = Math.floor((n + 7) / 8);

            while (data.length < n_bytes) {
                data = [0].concat(data);
            }
            data = data.slice(data.length - n_bytes);

            for (var i in data) {
                while (mask) {
                    addBit(data[i] & mask);
                    mask >>= 1;
                }
                mask = 0x80;
            }
        },

        "getData": function() {
            return data_store.slice();
        },

        "getLength": function() {
            return n_bits;
        },
    }
}

function BitUnpacker(data) {
    var data_store = data;
    var pos = 0;

    function getBit(pos) {
        var byte_pos = Math.floor(pos / 8);
        var phase = pos % 8;
        var mask = 0x80 >> phase;
        var shift = 7 - phase;
        return (data_store[byte_pos] & mask) >> shift;
    }

    function unpack(n) {
        var n_bytes = Math.floor((n + 7) / 8);
        var output = [];

        var phase = (8 - (n % 8)) % 8;
        if (phase !== 0) {
            output.push(0);
        }
        var mask = 0x80 >> phase;
        for (var i = 0; i < n; i++) {
            var bit = getBit(pos + i);
            if (phase == 0) {
                output.push(0x00 | (bit ? mask : 0x00));
            } else {
                output[output.length - 1] |= (bit ? mask : 0x00);
            }

            phase = (phase + 1) % 8;
            mask = 0x80 >> phase;
        }

        pos += n;

        return output;
    }

    return {
        "unpack": unpack,

        "peek": function(n, p) {
            if (typeof p === "undefined") {
                p = 0;
            }
            if (p > 0) unpack(p);
            var data = unpack(n);
            pos -= n + p;
            return data;
        },

        "getPosition": function() {
            return pos;
        },

        "setPosition": function(new_pos) {
            pos = new_pos;
        },

        "getLength": function() {
            return data.length * 8;
        }
    }
}


function SObject(kvs) {
    return {
        "serialize": function(json, bit_packer) {
            for (var i in kvs) {
                var name = kvs[i][0];
                var schema = kvs[i][1];
                schema.serialize(json[name], bit_packer);
            }
        },

        "deserialize": function(bit_unpacker) {
            var output = {};
            for (var i in kvs) {
                var name = kvs[i][0];
                var schema = kvs[i][1];

                if (schema.extract === undefined) {
                    output[name] = schema.deserialize(bit_unpacker);
                } else {
                    var _kvs = schema.extract(bit_unpacker);
                    output[name] = schema.deserialize(bit_unpacker);
                    for (var j in _kvs) {
                        output[_kvs[j][0]] = _kvs[j][1];
                    }
                }
            }
            return output;
        }
    }
}


function SList(element_schema) {
    const ELEMENT_SEPARATOR = 0xFF;
    const LIST_TERMINATOR = 0x00;
    return {
        "serialize": function(json, bit_packer) {
            for (var i in json) {
                element_schema.serialize(json[i], bit_packer);
                bit_packer.pack([(i == (json.length - 1)) ? 0 : 1], 1);
            }
        },

        "deserialize": function(bit_unpacker) {
            var output = [];
            while (true) {
                if (bit_unpacker.position >= bit_unpacker.length) {
                    throw "Reached end of serialized!";
                }
                output.push(element_schema.deserialize(bit_unpacker));
                if (bit_unpacker.peek(1)[0] === 0) {
                    bit_unpacker.unpack(1);
                    break;
                }
                bit_unpacker.unpack(1);
            }
            return output;
        }
    }
}


function SString() {
    return {
        "serialize": function(value, bit_packer) {
            bit_packer.pack(typedArrayToArray(
                new TextEncoder("utf-8").encode(value)));
            bit_packer.pack([0], 8);
        },

        "deserialize": function(bit_unpacker) {
            var n = 0;
            while (bit_unpacker.peek(8, n*8)[0] !== 0) {
                if (n >= bit_unpacker.length) {
                    throw "Reached end of serialized!";
                }
                n += 1;
            }
            var bytes = bit_unpacker.unpack(8*n);
            bit_unpacker.unpack(8);
            return new TextDecoder("utf-8").decode(new Uint8Array(bytes));
        }
    }
}


function SUint32() {
    return {
        "serialize": function(value, bit_packer) {
            bit_packer.pack([
                (value >> 24) & 0xFF,
                (value >> 16) & 0xFF,
                (value >>  8) & 0xFF,
                (value      ) & 0xFF
            ], 32);
        },

        "deserialize": function(bit_unpacker) {
            var bytes = bit_unpacker.unpack(32);
            return (bytes[0] << 24) |
                   (bytes[1] << 16) |
                   (bytes[2] <<  8) |
                   (bytes[3]      );
        }
    }
}


function SBoolean() {
    return {
        "serialize": function(value, bit_packer) {
            bit_packer.pack(value ? [1] : [0], 1);
        },

        "deserialize": function(bit_unpacker) {
            return bit_unpacker.unpack(1)[0] === 0 ? false : true;
        }
    }
}


function SHex(size, case_) {
    if (typeof case_ === "undefined") {
        case_ = "lower";
    }
    return {
        "serialize": function(value, bit_packer) {
            bit_packer.pack(hexToBytes(value));
        },

        "deserialize": function(bit_unpacker) {
            var hex = bytesToHex(bit_unpacker.unpack(size * 8));
            if (case_ === "lower") {
                return hex.toLowerCase();
            } else if (case_ === "upper") {
                return hex.toUpperCase();
            }
        }
    }
}


function SHexTuple(sizes, joint, case_) {
    var hexes = sizes.map(sz => SHex(sz, case_));
    return {
        "serialize": function(value, bit_packer) {
            var split = value.split(joint);
            for (var i in hexes) {
                hexes[i].serialize(split[i], bit_packer);
            }
        },

        "deserialize": function(bit_unpacker) {
            segments = hexes.map(hex => hex.deserialize(bit_unpacker));
            return segments.join(joint);
        }
    }
}


function SB64(size, padded) {
    if (typeof(padded) === "undefined") {
        padded = true;
    }
    return {
        "serialize": function(value, bit_packer) {
            var raw_bytes = atob(value);
            bit_packer.pack(stringToBinary(raw_bytes))
        },

        "deserialize": function(bit_unpacker) {
            var raw_bytes = bit_unpacker.unpack(size * 8);
            var b64 = btoa(binaryToString(raw_bytes));
            return padded ? b64repad(b64) : b64;
        }
    }
}


function SB64Tuple(sizes, joint, padded) {
    var b64s = sizes.map(sz => SB64(sz, padded));
    return {
        "serialize": function(value, bit_packer) {
            var split = value.split(joint);
            for (var i in split) {
                b64s[i].serialize(split[i], bit_packer);
            }
        },

        "deserialize": function(bit_unpacker) {
            var segments = b64s.map(b64 => b64.deserialize(bit_unpacker));
            return segments.join(joint);
        }
    }
}


function SJWTPayload(kvs) {
    return {
        "serialize": function(json, bit_packer) {
            for (var i in kvs) {
                var name = kvs[i][0];
                var schema = kvs[i][1];
                schema.serialize(json[name], bit_packer);
            }
        },

        "deserialize": function(bit_unpacker) {
            return kvs.map(kv => [kv[0], kv[1].deserialize(bit_unpacker)]);
        }
    }
}


function b64decode_urlsafe(b64) {
    var b64 = b64.replaceAll("-", "+").replaceAll("_", "/");
    return atob(b64);
}

function b64encode_urlsafe(data) {
    var b64 = btoa(data);
    return b64.replaceAll("+", "-").replaceAll("/", "_");
}

function SJWToken(alg_typ, payload_schema, payload_extractions) {
    var signature_size = Math.floor(Number(alg_typ[0].slice(2)) / 8);
    var header_string = '{"alg":"' + alg_typ[0] + '",' + 
                         '"typ":"' + alg_typ[1] + '"}';

    return {
        "serialize": function(value, bit_packer) {
            var split = value.split(".");
            var payload = JSON.parse(b64decode_urlsafe(split[1]));
            var signature = stringToBinary(b64decode_urlsafe(split[2]));

            payload_schema.serialize(payload, bit_packer);
            bit_packer.pack(signature);
        },

        "deserialize": function(bit_unpacker) {
            var payload = payload_schema.deserialize(bit_unpacker);
            var signature = binaryToString(bit_unpacker.unpack(signature_size * 8));

            var kv_strings = payload.map(function (nv) {
                var obj = {}; obj[nv[0]] = nv[1];
                return JSON.stringify(obj).slice(1, -1);
            });
            var payload_string = "{" + kv_strings.join(",") + "}";

            return ([
                b64unpad(b64encode_urlsafe(header_string)),
                b64unpad(b64encode_urlsafe(payload_string)),
                b64unpad(b64encode_urlsafe(signature)),
            ]).join(".");
        },

        "extract": function(bit_unpacker) {
            var position = bit_unpacker.getPosition();
            var payload = payload_schema.deserialize(bit_unpacker);
            bit_unpacker.setPosition(position);

            var output = [];

            var payload_object = {};
            for (var i in payload) {
                payload_object[payload[i][0]] = payload[i][1];
            }

            return payload_extractions.map(pe => [pe[1], payload_object[pe[0]]]);
        }
    }
}


// https://stackoverflow.com/a/29415858
// By Steve Hanov. Released to the public domain.
function encode_ascii85(input) {
  var output = "<~";
  var chr1, chr2, chr3, chr4, chr, enc1, enc2, enc3, enc4, enc5;
  var i = 0;

  while (i < input.length) {
    // Access past the end of the string is intentional.
    chr1 = input.charCodeAt(i++);
    chr2 = input.charCodeAt(i++);
    chr3 = input.charCodeAt(i++);
    chr4 = input.charCodeAt(i++);

    chr = ((chr1 << 24) | (chr2 << 16) | (chr3 << 8) | chr4) >>> 0;

    enc1 = (chr / (85 * 85 * 85 * 85) | 0) % 85 + 33;
    enc2 = (chr / (85 * 85 * 85) | 0) % 85 + 33;
    enc3 = (chr / (85 * 85) | 0 ) % 85 + 33;
    enc4 = (chr / 85 | 0) % 85 + 33;
    enc5 = chr % 85 + 33;

    output += String.fromCharCode(enc1) +
      String.fromCharCode(enc2);
    if (!isNaN(chr2)) {
      output += String.fromCharCode(enc3);
      if (!isNaN(chr3)) {
        output += String.fromCharCode(enc4);
        if (!isNaN(chr4)) {
          output += String.fromCharCode(enc5);
        }
      }
    }
  }

  output += "~>";

  return output;
}

// https://stackoverflow.com/a/31741264
function decode_ascii85(a) {
  var c, d, e, f, g, h = String, l = "length", w = 255, x = "charCodeAt", y = "slice", z = "replace";
  for ("<~" === a[y](0, 2) && "~>" === a[y](-2), a = a[y](2, -2)[z](/\s/g, "")[z]("z", "!!!!!"), 
  c = "uuuuu"[y](a[l] % 5 || 5), a += c, e = [], f = 0, g = a[l]; g > f; f += 5) d = 52200625 * (a[x](f) - 33) + 614125 * (a[x](f + 1) - 33) + 7225 * (a[x](f + 2) - 33) + 85 * (a[x](f + 3) - 33) + (a[x](f + 4) - 33), 
  e.push(w & d >> 24, w & d >> 16, w & d >> 8, w & d);
  return function(a, b) {
    for (var c = b; c > 0; c--) a.pop();
  }(e, c[l]), h.fromCharCode.apply(h, e);
}

var json_compactor = (function() {
    return {
        "compact": function(json_obj, schema, raw_binary) {
            if (typeof raw_binary === "undefined") {
                raw_binary = false;
            }
            bit_packer = BitPacker();
            schema.serialize(json_obj, bit_packer);
            var compacted = bit_packer.getData();
            if (raw_binary) {
                return binaryToString(compacted);
            } else {
                return encode_ascii85(binaryToString(compacted)).slice(2, -2);
            }
        },

        "decompact": function(compacted, schema, raw_binary) {
            if (typeof raw_binary === "undefined") {
                raw_binary = false;
            }
            if (!raw_binary) {
                compacted = decode_ascii85("<~" + compacted + "~>");
            }
            compacted = stringToBinary(compacted);
            var bit_unpacker = BitUnpacker(compacted);
            return schema.deserialize(bit_unpacker);
        }
    };
}());

Demonstration

This webpage shows the results of compacting the JSON and the reverse process.

In a directory, create a file named index.html and copy-paste the following into it:

index.html

var json_obj = {
    "accounts": [{
            "confirmed": true,
            "value": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3",
            "type": "123"
        }],
    "encryptKey": "LspSbOsdao2Xx3vk0GcNdQ==:CIIXpKs5M9ZCWY/9ZZpT+a8sQt+kUvhIzkY0ayWSjSY=",
    "encryptPassword": "a+IJwYkhpbqIWMEiUm39og==:yX5wpfLNkYEfV5WZHcJSd3mOUINAWjdzpEC0PX+vv5k=:p9yKqvRAUwqWC/jNaF1P5A==",
    "encryptPrivateKey": "9t+aVC3kZRr8RJH1rY8exw==:9s4z6/o2e1xglB7mMliLQhPPEI6DQiwv15+OlvpqIoc=:I+1xqssViXcPCYO7AwB4NA==",
    "encryptSalt": "yVPkKGAVg8fii5BZzmpQ+A==:UXA7ZCsRO+Mv2um91rz52p7WwxSVRZ6eHk68TWnLmFE=:9i1RxBNdegE+qFfXGFWnVw==",
    "fingerPrintAuthEnabled": false,
    "invite_code": "Startup2018",
    "isChaseType": false,
    "password": "123456",
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxNWU4ZDkyYi02MTFiLTQ2YTgtYWQ2ZS04ZTAyOTIzMjJkNTEiLCJpYXQiOjE1MzczNTE1ODh9.TdJ3TpAnqVz0jv6AEKDi2jVQEivZ8lc-2_lTgI5cca8",
    "id": "15e8d92b-611b-46a8-ad6e-8e0292322d51"
}


var json_schema = SObject([

    ["accounts", SList(
        SObject([
            ["confirmed", SBoolean()],
            ["value", SHex(32, "lower")],
            ["type", SString()]
        ])
    )],
    ["encryptKey", SB64Tuple([16, 32], ":")],
    ["encryptPassword", SB64Tuple([16, 32, 16], ":")],
    ["encryptPrivateKey", SB64Tuple([16, 32, 16], ":")],
    ["encryptSalt", SB64Tuple([16, 32, 16], ":")],
    ["fingerPrintAuthEnabled", SBoolean()],
    ["invite_code", SString()],
    ["isChaseType", SBoolean()],
    ["password", SString()],
    ["token", SJWToken(
        ["HS256", "JWT"],
        SJWTPayload([
            ["userId", SHexTuple([4, 2, 2, 2, 6], "-", "lower")],
            ["iat", SUint32()]
        ]),
        [["userId", "id"]]
    )]
    // id is taken from the token
]);


//var json_obj = JSON.parse(json_string);
var compacted_raw = json_compactor.compact(json_obj, json_schema, true);
var compacted_b85 = json_compactor.compact(json_obj, json_schema);
var decompacted   = json_compactor.decompact(compacted_b85, json_schema);
//var decompacted   = json_compactor.decompact(compacted_raw, json_schema, true);


const entityMap = {
  '&': '&amp;',
  '<': '&lt;',
  '>': '&gt;',
  '"': '&quot;',
  "'": '&#39;',
  '/': '&#x2F;',
  '`': '&#x60;',
  '=': '&#x3D;'
};

function escapeHtml (string) {
  return String(string).replace(/[&<>"'`=\/]/g, function (s) {
    return entityMap[s];
  });
}


function reduction_percentage(original, reduced) {
    var reduction = -(1.0 - reduced / json_size);
    return (Math.round(1000 * reduction) / 10.0) + "%";
}


var json_size = (JSON.stringify(json_obj)).length;
var b85_size = compacted_b85.length;
var raw_size = compacted_raw.length;
var b85_reduction = reduction_percentage(json_size, b85_size);
var raw_reduction = reduction_percentage(json_size, raw_size);
var escaped_b85 = escapeHtml(compacted_b85);
var raw_hex = bytesToHex(stringToBinary(compacted_raw));

document.getElementById("input").innerHTML = JSON.stringify(json_obj, null, 4);
document.getElementById("json-size").innerHTML = json_size;

document.getElementById("compacted-b85").innerHTML = escaped_b85;
document.getElementById("compacted-b85-size").innerHTML = b85_size;
document.getElementById("compacted-b85-reduction").innerHTML = b85_reduction

document.getElementById("compacted-raw").innerHTML = raw_hex;
document.getElementById("compacted-raw-size").innerHTML = raw_size;
document.getElementById("compacted-raw-reduction").innerHTML = raw_reduction

document.getElementById("decompacted").innerHTML = JSON.stringify(decompacted, null, 4);


// QR Code (Base85)
var qrcode = new QRCode(
    document.getElementById("qrcode-b85"), {
        text: compacted_b85,
        correctLevel: QRCode.CorrectLevel.L
});

// QR Code (raw binary)
var qrcode = new QRCode(
    document.getElementById("qrcode-raw"), {
        text: compacted_raw,
        correctLevel: QRCode.CorrectLevel.L
});

Then create a directory named js and download the required file shown up above (qrcode.js) into it.

Create a file named json_compactor.js and copy-paste the library code from up above into it.

Finally, create a file named main.js and copy-paste the following code into it:

main.js

var json_obj = {
    "accounts": [{
            "confirmed": true,
            "value": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3",
            "type": "123"
        }],
    "encryptKey": "LspSbOsdao2Xx3vk0GcNdQ==:CIIXpKs5M9ZCWY/9ZZpT+a8sQt+kUvhIzkY0ayWSjSY=",
    "encryptPassword": "a+IJwYkhpbqIWMEiUm39og==:yX5wpfLNkYEfV5WZHcJSd3mOUINAWjdzpEC0PX+vv5k=:p9yKqvRAUwqWC/jNaF1P5A==",
    "encryptPrivateKey": "9t+aVC3kZRr8RJH1rY8exw==:9s4z6/o2e1xglB7mMliLQhPPEI6DQiwv15+OlvpqIoc=:I+1xqssViXcPCYO7AwB4NA==",
    "encryptSalt": "yVPkKGAVg8fii5BZzmpQ+A==:UXA7ZCsRO+Mv2um91rz52p7WwxSVRZ6eHk68TWnLmFE=:9i1RxBNdegE+qFfXGFWnVw==",
    "fingerPrintAuthEnabled": false,
    "invite_code": "Startup2018",
    "isChaseType": false,
    "password": "123456",
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxNWU4ZDkyYi02MTFiLTQ2YTgtYWQ2ZS04ZTAyOTIzMjJkNTEiLCJpYXQiOjE1MzczNTE1ODh9.TdJ3TpAnqVz0jv6AEKDi2jVQEivZ8lc-2_lTgI5cca8",
    "id": "15e8d92b-611b-46a8-ad6e-8e0292322d51"
}


var json_schema = SObject([

    ["accounts", SList(
        SObject([
            ["confirmed", SBoolean()],
            ["value", SHex(32, "lower")],
            ["type", SString()]
        ])
    )],
    ["encryptKey", SB64Tuple([16, 32], ":")],
    ["encryptPassword", SB64Tuple([16, 32, 16], ":")],
    ["encryptPrivateKey", SB64Tuple([16, 32, 16], ":")],
    ["encryptSalt", SB64Tuple([16, 32, 16], ":")],
    ["fingerPrintAuthEnabled", SBoolean()],
    ["invite_code", SString()],
    ["isChaseType", SBoolean()],
    ["password", SString()],
    ["token", SJWToken(
        ["HS256", "JWT"],
        SJWTPayload([
            ["userId", SHexTuple([4, 2, 2, 2, 6], "-", "lower")],
            ["iat", SUint32()]
        ]),
        [["userId", "id"]]
    )]
    // id is taken from the token
]);


//var json_obj = JSON.parse(json_string);
var compacted_raw = json_compactor.compact(json_obj, json_schema, true);
var compacted_b85 = json_compactor.compact(json_obj, json_schema);
var decompacted   = json_compactor.decompact(compacted_b85, json_schema);
//var decompacted   = json_compactor.decompact(compacted_raw, json_schema, true);


const entityMap = {
  '&': '&amp;',
  '<': '&lt;',
  '>': '&gt;',
  '"': '&quot;',
  "'": '&#39;',
  '/': '&#x2F;',
  '`': '&#x60;',
  '=': '&#x3D;'
};

function escapeHtml (string) {
  return String(string).replace(/[&<>"'`=\/]/g, function (s) {
    return entityMap[s];
  });
}


function reduction_percentage(original, reduced) {
    var reduction = -(1.0 - reduced / json_size);
    return (Math.round(1000 * reduction) / 10.0) + "%";
}


var json_size = (JSON.stringify(json_obj)).length;
var b85_size = compacted_b85.length;
var raw_size = compacted_raw.length;
var b85_reduction = reduction_percentage(json_size, b85_size);
var raw_reduction = reduction_percentage(json_size, raw_size);
var escaped_b85 = escapeHtml(compacted_b85);
var raw_hex = bytesToHex(stringToBinary(compacted_raw));

document.getElementById("input").innerHTML = JSON.stringify(json_obj, null, 4);
document.getElementById("json-size").innerHTML = json_size;

document.getElementById("compacted-b85").innerHTML = escaped_b85;
document.getElementById("compacted-b85-size").innerHTML = b85_size;
document.getElementById("compacted-b85-reduction").innerHTML = b85_reduction

document.getElementById("compacted-raw").innerHTML = raw_hex;
document.getElementById("compacted-raw-size").innerHTML = raw_size;
document.getElementById("compacted-raw-reduction").innerHTML = raw_reduction

document.getElementById("decompacted").innerHTML = JSON.stringify(decompacted, null, 4);


// QR Code (Base85)
var qrcode = new QRCode(
    document.getElementById("qrcode-b85"), {
        text: compacted_b85,
        correctLevel: QRCode.CorrectLevel.L
});

// QR Code (raw binary)
var qrcode = new QRCode(
    document.getElementById("qrcode-raw"), {
        text: compacted_raw,
        correctLevel: QRCode.CorrectLevel.L
});

You should now have 3 files in the js directory.

Notes

QR codes are meant more to hold printable text than raw binary data. As such, there aren't many QR code readers that can successfully read off the binary data without mangling it. For example, ZXing Decoder seems to work, but ZBar puts out garbage.

Therefore, it might preferable to use the Base85 representation, which is the default behavior.

Edit 1: Overhauled the code for better compaction and a cleaner schema representation that's easier to work with. Got rid of deflate.

Edit 2: Cleaned up the code and got rid of some minor bugs. Added reduction percentage and hex representation of the raw binary output in the demo.

Edit 3: Fixed the default argument bug (was missing quotes around undefined). Added bit packing to shave off 3 bytes.

Nice! This is very close to what I had in mind originally. I was eager to see how low I can drive the encoded input size and started implementing it yesterday. I'll try to finish my version today, to see how it stacks up with yours.
Wuddrum 24 days ago
Haha, way to prove me wrong that binary size can't be reduced any further (I do redact the statement since I got an idea how to save an extra byte or maybe even more). For a split second, I was thinking of packing other values as well, but I didn't want to risk any collisions and my boolean packing of the first account's confirmed value already felt a little hacky.
I'll have to give your solution a spin later today.
Wuddrum 23 days ago
Winning solution

Here are my solutions: Base64 JSFiddle | Base91 JSFiddle | New Binary/Base91 Solution

Result

Base64 version: Encoded QR input size: 483 bytes | QR Image

Base91 version: Encoded QR input size: 448 bytes | QR Image

New Binary/Base91 version: Encoded QR binary input size: 347 bytes | Encoded QR input base91 size: 427 bytes | Binary QR Image | Base91 QR Image

Dependencies

Base64 version: none

Base91 version: base91.js

New Binary/Base91 version: base91.js | qrcode.js

Solution specifics

The solution is specifically tailored for the provided json payload and makes a couple of assumptions about the JWT:

1. It assumes that the header won't change, since it will be generated by the same app.

2. It assumes that the payload contents won't change (userId and iat).

3. It assumes that userId will always match id from base payload entries.

I can correct any/all of these, if they happen to be incorrect.

Suggestions

I don't know the nature of the app, but it's probably best to add 2 characters(hex representation) as a header to the encoded payload, to indicate the encoding version. And if there are multiple payloads, then another two characters to represent the payload type. Just to ensure better future-proofing.

Further questions

Could you clarify a little more on QR output should be readable by common readers (meaning nothing proprietary)?

Will the QR code be read by 3rd party apps and then get sent to the main app for processing? Or was it meant that you'll be using a 3rd party QR reader component in the main app, and the possibility to read the QR code as binary remains?

I'm asking this because I'm on a somewhat similar path as CyteBode is, but I wanted to ask if binary QR code output is allowed before implementing anything.

EDIT: Improved the solution and fixed late night formatting.

EDIT2: Shoved off 4 more bytes.

EDIT3: Added another solution that switches to Base91 encoding. This helps to shave off 32 more bytes.

EDIT4: Shoved off 3 more bytes for the Base91 solution.

EDIT5: Added a new binary/base91 solution. At this point, I don't think the binary size can be reduced any further. I'll try some things with the base91 version tomorrow, but I don't expect it to go any lower either.

EDIT6: Shoved off 2 more bytes for binary and 3 more bytes for base91 inputs in the new solution.

As always, my hat's off to you. Removing the trailing equals from the Base64 strings is clever enough, but transcoding the id and the JSON web token is truly genius. Not only does it take the size a fair bit below what I achieved, but it does it without compression! You could actually get even smaller by transcoding the accounts' value fields as well, and using a single character to escape the ':''s in the Base64 tuples.
CyteBode 25 days ago
Thanks! You're absolutely right, I completely forgot to minify the account's value field.
I was thinking of using some other character for splitting, instead of ':', so I wouldn't have to escape the base64 tuples at all, but it was getting way too late for me, so I just decided to post the solution and do some minor improvements today.
Wuddrum 25 days ago
Nice shot beating my solution by 2 bytes! I thought of packing the bool's in a bit field as well, since it's so wasteful to use a whole byte per bool, but it wouldn't have worked well with my recursive schema. So I thought of bit packing the whole thing, giving me additional savings from using a single bit to separate/terminate the list. It took some effort to implement and debug, but I got working, which puts me down to 348 bytes, with a nibble to spare :P.
CyteBode 23 days ago
Oh, you had to resort to some pretty nasty tricks this time! I'm gonna let you win as I'm running dry on ideas. That minification idea of yours was so pivotal to getting the size down, I always felt you deserved the main reward anyway. That pissing contest of ours was fun, but I'd rather have my life back at this point!
CyteBode 22 days ago
I'd say only the assumption that account's type won't be longer than 34 bytes is a nasty solution (but I can't really imagine it ever being longer in a realistic scenario, at least when using one-byte characters).
Restructuring the input payload seems like a very valid solution however (granted, it saves only one byte), since it eliminates the need for any control bytes to determine list's size/end.
I agree, it was fun, but I'm also out of further ideas at this point.
Wuddrum 22 days ago
Yeah, it's mainly the 34 bytes assumption that rubs me the wrong way a bit. The reordering to terminate the final string with the end of the stream was really clever and rather clean. I could do it too, by letting my algorithm reorder the schema internally to achieve the same, but that would only put us ex aequo without bringing anything new to the table.
CyteBode 22 days ago
View Timeline