Formats for QR codes, URIs, NFCs, ...

bkolobara

Introduction

There was a discussion on Slack about the Stellar QR code format. You can find it archived here.

I will try to summaries this discussion and add some of my own ideas.

NOTE: The formats proposed here are the result of a brainstorming session and will need multiple iterations before being production ready. The goal of this post is to start the discussion about it, learn about multiple use cases and find the best solution.

Use cases

QRcodes/URIs/NFCs/... can hold any kind of data and can be used to solve different problems for Stellar. One of the use cases is to store secret keys for easier backups and later imports. In this post I will mostly focus on the format for payments/invoices. If you have an idea for the secret key import/export format or other use cases please post it in this thread. It would be great to have them all in one place.

In the end we may end up with one über format that is able to hold information about payments, secret key backups and other use cases in one structure.

Online vs Offline transaction signing

I see two main use cases for sharing Stellar data through QR codes/URIs/NFCs:

The user wants to send an asset to someone on the Stellar network and needs their public key/federation address.
The user is checking out in a shop and need to pay for his purchase.

In the first case you only need the public key of the receiver and maybe information about the asset type, amount and a memo. When you receive the information your wallet builds a transaction and publishes it on the stellar network.

But int the second case the point of sale (POS) system can already prepare the transaction for the user, transfer it to their phone over NFC and the user just needs to sign it and send it back. In this case the user doesn't need to have an internet connection and the purchase can be done much faster as the point of sale system has control about publishing the transaction.

Format vs protocol

When I talk about the format I mean a set of key/value pairs that are used to exchange information between devices/users. If the same keys are part of a json structure in a QR code or attributes in an URI I consider them to be the same format. I think that we should not enforce json on everything as it is not a natural fit.
JSON can be used for QR codes and NFC (NDEF), but the same keys can be translated into attribute/value pairs in an URI. We would not gain anything by enforcing json in an URI as the wallet still needs a custom URI parser to extract the JSON data.

The protocol is a set of messages that are exchanged. During an NFC POS offline payment we would need to exchange multiple messages between the POS system and mobile phone. First the POS system would send a request to the mobile phone to sign the transaction and then the mobile phone needs to send the signed message back. Afterwards the POS system can send a confirmation over NFC. The formats for all this messages would be defined as part of the NFC POS protocol.

In some other cases the protocol would just consist of one message/format (QR code with information about the payment).

It does not make sense to use all formats for all protocols. It would be extremely inefficient to use QR codes to exchange data about offline signing. NFC fits this use case much better.

Formats

I would prefer to keep the format as a flat set of key/value pairs. URIs for example don't support nesting elements and you would need to additionally encode them. You can always flatten out the structures with prefixes. For example:

{
  'account': {'id': 2344,
		      'name': 'Stellar'}
}

would become

{
  'account_id': 2344,
  'account_name': 'Stellar',
}

Receiving payments

As Scott suggested on Slack, a really simple format to ask for payments would be:

{
  'receiver': 'GCA4S3...',
  'amount': '100',
  'currency': 'XLM',
  'memo': 'internal_tx_id_like_12839109'
}

Notice that this format is in JSON, but it can also be represented as an URI:
stellar:?receiver=GCA4S3...&amount=100&currency=XLM...

that can be used as a link on a webpage. When you click it, it opens your wallet and prepares the transaction. Similar to torrent magnet links.

Another format, used by sacarlson's wallet is:

accountID : the public address of recieved funded account (optional as it is created with just seed ) value is a string of uncoded public address
seed : secreet seed of a recieved funded account value is a string of private seed that is optionaly encrypted
env_b64 : an envelope blob to be signed or transacted value is a string in base64 format of a tx envelope
amount : amount that would be sent in an invoice value is floating number
memo : a memo that will be attached to the invoice of the store name or item being purchased value is a string max 32 long
asset : type of asset that would be sent to the store from the invoice value is a string of an asset class
issuer : issuer of the asset that the store is expecting to recieve value is a string of a public address of the issure of an asset
destination : the public address of the store or person that is to recieve the invoice billed funds.```

sacarlson: quotes in my format are replaced with %22 and space is replaced with %20 to allow url link compatibility

And Johan suggested the following format:

invoice: {
    asset: {
        assetcode: 'USD',
        issuer: 'address of issuer'
    },
    amount: '3.1415',
    destination: 'seller*seller.com',
    memo: {
        type: 'MEMO_HASH',
        data: 'de42425863ffbb12..'
    },
    message: 'Thank you for shopping at seller.com'
}

I prefer Johan's format, as the naming of keys is much clearer. Just maybe have 2 different destination types, for federated addresses and public keys. The wallet needs to handle them differently. And I would flatten out the structure like this:

type : 'invoice',
asset_code: 'USD',
asset_issuer: 'address of issuer',
amount: '3.1415',
destination: 'seller*seller.com',
memo_type: 'MEMO_HASH',
memo_data: 'de42425863ffbb12..'
message: 'Thank you for shopping at seller.com'

We still need to think about what kind of types we want (e.g. invoice, offline_invoice,...). In this case we can define also types for importing/exporting secret keys. We can have one type for plain text secret keys and another for encrypted keys.

Some protocols would consist of multiple messages, like the offline_invoice type and we will probably need sub types to distinguish between them.

Maybe it would be good to have an version field to specify the version of the format so we can introduce changes that are not backwards compatible at some point.

This was mostly a brain-dump of my ideas. But I think it's a good starting point for the discussion.

scottc

This is great. Here are a couple of desiderata I think are worth considering. They are not all internally consistent so some trade-offs will probably be necessary.

Consistency with SCP. It would be nice if the key names and other things mirrored as closely as possible the names that stellar-core and horizon use for similar objects so that this doesn't become a "second layer". We want it to seem like it was built in from the start.
Conduit consistency. I think it would be great if whether you passed the information as a URI, over QRCode, NFC, Bluetooth, or some other way, the format of the data that is passed is the same. So while a really long base64 message is not ideally written as a URI, it could be written that way if one wanted. Relatedly, the protocol should not require that the data be passed over the web. It should be possible to pass by tcp, QRCode, NFC, morse code, carrier pigeon, etc.
Transparency. Ideally the key names would clearly indicate what the value represents. Also, except where security requires them, I think its good to avoid hashes and non utf-8 encodings. Human readable data that doesn't have to be decoded is preferable to the alternative, all else being equal.
Optionality and Extensibility. This is more on the implementation side. Ideally the software used to process this would be able to handle any subset of the possible keys. Obviously if some important information is missing the wallet might not be able to do anything without getting that information, but it hopefully wouldn't choke. On the other side, the software should allow for arbitrary additions to the keys we establish so that the protocol could be extended if necessary.

scottc

I like Johan and bkolobara's version. To match what is here: https://www.stellar.org/developers/guides/concepts/transactions.html

I might append to it like this:

type : 'invoice|transaction',
asset_code: 'USD',
asset_issuer: 'address of issuer',
amount: '3.1415',
destination: 'seller*seller.com',
memo_text: '123849 on 15 Jul 2016',
memo_id: '12487302'
memo_hash: 'gadsfe476ade...'
memo_return: ''
message: 'Thank you for shopping at seller.com'
tx_envelope: 'transaction envelope'

So an example of an invoice could be

type : 'invoice',
asset_code: 'USD',
asset_issuer: 'address of issuer',
amount: '3.1415',
destination: 'seller*seller.com',
memo_text: '123849 on 15 Jul 2016',
memo_id: '12487302'
memo_hash: 'gadsfe476ade...'
memo_return: ''
message: 'Thank you for shopping at seller.com'

And the reverse (sending the signed transaction back, if desired) could be as simple as

type : 'transaction',
tx_envelope: 'transaction envelope'

although it might be more useful to have it sent back as

type : 'transaction',
tx_envelope: 'transaction envelope'
memo_text: '123849 on 15 Jul 2016',
memo_id: '12487302'
memo_hash: 'gadsfe476ade...'
memo_return: ''

As a URI, these could look something like

Invoice:

stellar://example.com/invoice?asset_code=USD&asset_issuer='address of issuer'&amount=3.1415&destination=seller*seller.com&memo_text=123849%20on%2015%20Jul%202016&memo_id=12487302&memo_hash=gadsfe476ade...&message=Thank%20you%20for%20shopping%20at%20seller.com

(QR Code for the invoice)

https://api.qrserver.com/v1/create-qr-code/?data=stellar%3A%2F%2Fexample.com%2Finvoice%3Fasset_code%3DUSD%26asset_issuer%3D%27address+of+issuer%27%26amount%3D3.1415%26destination%3Dseller*seller.com%26memo_text%3D123849%2520on%252015%2520Jul%25202016%26memo_id%3D12487302%26memo_hash%3Dgadsfe476ade...%26message%3DThank%2520you%2520for%2520shopping%2520at%2520seller.com&size=220x220&margin=0

Signed Tx:

stellar://example.com/transaction?tx_envelope=transaction_envelope&memo_text=123849%20on%2015%20Jul%202016&memo_id=12487302&memo_hash=gadsfe476ade...

QR Code for the transaction:

https://api.qrserver.com/v1/create-qr-code/?data=stellar%3A%2F%2Fexample.com%2Ftransaction%3Ftx_envelope%3Dtransaction_envelope%26memo_text%3D123849%2520on%252015%2520Jul%25202016%26memo_id%3D12487302%26memo_hash%3Dgadsfe476ade...&size=220x220&margin=0

Anyone know off the top of their heads how big a transaction that is ready to be sent to horizon is? Can this be fit in a qr code?

dzham

scottc Anyone know off the top of their heads how big a transaction that is ready to be sent to horizon is? Can this be fit in a qr code?

You've got up to 100 operations and up to 20 signatures, so they can be big. Your median transaction won't have any problems fitting in a QR code though.

dzham

I'll write something up later, but for now, this was my proposal for URI's back in the old day
https://github.com/pollen23/stellar-uri

sacarlson

What I learned about QR-code that I didn't expect is it is also dependent on how good your camera is on your phone or like on my computer I use my crapy webcam 640 X 480 resolution cam. I wasn't even able to read the Centaurus seed format that embeds both the public and secrete seed within the qr-code for it's export key format. I found if I just have the secrete seed in the qr-code that it requires less resolution for my camera to read it and I can then import accounts from a crapy camera with just that. so just keep that in mind with qr-code. put as little in it as possible. Maybe future Qr-code formats and better decoder libs won't have this problem, I'm not sure.

sacarlson

My thoughts of what I think the QR-code format should be. I now already have a working system on my_wallet for sending invoices, sending transactions or even sending entire accounts with just a URL link to my_wallet. The links can also be used to setup more complex functions on my_wallet including setup of trades between assets and changes to account settings and sending of multi-signature request. These links can also be converted into qr-code and read from any presently written qr-code reader on any android and or iphone that normally parse URL links from qr-code. I propose that we add one more feature to my present method. That being we create what I will now call a “auto wallet forwarder” or “auto wallet redirector” . The auto wallet forwarder will be setup as a trusted URL that all it does is forwards a qr-code link or link sent by other methods to the users desired trusted wallet URL point. The wallet forwarder is a one time setup website for the user. If the user has never been there the forwarder will ask the user what the URL of his trusted wallet is. It will then store this users URL address in this users browser localstorage. So the next time the user visits this wallet forwarder. The forwarder will see what the user wants and just forward him directly to his preset trusted wallet URL. The URL wallet points can also function as websocket push points that will not be wallets but only push the function data at the end of the URL to possible remote android or other devices that will then signal the user with an alarm on his phone that a transaction to sign has been received to be accepted or denied or just ignored. In the event the qr-code is being read directly from a wallet then the URL link part of the qr-code can be ignored and the wallet can process the function directly depending on weather the wallet has the features present in the function packet. So I'm totally up for any changes you all see are needed on my present json function format I now have write and is tested and working, but I think these are the type of functions we would want to have. We should have at least the minimal functions standardized with sending remittance and sending invoices and sending full account seeds (encrypted or not). Other functions will be custom to some wallets and not needed in all. Adding standardized functionality like this will make it easy for a web stores to send a bill or even person to person transaction to take place using qr-code or any other methods of sending URL links via chat or email with any wallet or no wallet at all. example invoice link: http://sacarlson.github.io/my_wallet/index.html?json=%7B%22amount%22:%225.24%22,%22asset%22:%22native%22,%22destination%22:%22GAX4CUJEOUA27MDHTLSQCFRGQPEXCC6GMO2P2TZCG7IEBZIEGPOD6HKF%22,%22memo%22:%22test%20invoice%22%7D The wallet I now speak of is my_wallet with the published source code point being: https://github.com/sacarlson/sacarlson.github.io/tree/master/my_wallet and can be run from: https://sacarlson.github.io/my_wallet/

sacarlson

Example qr-code invoice image: https://drive.google.com/file/d/0B4pSHwxHXW9SVGZEbFNibUJuWTg/view?usp=sharing of this: http://sacarlson.github.io/my_wallet/index.html?json=%7B%22amount%22:%225.24%22,%22asset%22:%22native%22,%22destination%22:%22GAX4CUJEOUA27MDHTLSQCFRGQPEXCC6GMO2P2TZCG7IEBZIEGPOD6HKF%22,%22memo%22:%22test%20invoice%22%7D best you already see how many cameras can read this big of a qr-code before you continue any other ideas. Also remember I'm reading it from a big computer screen. As you try to read smaller android phone screens you again get more limited data sizes again depending on the quality of the camera on the other phone.

sacarlson

https://play.google.com/store/apps/details?id=io.futuretense.stargazer.wallet has also documented his method of qr-code that we now can use as another example method and features to help decide on a standard.
One thing we should think about to start is at least add a version code in each that would be unique for each so that until a standard is determined some code would be able to determine what format it is and use it in any case.

sacarlson

I've now created and published a prototype of the Auto Wallet Forwarder that can be used as an example of how we can have a set URL address that is setup by the user on a first time visit to the site that auto forwards to the users desired web app URL on future visits. It can also be later changed with an added setup link that is explained here: https://github.com/sacarlson/sacarlson.github.io/tree/master/wallet_forwarder. it's now setup and running on github at: http://sacarlson.github.io/wallet_forwarder/forwarder.html for you to try out. With something like this site embedded in the QR-code and or URL links, a vendor doesn't have to worry what the user prefers as his wallet app url. With other non web apps they will just ignore the added url at the lead of the json coded package. Just one single universal URL will handle any and all wallet URL's using the preferences of the user and still remain compatible with non web apps as well. The code is viewable here: https://github.com/sacarlson/sacarlson.github.io/blob/master/wallet_forwarder/forwarder.html.

sacarlson

I've added some new keys to what we already have setup in my_wallet that adds trading setup params from a URL link that is now already setup and being used in our Stellar Asset Price History table http://www.funtracker.site/chart/index.html that allows setting up a trade with any listed asset with just a click. The added keys at preset (open to change) are:
selling_asset_code
selling_asset_issuer
selling_amount
selling_price
buying_asset_code
buying_asset_issuer

An example link with the params being used:
http://sacarlson.github.io/wallet_forwarder/forwarder.html?json=%7B%22selling_asset_code%22:%22USD%22,%22selling_asset_issuer%22:%22GBEK5BFCXBYPZ5JAP2XUWM637PYBQNTL5Y2MVZYV2NBRH6OYS4HCWTWO%22,%22selling_price%22:%221.34%22,%22buying_asset_code%22:%22FUNT%22,%22buing_asset_issuer%22:%22GBEK5BFCXBYPZ5JAP2XUWM637PYBQNTL5Y2MVZYV2NBRH6OYS4HCWTWO%22,%22selling_amount%22:%221.123%22%7D

This format is also compatible with the Auto Wallet Forwarder above that takes the end ?json= contents and sends it on to the users preferred wallet URL to transact trades.

At present my_wallet will also accept federation address in place of a public stellar address for issuers or any other address. Also if no issuer address is provided, the default value setup in my_wallet will be used. I have some thoughts as to also accept an abbreviated stellar issuer address with just 5 of the lead letters with 3 added ... at the end indicating abbreviation as I presently use in other parts of my_wallet but this is not implemented yet. This will make the URL shorter and easier for humans to read and smaller to allow being put into a QR-code compatible for low resolution cameras.

dzham

sacarlson
Custom URI protocols

https://developer.mozilla.org/en/docs/Web-based_protocol_handlers
https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler

sacarlson

@dzham the web-based_protocal_handlers link looks to be a better solution to replace the page forwarder that I suggested. I've never seen that used before on any other sites, looks cool. Anyone want to write that method into an example wallet for our case? The key param values that we pass look like they would still work in any case.

dzham

I'll just add the latest revision of what I'm using in Stargazer, to get the discussion going again

https://github.com/johansten/stargazer/blob/master/docs/qr-codes.md

dzham

After trying out metamask a couple of days I got inspired and started thinking about making Stargazer desktop into a browser extension instead. That would've been a perfect place to start trying out a protocol handler..

Of course, it turns out than protocol handlers in navigator.registerProtocolHandler() are restricted in what protocols they support.

sacarlson

I've gone and supported dzham's stargazer format and added a optional twist to it that makes the data needed to be read as much or more than 1/3 smaller in size that is now setup and working on the OpenCart Stellar Payment Plugin.

A quick view to give you a slight idea of V2.1 and V3.0 formats that uses callback to pull the detailed info from the wire:
{"tx_tag":"123", "callback":"https://callback.website.com","ver":"3.0"} .
With the version determining what the call back is expecting back in return. this provide overlaping evolution of format over time as well as big reduction is qr-code data size.

I've now documented my presently supported methods used in the OpenCart Stellar Payment Plugin seen here: https://github.com/sacarlson/OpenCart_stellar_plugin. At checkout there is now 5 options of QR-code to choose from so at this point you can compare and test all the formats we now have. I personally prefer what I call V2.1 for non-escrow formated transactions at this point. It is a derivative of Stargazer format with callback fetch of data instead of direct feed. Then there is my new V3.0 that is a branch of Stargazer formated json data with added data needed for escrow contract setup, but again captured using the callback method developed in V2.1.

We now have a working demo of the OpenCart store running that you can all check out and play with it with fake testnet money at: https://www.funtracker.site/test_store/ with a funded wallet with play money to use to test it here: https://wallet.funtracker.site/?json=%7B%22seed%22:%22SAYRX7D4XXTMBKNGDG7TV2WLVJ2UEJPNGS3JQTDADX4J4X66XAMVDGHT%22%7D ok have fun

bnogal

It is an old topic, but i would like write about.
I also end up supporing the stargaze wallet QR protocol in my wallet, but i really think we should really agree a stantard.

If it is a JSON format, it should be in my opinion plain, without multiple levels, each key will represent what the data is. The receiver should understand what is in and what to do with that. If you want that the QR decides what the reader should do, it should be added as a value and not depend on the structure.
An example of this problem is that stargaze can only add a user from contact QR code but not from a receive QR code

As it will be plain json without multilevels, it could be easy converted to a URI format.
NFC could use this same URI format

QR codes are very sensitive to the size, so we could derive from this plain JSON format a "binary" encoding. Keys could be converter to a byte code, 56 byte address could be encoded in base32 taking 35 bytes.
Of course this binary format should be included in the official SDKs so we all use the same language.

Other option is to be lazy and apply zlib to the json.

umbrel

Yeah, I've been thinking about it for a while and agree with every word:

bnogal If it is a JSON format, it should be in my opinion plain, without multiple levels, each key will represent what the data is. The receiver should understand what is in and what to do with that. If you want that the QR decides what the reader should do, it should be added as a value and not depend on the structure.
An example of this problem is that stargaze can only add a user from contact QR code but not from a receive QR code

I don't see the point of embedding binary data into QR code, I'd rather keep the same format as a normal link.
I'm not ready yet to put it up as a proposal, but what do you think of following format for links and QR codes:

stellar:<network>/?action=<action>&... other params

Action is not mandatory and serves rather a suggestion purpose. User should be able to change the action in his wallet. For example he may choose to import dest account as contact from payment request link.

Actions:

trust
payment
offer
merge
data
challenge
import (account or account + seed)
options

Other parameters vary from action to action and can be:

account, examples: umbre1*papayame.com or GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74
memo_type (id, text, hash, etc)
memo (124356)
fee (0.00001)
time_bounds (1501763675-1501763676)
amount (100 or 10/USD/GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74)
selling, buying same as amount
rate (0.0000567), offer_id (12312)
etc

Examples:

stellar:7ac33997?action=trust&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&asset=USD

stellar:7ac33997?action=trust&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&asset=100/BTC

stellar:7ac33997?action=import&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74

stellar:7ac33997?action=import&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&secret= SA5NG2H2RKZ6NAZTNM3Q6PL47OXGAF4T3ISAN5MLLQV6YXLGS6BWPN6S

stellar:7ac33997?action=merge&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74

stellar:7ac33997?action=challenge&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&msg=Sign%20me&url=http%3A%2F%2Fsubmit.me%2Fhere%0D%0A

stellar:7ac33997?action=payment&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&amount=100/USD/GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74

stellar:7ac33997?action=offer&selling=100/USD/GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&buying=50000&offer_id=12312321

stellar:7ac33997?action=options&inflation=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74

stellar:7ac33997?action=data&account=GAIV6HCKHTVRTELHGYMZ6HHVI6RTLS43T25VKOJ7EIJNPMCQRXZDAX74&data=key:value

dzham

We're already using XDR in the binary blobs that go to/from the network, so using that could make sense

bnogal

Well, we can describe that binary format in XDR, looks appropriate. I invite anyone to try to use any XDR code generator on C/C++ that actually is valid for the normal NDK of android ( i ended up writing the code myself) but it is still nice to describe the format.

We just need to agree the "keys" for the fields (we can use the ones appearing in the json responses), and the map of these keys to a enum. It should be enough with an enum with byte underlaying type.

sacarlson

bnogal we really don't need to worry so much about size if we use the callback method to pull the real data (json) over wire. The callback method is also flat (not multi level json) so is also compatible with URL links. So I don't think reformating the keys to save a few bytes is a good idea. Adding too much complication for little gain. Also in the present 56 byte key format you can also decode the qr-code and cut parts out to use the values manually if needed.

But I think your right, all we really need to start is to agree on what the key fields are. Enum also sounds good that will also save some space too.

I'm also for the flat not multi level json format for there are times when callback pull method won't work in the field for p2p (person 2 person) applications that is already flat. Callback will only work from user to a server. In this situation for p2p like phone to phone, multi level json might be difficult to convert to a URL link if needed. I think it may also be adding more unneeded valuable bytes to the QR-code.