Introducing the Puzzle API

An introduction to the interface of the Puzzle backend for developers.

▽Dec. 21, 2020|Bernhard Kauer

developerwork-in-progress

1. Overview

The Puzzle API allows one to interface with the Puzzle backend directly without going through the Puzzle App. In fact, this is the very same programming interface the App is using. In the following I will introduce the basic concepts. More details will be covered in follow-up articles.

• Users and Environments

Be aware that neither the backend, its interface, nor this documentation are in a final state, yet. Instead they are actively worked on and modified every other day. Any consistency between implementation and documenation should therefore not be assumed.

It is up to you, the reader, to improve this documentation by reporting back on discrepancies, omissions, errors or even hard to comprehend language, so that every other developer will benefit from your insights. If something should be additionally covered in this document let us know as well. Thank you for your support!

1.1 Recent Changes

The following parts of this document have been updated recently:

• Ancient links

2. REST API

The API design follows the REST principles. Especially it assumes:

• a client-server architecture
• each request contains all parameters it need - the server is stateless
• an uniform interface - all resources are accessible through URLs

2.1 URL Prefix

All URLs start with the same prefix:

• /pzl

This value could change when incompatible versions of the API are published in the future. However, using an accept header to indicate the supported API version is currently the prefered choice.

There is no need to hard-code any URL in a program except this prefix, since all other URLs can be extracted from the responses of the service.

2.2 HTTP/2

The API is accessed over HTTP/2 with transport encryption provided by TLS. Using HTTP/2 only enables:

• parallel streams
• server push
• newer TLS versions

Accessing the service with an older HTTP version leads to a 400 Bad Request error:

$ curl --http1.1 -s -w "%{http_code} HTTP/%{http_version}\n"  https://$HOST/pzl

{"reason": "not accessed via http/2"}
400 HTTP/1.1

2.3 Transport encryption

Transport encryption is provided by TLS. The server supports TLS v1.3. Older versions below 1.2 are unsupported:

$ curl -s -S --tls-max 1.1 https://$HOST 2>&1

curl: (35) error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available

Note that accessing the backend over TLSv1.2 is deprecated. It will be disabled when more tools are converted to the newer and more secure versions.

The server certificate is issued by Let’s Encrypt:

$ curl -v https://$HOST  2>&1 | grep issuer

*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3

The CAA DNS records points to Let’s Encrypt as well.

$ host -t CAA $HOST | cut -d' ' -f 3-

CAA record 128 issue "letsencrypt.org"

To avoid man-in-the-middle attacks, it is up to the client to correctly verify the server certificate.

2.4 Authentication

Digital signatures are employed to authenticate the client against the server. We utilize the ed25519 public-key signature scheme for this purpose since it is:

• more secure than any password scheme
• fast to calculate
• widely supported

Dozens of libraries can calculate ed25519 signatures, for instance:

• C: libsodium
• Python: PyNaCl
• Apple: CryptoKit

Since ed25519 is a public-key scheme, the private key must never leave the device it was generated on. The backend supports multiple public-keys per user account to enable access from multiple devices and for backup purposes.

A detailed description of the Puzzle authentication scheme is available in another article.

3. HTTP Requests

3.1 Methods

The following HTTP methods are used for the API:

• GET - retrieves an object or file
• HEAD - same as GET but without the body in the reply
• POST - allocates or modifies an object
• DELETE - deletes an object
• OPTIONS - retrieves choices and parameters

If another HTTP method, like PUT is called, a 501 error will be returned.

$ curl -X PUT -s -w "%{http_code}\n"  https://$HOST/pzl

501

Not all methods are implemented for all URLs. A 404 error will be returned if no handler is found for the given URL.

$ curl -s -w "%{http_code}\n"  https://$HOST/pzl

404

3.2 Headers

The following HTTP request headers are understood:

• authorization - for authenticating the request
• content-type - must be application/json for POST requests. Any value supported for attachments.
• if-none-match - for conditional GET and HEAD requests
• last-modified - the time-stamp for attachments
• prefer - for long-polling

The following HTTP headers may be send in a reply:

• cache-control - to disable caching at proxies and internal caches
• content-type - either application/json or the content-type of the attachment
• date - for the freshness of the answer
• etag - hash of the object to be used for conditional GET and HEAD requests
• from - the source of an attachment
• last-modified - the time-stamp for attachments
• location - path of the allocated object

Request parameters must be given in the body in JSON format encoded as UTF-8.

Creation

To create new objects all mandatory parameters must be given. These are the ones that cannot be changed afterwards or that are needed to continue. Adding a new main account for instance just requires a currency, as this value is fixed during the lifetime of the object.

{"currency": "EUR"}

Registering a new user needs the name of an environment where this user should be placed. Furthermore a public key to authorize further requests is required.

{"env": "sandbox", "keytype": "ed25519", "pubkey": "ugx7f8f2JIqXjlxyhZcPk_Tgkc1reR_YBrKijRzAaHg="}

Updates

Updates have only optional parameters. To change the name and the description of an account one sends something like:

{"name": "Shoes", "description": "for buying red shoes at various internet shops"}

Retrievals

Object retrievals with GET and HEAD do not need parameters and therefore have no body. Similarly the OPTIONS calls are without any parameter.

3.4 Options and Choices

Some parameters vary through time and even between users. The OPTIONS method should be used to achieve a certain degree of forward-compatibility.

One example is the currency of an account which might depend on the country the user is located in. To get a list of these choices, one may utilize the OPTIONS method before performing the creation or update call.

$ puzzle-client  -q options /pzl/s3e8/accounts

{
 "choice":{
  "currency":[
   "EUR",
   "GBP",
   "JPY",
   "PLN",
   "PP",
   "USD"
  ]
 },
 "mandatory":[
  "currency"
 ],
 "optional":[
  "description",
  "icon",
  "name"
 ]
}

The reply also lists the mandatory and optional parameters. For the user-info the result would be:

$ puzzle-client  -q options /pzl/s3e8/info

{
 "optional":[
  "description",
  "icon",
  "name"
 ]
}

There might be optional services available to the user. An OPTIONS call reveals them as well:

$ puzzle-client  -q options /pzl/s3e8

{
 "service":[]
}

3.5 Long Polling

Polling for asynchronous updates is an inefficient approach. One either wastes resources by checking to often or updates might be seen relatively late. To optimize this task, the backend supports long-polling.

An request that would normally directly answered with an empty 304 reply, now includes a small body. The headers are still send immediately, to make the client aware the server has accepted the request, but the body will be delayed upto 60 seconds. The body will be send earlier if the underlying user object is modified. If the 60 second timeout occurs the whole connection is closed. The client is expected to repeat the operation in any case, as spurious wakeups can happen.

A client needs to provide the following to enable the long-polling feature:

1. The request must address a current link.
2. Specify the etag of the current version in the if-none-match header.
3. Include a prefer header with the value wait.

If one violates the first requirement, a 400 error is returned:

$ ETAG=$(puzzle-client get /pzl/s3e8  | grep etag | cut -c 9-); puzzle-client -q get /pzl/s3e8.$ETAG -H "if-none-match: $ETAG" -H "prefer: wait"

{
 "reason":"not a current link"
}

3.6 Error Codes

The following HTTP error codes will be returned on a request.

A reply to a GET or POST request with return code 200 has as body the object in JSON format. It may also return the requested file, if an attachment was addressed.

The errors 400 and 401 return a reason string as JSON body. Examples are:

400: {"reason": "invalid JSON"}
400: {"reason": "invalid currency"}
401: {"reason": "expired"}
401: {"reason": "invalid signature"}

All errors at 400 and above might return a numeric code as well. This can be used during debugging sessions for fine-grained error matching in the backend.

401 {"code": 198, "reason": "authorization missing"}
404 {"code": 118}

4. Object Representation

Objects are retrieved in JSON format. Getting a user returns something like:

$ puzzle-client -q get  /pzl/s3e8

{
 "ancient":"/pzl/s3e8.g19jx9fU_xQSfh37WOAemMW2Evod3nWb6HRbH71d_5W2",
 "canonical":"/pzl/s3e8.g9iXKsn0EheFtGmGLFkgbJSagKqGvxCy_LhbwJ62ZJNY",
 "children":{
  "accounts":"gyAmcPE0D0cR0hpfZ62Z-V7STyvn1XUO344L4oS0FhmF",
  "auths":"g4t-iMZaQNvSAV2FqptiKlkaFxFLPKggH_nPoamd85Wx",
  "info":"g1ZB1MD5TXNnRkZJ3DKUV2UboMJJ7GswSia_GTjaJ-Ec"
 },
 "current":"/pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug",
 "event":{
  "author":"puzzle-service api <267976@n18-1>",
  "category":"user",
  "date":1608582480.928915,
  "name":"s:3e8",
  "operation":"merge",
  "subcategory":"account",
  "versions":"0-0"
 },
 "prev":"/pzl/s3e8.g19jx9fU_xQSfh37WOAemMW2Evod3nWb6HRbH71d_5W2",
 "type":"user",
 "version":1
}

The canonical and current fields are present in all objects, as they enable versioning. The type field is also mandatory. All other fields are optional. For instance the first version of an object has no prev link and a leaf in the object tree will have no children.

The children, prev and ancient links are used for navigation in the tree of objects. They are not always present. The version field can be used to sort versions of a certain object in time.

The event fields are only present in user and account objects.

4.1 Object Types

The type attribute allows to distinguish different object classes. The following types are currently defined.

4.2 Versions

All objects are versioned. Whenever any detail changes, a new version is created. The version attribute gives an monotonically increasing, albeit sparse, number that can be used to sort object versions in time.

The canonical attribute defines a path that never changes. The tag in the canonical link is a hash over the object state and unique over time. This is similar to the commit ID in a GIT repository. Requesting the canonical path returns exactly the very same object in the future, even if the object is marked as deleted in a newer version.

The prev attribute, if present, links to the previous version of this object. This enables backward iteration of the versions. The ancient attribute, if present, points to some older version of the object. This enables a binary search for a particular version. The initial version of the aforementioned user object is:

$ puzzle-client -q get --version=0 /pzl/s3e8

{
 "canonical":"/pzl/s3e8.g19jx9fU_xQSfh37WOAemMW2Evod3nWb6HRbH71d_5W2",
 "children":{
  "accounts":"g4Krr2gT1hqmVBirI5tCGY-9f3yULCRpIrJhd54Pga8G",
  "auths":"g4t-iMZaQNvSAV2FqptiKlkaFxFLPKggH_nPoamd85Wx",
  "info":"g1ZB1MD5TXNnRkZJ3DKUV2UboMJJ7GswSia_GTjaJ-Ec"
 },
 "current":"/pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug",
 "event":{
  "author":"puzzle-service api <267976@n18-1>",
  "category":"user",
  "date":1608582204.209815,
  "name":"s3e8",
  "operation":"init"
 },
 "type":"user"
}

The current attribute can be used to retrieve the newest version of an object. One can use this path together with an if-none-match header to implement efficient long-polling.

4.3 Children

An object may have children. The attribute defines a mapping between child names to their etag. By comparing two different versions of an object, one can therefore deceide whether a child was modified or not.

To retrieve the child, one combines the name of the child with the canonical or current path of the parent. The former gives a consistent view on the whole state at a certain version, whereas the later includes newer updates.

The list of accounts for instance looks like:

$ puzzle-client -q get /pzl/s3e8/accounts

{
 "canonical":"/pzl/s3e8/accounts.gyAmcPE0D0cR0hpfZ62Z-V7STyvn1XUO344L4oS0FhmF",
 "children":{
  "s:3e8":"g6osu_EUBRpNut6QQeIfGhnZ0wMl0ueFFMD2koHz8TXz"
 },
 "current":"/pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug/accounts",
 "prev":"/pzl/s3e8.g19jx9fU_xQSfh37WOAemMW2Evod3nWb6HRbH71d_5W2/accounts",
 "type":"account-list",
 "version":1
}

4.4 Events

The event attribute of an object looks like:

$ puzzle-client -q get --version=1 /pzl/s3e8 | jshon -S -e event

{
 "author": "puzzle-service api <267976@n18-1>",
 "category": "user",
 "date": 1608582480.928915,
 "name": "s:3e8",
 "operation": "merge",
 "subcategory": "account",
 "versions": "0-0"
}

One can see that at a certain date, the versions 0-0 from the account named s:3e8 were merged into this user repository. The merge was performed by a certain author.

Not all objects have event attributes. Currently, only the users and the accounts have them.

4.5 Properties

Certain objects have properties. Every main account for instance has a balance:

$ puzzle-client -q get  /pzl/s3e8/accounts/s:3e8

{
 "canonical":"/pzl/s3e8/accounts/s:3e8.g6osu_EUBRpNut6QQeIfGhnZ0wMl0ueFFMD2koHz8TXz",
 "children":{
  "incoming":"g6ZlJeFf6JEMCv6HI0lc5gRsgpyhppwkOMxV9JXTMcvZ",
  "info":"g3wcaW-G37dJBi_kSPBElBoBkwCLNKNgLuBzxKdE7K9K",
  "outgoing":"g8QOHE-Ob21ZPbS5jZax07Jjg3ARMq88ALyN3CRH8tDA"
 },
 "current":"/pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug/accounts/s:3e8",
 "event":{
  "author":"puzzle-service api <267976@n18-1>",
  "category":"account",
  "date":1608582480.926723,
  "name":"s:3e8",
  "operation":"init"
 },
 "properties":{
  "balance":{
   "available":"0",
   "fee":"0",
   "reserve":"0",
   "volume":"0"
  }
 },
 "type":"account"
}

A property can be a plain value or may have sub-fields. These are the sub-fields for the account->balance:

• available - the available money
• fee - the fee still to be paid to Puzzle
• reserve - the money reserved for ongoing transactions
• volume - lifetime volume that went to accounts of other users

5. Traces

This section contains several traces of HTTP request and responses. Note that any JSON is pretty printed, so neither the content-length fields nor the signatures are valid anymore.

5.1 GET request

GET requests need an authorization.

$ puzzle-client -d get  /pzl/s3e8/auths

> HTTP/2 200
> cache-control: no-cache
> content-length: 230
> content-type: application/json
> date: Mon, 21 Dec 2020 20:28:23 GMT
> etag: g4t-iMZaQNvSAV2FqptiKlkaFxFLPKggH_nPoamd85Wx
> 
> {
>  "canonical": "/pzl/s3e8/auths.g4t-iMZaQNvSAV2FqptiKlkaFxFLPKggH_nPoamd85Wx",
>  "children": {
>   "x1": "gwIreKIDYGsoAZW4DbAmKwZNhh-zMyXTqWmKqaQxRn-v"
>  },
>  "current": "/pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug/auths",
>  "type": "auth-list"
> }

5.2 OPTIONS request

OPTIONS need a signature as well.

$ puzzle-client -d options  /pzl/s3e8/info

> HTTP/2 200
> content-length: 46
> content-type: application/json
> date: Mon, 21 Dec 2020 20:28:23 GMT
> 
> {
>  "optional": [
>   "description",
>   "icon",
>   "name"
>  ]
> }

Without authorization a 401 is returned.

$ puzzle-client  -d -q options /pzl/s38/auths

{
 "reason":"authorization missing"
}

The only exception to this rule is the OPTIONS call to the entry point.

$ curl -X OPTIONS -s https://$HOST/pzl

{"choice": {"keytype": ["ed25519"]}, "mandatory": ["env", "keytype", "pubkey"], "optional": ["description", "icon", "name"]}

5.3 UPDATE request

Unknown parameters of an UPDATE are ignored.

$ echo "info foo=bar" | puzzle-client  -d update /pzl/s3e8 2>&1

< POST /pzl/s3e8.AK1aUhxbqNr-dL_HKGfxIFU7GHMuB6Ug/info HTTP/2
< authorization: pzl time=1608582504+60,sig=1R3SmbT6pLDEOUsseLAZTUsSR6t25RCFEYKXsPHmMayemeyUa996jZixDBBmDbHHsSIia4SEYf4TfBFAG21MDQ
< content-type: application/json
< 
< {"foo":"bar"} 

> HTTP/2 204
> date: Mon, 21 Dec 2020 20:28:24 GMT
>