Administration

Much of the administration of an individual node can be done via the graphical user interface provided at x-admin.your_host.com, where your-host is the public address of your psibase infrastructure node (e.g. psibase.127.0.0.1.sslip.io for local nodes). To learn more about the administration app, see the documentation on x-admin. For more complex administration requirements, psinode exposes many services and configuration options over an http interface.

Booting a network

Booting a network is only a valid operation if psinode does not yet have any chain. It can be done either with the psibase CLI tool, or by using the GUI provided by the x-admin service. Alternatively, the POST /native/push_boot endpoint can be used manually in conjunction with POST /native/push_transaction to perform a custom boot sequence.

The body of the POST /native/push_boot request contains a list of transactions that will each be executed in order in the first block. The first transaction is a special transaction known as the genesis transaction. The genesis transaction uploads the core services to the blockchain and it is not permitted to do anything else.

A typical boot sequence contains more configuration than is able to fit in a single block. To understand how such a boot sequence is accomplished, see the description of SystemService::Transact::startBoot.

Peering with others

➕ TODO: document the /native/p2p endpoint.

Node administrator services

The administrator API under /native/admin provides tools for monitoring and controlling the server. All APIs use JSON (Content-Type should be application/json). Authorization to access this API is controlled by the server's admin-authz configuration option.

MethodURLDescription
GET/native/admin/statusReturns status conditions currently affecting the server
POST/native/admin/loginReturns a bearer token that can be used to access the admin API
POST/native/admin/shutdownStops or restarts the server
GET/native/admin/peersReturns a JSON array of all the peers that the node is currently connected to
POST/native/admin/connectConnects to another node
POST/native/admin/disconnectDisconnects an existing peer connection
GET/native/admin/keysReturns a JSON array of the public keys that the server can sign for
POST/native/admin/keysCreates or imports a key pair
GET/native/admin/keys/devicesLists available cryptographic devices
POST/native/admin/keys/unlockUnlocks a cryptographic device
POST/native/admin/keys/lockLocks a cryptographic device
GET/native/admin/configReturns the current server configuration
PUT/native/admin/configSets the server configuration
GET/native/admin/perfReturns performance monitoring data
GET/native/admin/logWebsocket that provides access to live server logs

Server status

/native/admin/status returns an array of strings identifying conditions that affect the server.

StatusDescription
"slow"psinode was unable to lock its database cache in memory. This may result in reduced performance. This condition can be caused either by insufficient physical RAM on the host machine, or by a lack of permissions. In the latter case, the command sudo prlimit --memlock=-1 --pid $$ can be run before lauching psinode to increase the limits of the current shell.
"startup"psinode is still initializing. Some functionality may be unavailable.
"shutdown"psinode is shutting down. Some functionality may be unavailable.

POST to /native/admin/shutdown stops or restarts the server. The POST data should be JSON with any of the following options:

FieldTypeDescription
restartBooleanIf set to true, the server will be restarted
forceBooleanIf set to true, the server will close all connections immediately without notifying the remote endpoint. Since this includes the connection used to send the shutdown, a request with force set may not receive a response.
softBooleanApplies to restarts only. If set to true, psinode will keep the current process image.

POST to /native/admin/login returns a bearer token that can be used to access the admin API. Requires bearer tokens to be enabled by the server configuration.

FieldTypeDescription
expNumericDateThe time at which the token expires in seconds since the UNIX epoch. If not specified, the token will expire 1 hour from the current time.
modeStringShould be either "r" or "rw". If not specified, the token will have the same access rights as the client.
FieldTypeDescription
accessTokenString
expNumericDate
modeString

Peer management

/native/admin/peers lists the currently connected peers.

Each peer has the following fields:

FieldTypeDescription
idNumberA unique integer identifying the connection
endpointStringThe remote endpoint in the form host:port
urlString(optional) The peer's URL if it is known

/native/admin/connect creates a new p2p connection to another node. To set up a peer that will automatically connect whenever the server is running, use the peers config field.

FieldTypeDescription
urlStringThe remote server, e.g. "http://psibase.io/"

/native/admin/disconnect closes an existing p2p connection. Note that if the server drops below its preferred number of connections, it will attempt to establish a new connection, possibly restoring the same connection that was just disconnected.

FieldTypeDescription
idNumberThe id of the connection to close

Key ring

GET on /native/admin/keys lists the public keys that the server can sign for.

FieldTypeDescription
serviceStringThe name of the service that verifies signatures made with this key
rawDataStringA hex string containing public key information. The interpretation depends on the verify service.

POST to /native/admin/keys creates or uploads a new key pair. It returns the corresponding public key.

FieldTypeDescription
serviceStringThe name of the verify service
rawDataStringA hex string containing private key information. The interpretation depends on the verify service. If rawData is not present, the server will generate a new key.
deviceStringThe id of the cryptographic device which will hold the new key. The device must be unlocked.

GET on /native/admin/keys/devices lists available cryptographic devices that can store keys.

FieldTypeDescription
nameStringThe human readable name of the device
idStringAn opaque string that uniquely identifies the device
unlockedBooleantrue if the device is currently unlocked

POST to /native/admin/keys/unlock unlocks a cryptographic device.

FieldTypeDescription
deviceStringThe device id.
pinStringThe user pin for the device

POST to /native/admin/keys/lock locks a cryptographic device.

FieldTypeDescription
deviceStringThe device id.

Server configuration

/native/admin/config provides GET and PUT access to the server's configuration. Changes made using this API are persistent across server restarts. New versions of psibase may add fields at any time. Clients that wish to set the configuration should GET the configuration first and return unknown fields to the server unchanged.

FieldTypeDescription
p2pBooleanControls whether the server accepts incoming P2P connections.
peersArrayA list of peer URLs that the server may connect to. To manage the active connections, see peer management
autoconnectNumber or BooleanThe target number of out-going connections. If set to true, the server will try to connect to all configured peers.
producerStringThe name used to produce blocks. If it is empty or if it is not one of the currently active block producers defined by the chain, the node will not participate in block production.
pkcs11-modulesArrayPKCS #11 modules that provide access to cryptographic devices.
hostStringThe server's hostname.
listenArrayInterfaces that the server will listen on. Changes to the set of interfaces will take effect the next time the server starts.
listen[n].protocolStringOne of http, https, or local
listen[n].addressString(http or https) An IP address that refers to a local interface
listen[n].portNumber(http or https) The TCP port number
listen[n].pathString(local only) A path to a local socket
tlsObjectThe TLS context. Changes to the TLS context will take effect the next time the server starts.
tls.certificateStringThe path to the server's certificate chain in PEM format
tls.keyStringThe path to the file containing the private key for the server's certificate in PEM format
tls.trustfilesArrayA list of files containing trusted root CAs in PEM format.
servicesArrayA list of built in services. host is the virtual hostname for the service. If host ends with . the global host will be appended to it. root is a directory containing the content to be served.
adminStringControls service access to the admin API. * allows access for all services. static:* allows access for builtin services. The name of a service allows access for that service.
admin_authzArrayControls which clients are allowed to access the admin API
admin_authz[n].kindStringOne of any, loopback, ip, or bearer
admin_authz[n].modeStringOne of r or rw
admin_authz[n].addressString(ip only) The client's IP address
admin_authz[n].keyString(bearer only) The key used to verify bearer tokens
http_timeoutNumberThe timeout in microseconds before the server closes idle connections. If it is null or missing, connections do not time out.
loggersObjectA description of the destinations for log records

Example:

{
    "p2p": true,
    "peers": ["http://psibase.io/"],
    "producer": "prod",
    "pkcs11-modules": ["libsofthsm2.so"],
    "host": "127.0.0.1.sslip.io",
    "listen": [
        {
            "protocol": "http",
            "address": "0.0.0.0",
            "port": 8080
        }
    ],
    "tls": {
        "certificate": "psibase.cert",
        "key": "psibase.key",
        "trustfiles": []
    },
    "services": [
        {
            "host": "localhost",
            "root": "/usr/share/psibase/services/x-admin"
        }
    ],
    "admin": "builtin:*",
    "admin_authz": [{
        "mode": "rw",
        "kind": "bearer",
        "key": "swordfish"
    }],
    "loggers": {
        "console": {
            "type": "console",
            "filter": "Severity >= info",
            "format": "[{TimeStamp}] [{Severity}]: {Message}"
        },
        "file": {
            "type": "file",
            "filter": "Severity >= info",
            "format": "[{TimeStamp}]: {Message}",
            "filename": "psibase.log",
            "target": "psibase-%Y%m%d-%N.log",
            "rotationTime": "00:00:00Z",
            "rotationSize": 16777216,
            "maxSize": 1073741824
        },
        "syslog": {
            "type": "local",
            "filter": "Severity >= info",
            "format": "{Syslog:glibc}{Message}",
            "path": "/dev/log"
        }
    }
}

Performance monitoring

/native/admin/perf reports an assortment of performance related statistics.

FieldTypeDescription
timestampNumberThe time in microseconds since an unspecified epoch. The epoch shall not change during the lifetime of the server. Restarting the server may change the epoch.
transactionsObjectTransaction statistics
memoryObjectCategorized list of resident memory in bytes
tasksArrayPer-thread statistics

The transactions field holds transaction statistics. It does not include transactions that were only seen in blocks.

FieldTypeDescription
totalNumberThe total number of transactions received
unprocessedNumberThe current transaction queue depth
succeededNumberThe number of transactions that succeeded
failedNumberThe number of transactions that failed
skippedNumberThe number of transactions skipped. This currently means that the node flushed its queue when it was not accepting transactions.

The memory field holds a breakdown of resident memory usage.

FieldTypeDescription
databaseNumberMemory used by the database cache
codeNumberNative executable code
dataNumberStatic data
wasmMemoryNumberWASM linear memory
wasmCodeNumberMemory used to store compiled WASM modules
unclassifiedNumberEverything that doesn't fall under another category.

The tasks array holds per-thread statistics.

FieldTypeDescription
idNumberThe thread id
groupStringIdentifies the thread pool that that thread is part of
userNumberThe accumulated user time of the thread in microseconds
systemNumberThe accumulated system time of the thread in microseconds
pageFaultsNumberThe number of page faults
readNumberThe total number of bytes fetched from the storage layer by the thread
writtenNumberThe total number of bytes sent to the storage layer by the thread

Caveats: The precision of time measurements may be less than representation in microseconds might imply. Statistics that are unavailable may be reported as 0.

{
  "timestamp": "36999617055",
  "memory": {
    "database": "12914688",
    "code": "10002432",
    "wasmMemory": "7143424",
    "wasmCode": "7860224",
    "unclassified": "9269248"
  },
  "tasks": [
    {
      "id": 16367,
      "group": "chain",
      "user": "850000",
      "system": "190000",
      "pageFaults": "1",
      "read": "0",
      "written": "589824"
    },
    {
      "id": 16368,
      "group": "database",
      "user": "6370000",
      "system": "5750000",
      "pageFaults": "0",
      "read": "0",
      "written": "0"
    },
    {
      "id": 16369,
      "group": "http",
      "user": "120000",
      "system": "30000",
      "pageFaults": "0",
      "read": "0",
      "written": "0"
    }
  ],
  "transactions": {
    "unprocessed": "0",
    "total": "6",
    "failed": "2",
    "succeeded": "3",
    "skipped": "1"
  }
}

Logging

The loggers field of /native/admin/config controls the server's logging configuration.

The log configuration is a JSON object which has a field for each logger. The name of the logger is only significant to identify the logger. When the log configuration is changed, if the new configuration has a logger with the same name and type as one in the old configuration, the old logger will be updated to the new configuration without dropping or duplicating any log records.

All loggers must have the following fields:

FieldTypeDescription
typeStringThe type of the logger: "console", "file", or "local"
filterStringThe filter for the logger
formatString or ObjectDetermines the format of log messages

Additional fields are determined by the logger type.

Console logger

The console logger writes to the server's stderr. It does not use any additional configuration. There should not be more than one console logger.

File logger

The file logger writes to a named file and optionally provides log rotation and deletion. Multiple file loggers are supported as long as they do not write to the same files.

FieldTypeDescription
filenameStringThe name of the log file
targetStringThe pattern for renaming the current log file when rotating logs. If no target is specified, the log file will simply be closed and a new one opened.
rotationSizeNumberThe file size in bytes when logs will be rotated
rotationTimeStringThe time when logs are rotated. If it is a duration such as "P8H" or "P1W", the log file will be rotated based on the elapsed time since it was opened. If it is a time, such as "12:00:00Z" or "01-01T00:00:00Z", logs will be rotated at the the specified time, daily, monthly, or annually. Finally, a repeating time interval of the form R/2020-01-01T00:00:00Z/P1W (start and duration) gives precise control of the rotation schedule.
maxSizeNumberThe maximum total size of all log files.
maxFilesNumberThe maximum number of log files.
flushBooleanIf set to true every log record will be written immediately. Otherwise, log records will be buffered.

filename and target can contain patterns which will be used to generate multiple file names. The pattern should result in a unique name or old log files may be overwritten. The paths are relative to the server's root directory.

PlaceholderDescription
%NA counter that increments every time a new log file is opened
%y, %Y, %m, %d, %H, %M, %Sstrftime format for the current time

Both rotation and log deletion trigger when any condition is reached.

When log files are deleted, the oldest logs will be deleted first. All files that match the target pattern are assumed to be log files and are subject to deletion.

Example:

{
    "file-log": {
        "type": "file",
        "filter": "Severity >= info",
        "format": "[{TimeStamp}]: {Message}",
        "filename": "psibase.log",
        "target": "psibase-%Y%m%d-%N.log",
        "rotationTime": "00:00:00Z",
        "rotationSize": 16777216,
        "maxSize": 1073741824
    }
}

Local Socket Logger

The socket type local writes to a local datagram socket. Each log record is sent in a single message. With an appropriate format, it can be used to communicate with logging daemons on most unix systems.

FieldTypeDescription
pathStringThe socket path

Example:

{
    "syslog": {
        "type": "local",
        "filter": "Severity >= info",
        "format": "{Syslog:glibc}{Message}",
        "path": "/dev/log"
    }
}

Pipe logger

The pipe logger sends log messages to the stdin of a subprocess. The format MUST include any necessary framing.

FieldTypeDescription
commandStringThe command will be run as if by popen(3) with the working directory set to the server's root directory.

The command SHOULD exit after receiving EOF. If it fails to do so, a configuration change that updates the command may terminate the previous command with a signal.

Examples:

{
    "syslog": {
        "type": "pipe",
        "filter": "Severity >= info",
        "format": "{FrameDec:{Syslog:rfc5424}{Message}}",
        "command": "nc --send-only --ssl-verify --ssl-cert logcert.pem --ssl-key logkey.pem logs.psibase.io 6154"
    },
    "alert": {
        "type": "pipe",
        "filter": "Severity >= error",
        "format": "notify-send -a {Process} -i /usr/share/psibase/icons/psibase.svg '{Process} {Channel}' \"{Escape:$`\\\":{Message}}\"\n",
        "command": "/bin/sh"
    }
}

Websocket logger

/native/admin/log is a websocket endpoint that provides access to server logs as they are generated. Each message from the server contains one log record. Messages sent to the server should be JSON objects representing the desired logger configuration for the connection.

FieldTypeDescription
filterStringThe filter for this websocket. If no filter is provided, the default is to send all possible log messages.
formatString or ObjectThe format for log messages. If no format is provided, the default is JSON.

The server begins sending log messages after it receives the first logger configuration from the client. The client can change the configuration at any time. The configuration change is asynchronous, so the server will continue to send messages using the old configuration for a short period after client sends the update but before the server processes it.