Release of Daml 2.7.0

Summary

Release 2.7.0 brings a wide range of improvements to Daml Enterprise. These include: better identity provider integration, observability enhancements, extensive security integrity improvements, new financial instruments for Daml Finance, and a new feature to allow for high throughput or complex queries.

What's New

Please see the section "Installation or Download" below for installation instructions.

• Expanded distributed tracing, from Ledger API server through all Canton components which will speed up application problem diagnosis.
• Security enhancements for the Canton consensus protocol, improving integrity security.  When a malicious transaction request from a rogue participant node(s) occurs, it will now be rejected so a malicious request cannot compromise integrity of the virtual shared ledger. This better supports deploying Canton networks across mutually distrusting organizations.
• Daml Finance: new financial instruments.   They are: Callable bonds, a EuropeanPhysical instrument to model physically settled call or put options, the BarrierEuropeanCash instrument models barrier options, and a Dividend instrument model.
• Participant Query Store (early access) is a new feature that allows client applications to leverage SQL JSONB queries regarding contract creation, contract archival, and exercise activities.
• A more secure Docker image is available, which reduces the potential attack surface.
• Developers will find several productivity improvements for the Daml Studio IDE.
• KMS support has been extended to include GCP, alongside the existing AWS capability.
• Explicit contract disclosure (Early Access).   It allows sharing contract data out of band and then using this contract data within a command submission to the ledger. It enables various use cases such as invitation contracts and sharing of reference data. 
• The ledger API error codes have additional information for diagnostic purposes.
• Identity Provider Config Service updates provide easier set up and management of several additional identity providers at runtime.

Impact and migration

Canton protocol version 5 is introduced and needs to be enabled to take advantage of the extensive security integrity and distributed tracing improvements. Please follow the documentation to make the Canton protocol upgrade.

Users of Oracle Java can run Canton but it does require a different command line than with OpenJDK.   In particular, if you are using Oracle JVM and testing security provider signatures then please note that the provided Canton JAR file embeds the BouncyCastle Provider as a dependency.  To enable the Oracle JVM to verify the signature and start Canton, you need to put the “bcprov” JAR on the classpath before the Canton Standalone JAR.  For example:

java -cp bcprov-jdk15on-1.70.jar:canton-with-drivers-2.7.0-all.jar com.digitalasset.canton.CantonEnterpriseApp

What’s New

Participant Query Store (early access)

Background

The term Operational Data Store (ODS) usually refers to a database that mirrors the ledger and that allows for efficient querying. The Participant Query Store (PQS) feature acts as an ODS for a participant node. It stores contract creation, contract archival, and exercise information in a PostgreSQL database using a JSONB column format. The PostgreSQL database is queried over JDBC.

The PQS allows for complex or high-throughput queries related to contracts. It can be used for:

• Application developers to access data on the ledger, observe the evolution of data, and debug their applications;
• Business analysts to analyze ledger data and create reports;
• Support teams to debug any problems that happen in production.

Impact and migration

This is a purely additive change. It is an early-access feature so it will evolve based on user feedback.

IDP and admin enhancements

Background

The identity provider config service makes it possible for participant-node administrators to set up and manage additional identity providers at runtime. This allows for use of access tokens from identity providers unknown at deployment time. When an identity provider is configured, independent IDP administrators can manage their own set of parties and users.

In order to secure internal access to the ledger API, users can be configured and granted permission to act or read on behalf of certain parties. Users authenticate to the ledger API using JWT tokens, and valid authorization services can be configured using the respective configuration settings.

If not all users are managed by a single authorization service, a user can be assigned to so-called identity providers (IDP) by setting the appropriate IdentityProvider field when generating the user.

Specific changes

This version now extends this support and allows:

• Re-assigning the IDP of a user and a party.
• To infer an IDP from a token when processing user management and party management calls that take identity_provider_id argument.  
• To specify the expected audience field that should appear in the JWT tokens submitted by clients of the ledger API

Impact and migration

This is a purely additive change.

Daml Finance Enhancements

Daml Finance continues to expand its capabilities with additional packages for new financial instruments and usability enhancements.

New instruments

Callable bonds are similar to existing bonds instrument but they can also be redeemed by the issuer before maturity. The callability is restricted to some (or all) of the coupon dates. In other words, these bonds have a Bermudan-style embedded call option.

The EuropeanPhysical instrument models physically settles call or put options. For this type of financial derivative the holder has the right to either buy or sell the underlying security at a predetermined price (strike price) upon exercise. These options differ from cash-settled options: Since this option instrument is physically settled, it means that the underlying asset will change hands.

The BarrierEuropeanCash instrument models barrier options. They are similar to the EuropeanCash instrument described above, but also contain a barrier that is used to activate (or, alternatively, knock out) the option.

The Dividend instrument models is a physically settled, manually exercised dividend options. For reference, a dividend option gives the holder the right to choose one out of several dividend payouts, on a specific expiry date in the future.

Other functional changes

Improvements to the Contingent Claims modeling framework are:

• Add an ObserveAt observation builder to facilitate the setup of complex observables (such as interest rate compounding).
• Simplification of the election workflow: Elected sub-trees are identified by a textual tag (e.g., “EXERCISE”) rather than by the elected sub-tree itself.

You can make use of the requires keyword to enforce the interface hierarchy. This simplifies the implementation of interfaces in templates, as a number of boilerplate methods are no longer needed.

Feature documentation

The documentation has been reorganized to improve the Daml Finance learning path and provide more tutorials with additional examples.

A tutorial explaining the basic concept of Holdings was added to the Getting started section . Furthermore there is list of new tutorials with example code for:

• Lifecycling
• Payoff modeling
• Advanced topics

Introducing a new section that provides an overview of all ‘out-of-the-box’ Daml Finance instruments. Each tutorial includes detailed explanations on how to set them up for different scenarios:

• Bond extension
• Equity extension
• Option extension
• Swap extension

More Secure Docker Images

Background

In order to better support production use cases, the Docker images have been slimmed down. The new images are built based on Google’s “distroless” base images to minimize the attack surface.

Specific changes

The following products benefit from the production-ready, minimal Docker image:

• The HTTP JSON API Service
• The Trigger Service
• Daml Script
• The OAuth2 Middleware
• The Trigger Runner
• Canton Enterprise

See the table in the “Installation or Download” section for exact image and Git repository names.

The new images are scanned for security vulnerabilities on a regular basis and are automatically kept up-to-date with Google’s “distroless” base images. This means that the default tags are mutable; for stable production deployments we recommend pinning images by SHA.

These new images have been retroactively created for the latest patch in each minor release in the 2.x line.

For each image, there is a tag per supported version (e.g., 2.7.0), as well as a corresponding “debug” tag. The Docker image with the debug tag (e.g., 2.7.0-debug) contains additional tools for investigating within the image. As the name suggests, a debug tag indicates that the Docker image is meant for debugging tasks and initial setup, and should not be used in production.

Impact and migration

The HTTP JSON API Service, Trigger Service, Daml Script, Daml-on-SQL, OAuth2 Middleware, and Trigger Runner images should be drop-in replacements for our previous images. If you’re using floating tags, you may have already picked up the new images.

The Canton Enterprise image no longer includes sample code and example configuration files to minimize the attach surface. If you relied on those, the new image is not a drop-in replacement and may require additional work on your part. To avoid mistakenly picking up the new, more secure image by existing scripts, the new image is published to a different location: digitalasset-docker.jfrog.io/canton-enterprise. The old location was: digitalasset.jfrog.io/digitalasset/canton-enterprise. Moving forward, we will stop pushing to the old location. Note that, for 2.7.0 specifically, we still publish an image to the old location, but it’s is not fully backwards-compatible with 2.6.x and earlier, nor is it as secure as the ones in the new location. 

Distributed Tracing Enhancements

Background

Distributed tracing is a technique used for troubleshooting performance issues in a microservices environment like Daml Enterprise. Canton does support distributed tracing but the support on the Ledger API was only partial, limited to the command submission service and not surfaced on the ledger updates processed within the Ledger API. There have been several enhancements in this release, such as the client application being able to pass in their trace and span ID on all Ledger API commands, so that the distributed trace is contiguous.

Specific changes

The logging context has been unified across all Canton components so the causal trace is no longer broken. The key changes are:

• Clients using the command service are able to start a span in their application and pass the trace context (e.g., trace ID and span ID) on to Canton. This already worked for the submission service.
• The trace ID is now visible for the ledger updates processed with the Ledger API.
• New spans have been added for the initial command evaluation, for the command tracking, and for the indexer.
• Spans within the ApiSubmissionService, CantonSyncService, and MessageDispatcher cover almost the entirety of Daml command processing by Canton, thus giving a very fair representation of what is happening in the system at any given point.
• The tracing ID is visible in the Canton logs.

As part of the cleanup, we've unified the logging backends, such that now all log messages of the Ledger API server component appear uniform with the rest and express the trace ID.

Please note that the external JSON API service does not support OpenTelemetry distributed tracing. Therefore this feature can only be used when using the gRPC API directly.

Impact and migration

This is a purely additive change.

Integrity Security Against Malicious Participants

Background

Canton is designed for applications where there are low trust assumptions between organizations. The end goal is that a Canton domain can be deployed across mutually distrusting organizations and avoid reconciliation so that the business operations can be automated. With this release, we have reached a new milestone in adding and improving security checks.

Specific changes

When a malicious transaction request from a rogue participant node(s) occurs, it will now be rejected so it cannot compromise integrity of the virtual shared ledger. This better supports deploying Canton networks across mutually distrusting organizations.

A limitation that is planned to be removed in a future release is, although a malicious request will not succeed, it is possible that it may halt the execution of a participant node.

Impact and migration

The new security enhancements require a domain to run protocol version 5 or higher, which is the new stable default version. The domain can be upgraded to protocol version 5 using these instructions.

Additional Key Management Support

GCP KMS (early access): By default, Canton nodes will store their private keys in the database. For higher security, a key management system can be used to manage the keys. Version 2.5 introduced support for AWS KMS with envelope encryption, used to encrypt the Canton keys at rest. Envelope encryption is now extended to GCP. This is an early access feature.

Full KMS delegation: This version now introduces the ability to delegate all encryption and signing to the KMS system. Please consult the documentation for further information.

KMS logging: In order to improve the auditability of the KMS activity, all external calls to the KMS are now logged. The logs will be written to the main log file, but can be directed into a separate file using the command line argument --kms-log-file-name.

Impact and migration

This is a purely additive change.

Explicit Contract Disclosure (Early Access Labs)

Background

Explicit contract disclosure allows a party to acquire read rights during command submission over a contract of which it is neither a stakeholder nor an informee. This allows sharing contract data out of band and then reusing this contract data within a command submission to the ledger, enabling various uses such as invitation contracts and sharing of reference data. 

As a streamlined mechanism to enable read delegation, the explicit contract disclosure is introduced as an early-access feature for Canton 2.7.0, where Daml users can easily delegate contract read rights to non-stakeholders using off-ledger data distribution.

For details, check the detailed documentation.

Specific changes

A new configuration key canton.participants.<participant-name>.ledger-api.explicit-disclosure-unsafe allows enabling the use of disclosed contracts in command submission.

The Ledger API Command Protobuf schema is extended with an additional field disclosed_contracts that allows users to attach explicitly disclosed contracts to command submissions. The Ledger API Protobuf schema for CreatedEvent is extended with an additional field metadata that allows constructing disclosed contract payloads to be used in command submissions.

Impact and migration

This is a purely additive change. It is an early access feature so it will evolve based on user feedback.

Users wishing to use explicit disclosure can do so on participants running at least protocol version 4 for all contracts created starting with protocol version 4. Support for this feature is being added to existing Daml tools but this is not yet complete.

Ledger API Error Code Enhancements

The error codes and metadata of GRPC errors returned as part of failed command interpretation from the Ledger API have been updated to include more information. Previously, most errors from the Daml engine would be given as either GenericInterpretationError or InvalidArgumentInterpretationError. They now all have their own codes and encode relevant information in the GRPC Status metadata. Specific error changes are as follows:

• GenericInterpretationError (Code: DAML_INTERPRETATION_ERROR) with GRPC status FAILED_PRECONDITION is now split into:
  • DisclosedContractKeyHashingError (Code: DISCLOSED_CONTRACT_KEY_HASHING_ERROR) with GRPC status FAILED_PRECONDITION
  • UnhandledException (Code: UNHANDLED_EXCEPTION) with GRPC status FAILED_PRECONDITION
  • UserError (Code: USER_ERROR) with GRPC status FAILED_PRECONDITION
  • TemplatePreconditionViolated (Code: TEMPLATE_PRECONDITION_VIOLATED) with GRPC status INVALID_ARGUMENT
• InvalidArgumentInterpretationError (Code: DAML_INTERPRETER_INVALID_ARGUMENT) with GRPC status INVALID_ARGUMENT is now split into:
  • CreateEmptyContractKeyMaintainers (Code: CREATE_EMPTY_CONTRACT_KEY_MAINTAINERS) with GRPC status INVALID_ARGUMENT
  • FetchEmptyContractKeyMaintainers (Code: FETCH_EMPTY_CONTRACT_KEY_MAINTAINERS) with GRPC status INVALID_ARGUMENT
  • WronglyTypedContract (Code: WRONGLY_TYPED_CONTRACT) with GRPC status FAILED_PRECONDITION
  • ContractDoesNotImplementInterface (Code: CONTRACT_DOES_NOT_IMPLEMENT_INTERFACE) with GRPC status INVALID_ARGUMENT
  • ContractDoesNotImplementRequiringInterface (Code: CONTRACT_DOES_NOT_IMPLEMENT_REQUIRING_INTERFACE) with GRPC status INVALID_ARGUMENT
  • NonComparableValues (Code: NON_COMPARABLE_VALUES) with GRPC status INVALID_ARGUMENT
  • ContractIdInContractKey (Code: CONTRACT_ID_IN_CONTRACT_KEY) with GRPC status INVALID_ARGUMENT
  • ContractIdComparability (Code: CONTRACT_ID_COMPARABILITY) with GRPC status INVALID_ARGUMENT
  • DevError (Code: DEV_ERROR) with GRPC status FAILED_PRECONDITION
• The ContractKeyNotVisible error (previously encapsulated by GenericInterpretationError) is now transformed into a ContractKeyNotFound to avoid information leaking.

Genesis Block-Based Ethereum Contract Deployment

Background

The console command ethereum.deploy_sequencer_contract was used to deploy the Ethereum solidity contract that was used to interface with the sequencer node. The deployment of a sequencing contract should only happen once on the blockchain, but this command was often mistakenly included in a bootstrapping script, resulting in a contract redeployment each time bootstrapping was done.  

Specific changes

To avoid this issue, the ethereum.deploy_sequencer_contract has been removed. Contract deployment should be done involving the genesis block. Please refer to example e03-ethereum-sequencer and docs.daml.com about Ethereum Domains for how this can be achieved.

Impact and migration

In an existing production environment no changes should be needed. For environments that are not stable (e.g., a test environment that is constantly rebooted) changing to using the genesis block deployment (or a separate deployment script) is advised.

Minor Improvements

Daml Studio IDE: The Daml Studio IDE has several user experience enhancements. First, when a test script was running in the background, it could be unexpectedly canceled by a mouse cursor or hover action. The test script will now continue to run in the background. Second, the error messages have been enhanced so that Daml-specific error messages are shown first rather than the lower-level Haskell error messages. The Haskell error messages are still available for expert users. Last, common error messages will now provide information about the modules to import when making a recommendation that can benefit from a new module.

We are soliciting your input about what error messages should be prioritized first for enhancement.  Please submit your input by filling out this form.

Daml test scripts: Daml test runs can now be stored externally so they can be compared from test run to test run. Several test runs can be aggregated into a single file. This is described in the section Serializing Results Workflows, which says:

The --save-coverage and --load-coverage flags enable you to write coverage results to a file and read them back out again. Multiple coverage results from different files can be aggregated, along with new coverage results from tests, and then written back to a new file.

This enables three new kinds of coverage testing which should be especially useful to those with large test suites: Single test iteration, multiple test aggregation, and test failure recovery. 

The Flags Controlling Serialization section describes the command line options.

API documentation enhancements:  The standard library and Daml Finance API documentation has been enhanced with:

• Adding signatories and choice controllers.
• Adding interfaces and template information.
• Linking to type information is greatly improved.

Configured Participant admin user: A bespoke participant admin user can be created on startup. In order to do that, you need to add the following line to your configuration
participants.participant.ledger-api.user-management-service.additional-admin-user-id = "my-admin-id"

New remote console commands: The admin commands repair.migrate_domain and pruning.find_safe_offset are now also supported on a remote participant setup.

Changed error code returned in rotate wrapper key command: The command keys.secret.rotate_wrapper_key, which allows for the rotation of the KMS wrapper key that is being used to encrypt Canton's private keys at rest, now returns a different error code. An INVALID_WRAPPER_KEY_ID error has been replaced by an INVALID_KMS_KEY_ID error.

Repair commands: The command repair.download which allows downloading active contracts for a set of parties has a new format. Therefore, the command repair.acs.load_from_file is not able to load generated using a previous version of Canton.  This new format is simpler to parse and to validate.

Usage of PartyId in Ledger API console commands:  The commands under ledger_api.users and ledger_api.parties have been switched to use PartyId consistently instead of LfPartyId to reduce the need to convert between different party id types. However, LfPartyId can still be used in a backwards compatible manner.

Ledger API console commands for Identity Provider Config: Identity Provider Config can now be updated via Canton console using commands under  ledger_api.identity_provider_config. It is now possible to create, delete, update, list, and get details of Identity Provider Configs defined on a given participant. It is also possible to assign users and parties to IDPs on creation by specifying identityProviderId inside ledger_api.users.create and ledger_api.parties.allocate calls respectively. These assignments can be altered by using ledger_api.users.update_idp and ledger_api.parties.update_idp.

Migrate and start: Canton now supports a "migrate and start" mode, where the database schema is automatically updated when a new minor version is deployed. This mode can be enabled by setting the appropriate storage parameter: canton.X.Y.storage.parameters.migrate-and-start = yes.

Additional crypto provider support (early access):  As an early access feature, a new crypto provider can be configured (JCE) instead of the default Tink. Participants can use different crypto providers to interact on the same domain. If you upgrade your participant (configured to use the default crypto provider Tink) to 2.7 and want to use it with a domain where other participants are using JCE, then you need to rotate the keys on the participant using Tink, as otherwise the JCE participants can't encrypt the data for your participant.

CommandService simplification:  The com.daml.ledger.api.v1.CommandService service has been simplified under the hood. On the API, this implies:

• The semantics of command-service.max-tracking-timeout has changed. It now configures for how long the command service will look for the completion of an in-flight submission before timing out to the client.
• command-service.input-buffer-size config is removed as the command service input buffer is removed.
• command-service.tracker-retention-period is removed as the underlying submission tracker behavior has changed.

These last two settings/configurations were never actively supported by Canton and have therefore been removed.

Change within Sequencer configuration:  The configuration of the Sequencer Client has been updated to accommodate multiple sequencers and their endpoints. This is a change in anticipation of  the upcoming 3.0 release. To manage these connections, a new class called "SequencerConnections" has been introduced. It is now accepted as part of the Sequencer Client configuration.  The method "SequencerConnection.addConnection" has been renamed to "SequencerConnections.addEndpoints" to better reflect the fact that it modifies an endpoint for the sequencer.

Multi-domain causality tracking has been removed:  An obsolete early access feature to enable causality tracking, related to preview multi-domain, was removed. If you enabled it, you need to remove the following config lines, as they will no longer compile:
participants.participant.init.parameters.unsafe-enable-causality-tracking = true
participants.participant.parameters.enable-causality-tracking = true

Activation of development or some early access features:  In order to activate unsupported features, you now need to explicitly enable dev-version-support on the domain (in addition to the non-standard config flag). More information can be found in the documentation.

Security and Bug Fixes

The following bugs were fixed in patch releases of 2.6 and are therefore mentioned here for completeness.

• Fixed a bug that caused mTLS to not be operating correctly when interacting with a Client Authentication enabled Fabric Network. Minor.  
• Fixed Besu version string matching for custom built images. Minor.
• Fixed a bug in Fabric integration that could cause blocks to be reordered. Moderate.  
• When using the repair service of a participant, the participant may fail to reconnect to the domains affected by the repairs. Moderate.
• Crash recovery bug in ACS commitment processing. A participant may fail to reconnect to a domain and the reconnect attempt seems stuck. Critical.
• Source Description of DARs that start with a ‘.’  When uploading a DAR whose name starts with a ‘.’, the file extension will not appear anymore in the sourceDescription. Cosmetic.
• Race condition during participant node initialization. A participant that initially starts as passive but then becomes active will not resume the node initialization and therefore remain stuck. Moderate.
• Missing support for HTTP health check monitoring for domain nodes. HTTP/1 health checks which are supported on participant, mediator, sequencer and domain manager nodes were missed on embedded domain nodes. Minor.
• Resource exhaustion due to concurrent Daml package loading during catchup. The Daml engine will cache packages once they are loaded. However, any concurrent interpretation will keep on requesting the package from the database, leading to the situation where the same package is loaded many times. Moderate.
• Lack of throttling on asynchronous message processing leading to resource exhaustion.  If the node has fallen behind sufficiently and is memory constrained, the lack of throttling of the async processing (only the synchronous part is throttled) can lead to OutOfMemory exceptions that will happen periodically until the node is caught up again. Moderate.
• ACS commitment processor crash recovery deadlock due to high load. Restarting Canton after a crash under high load can lead to a deadlock in the ACS commitment processor.  Critical.
• Phase37 synchronizer can be crashed due to a malicious mediator response.  When someone sends a fake mediator result with a wrongly declared type, the participant node would crash with a ClassCastException. Minor.
• Trigger service may now be configured (at service startup) with a ledger TLS client.
• There is a corner case where pruning of data may be unavailable after a hard domain connection upgrade. This has been resolved.  Moderate.

Installation or Download

The Daml 2.7.0 SDK has been released. You can install it using the command:  
daml install 2.7.0.

The table below lists how you can download Daml Enterprise or individual components.

Note: These Docker images are designed to be minimal in size and attack surface, and are thus suitable for production use. Minimal images can sometimes make debugging difficult (e.g., no shell in the containers). For convenience, we provide “debug” versions of each of the above images, which you can access by appending “-debug” to the image tag (e.g., digitalasset-docker.jfrog.io/http-json:2.7.0-debug).