Release of Daml 2.6.0

 Daml 2.6.0 release candidate has been released. You can install it using:

daml install 2.6.0

Summary

Release 2.6.0 brings a wide range of improvements to identity provider integration, observability, and Daml Finance, as well as new event querying APIs in early access.  Some security enhancements are also included.

• Identity Provider Config Service makes it possible for participant node administrators to set up and manage additional identity providers at runtime.
• New “Golden Signals” and key metrics have been added to support the monitoring of Daml according to industry best practices. They are now GA.
  
  • As part of this effort, the metrics exposed by Daml and Canton are improved and consolidated further.
• Daml Finance continues to expand its capabilities. For example, it added Interest Rate Swap sample trades to test several scenarios.
• Daml now supports Java 17.
• The Active Contract Service can now serve ACS snapshots at user provided offsets.
• An early access event query service now enables retrieving create and archive events associated with a contract id or contract key.

Impact and Migration

• Canton protocol version 2 is no longer supported. It had a bug that was fixed in Daml 2.3 and protocol version 3, and protocol version 2 was deprecated at that time. If you have a domain running protocol version 2, please upgrade to protocol version 3 using Daml 2.3-2.5 before moving to Daml 2.6+. Note that this is a rare exception to our backwards compatibility guarantees made only because of the severity of the bug in protocol version 2.
• The configuration parameters for enabling early access versions of Canton Protocol and Daml-LF have changed. These options are explicitly not for production use, but if you run development environments to evaluate early access features, you’ll need to adapt your configs.
  • In your Besu node configuration, the RPC HTTP API WEB3 now needs to be enabled ("--rpc-http-api=WEB3,...").
  • Canton’s support for Besu sequencers has been moved from Besu version 21.10 to 22.1. It has also been made safer by checking some required configurations:
  • Canton checks the version conditions and the sequencer node will throw a BESU_VERSION_MISMATCH error if they are not met.
  • We recommend switching to QBFT consensus.
• As part of the metrics improvements, some metrics were renamed. Please see details below for old and new metrics names.
• All other metrics reporters except Prometheus were deprecated.
• A bug fix was made in Daml Script that required a structural change in handling the try catch block. Please recompile any dependencies that contain script expressions with a try catch block that are using an older SDK version.

What’s new

Introduction of the Identity Provider Config Service

Background

Daml’s APIs use Json Web Token (JWT)-based authentication. The tokens are issued by an external identity provider (IDP).

The new Identity Provider Config Service makes it possible for participant node administrators to set up and manage additional identity providers at runtime. This allows using access tokens from identity providers unknown at deployment time. When an identity provider is configured, independent IDP administrators can manage their own set of parties and users.

Such parties and users have a matching identity_provider_id defined and are inaccessible to administrators from other identity providers. A user will only be authenticated if the corresponding JWT token is issued by the appropriate identity provider. Users and parties without identity_provider_id defined are assumed to be using the default identity provider, which is configured statically when the participant node is deployed..

Specific Changes

• We have added the ability to create an identity provider config via the CreateIdentityProviderConfig request.
• We have added the ability to delete an identity provider config via the DeleteIdentityProviderConfig request.
• We have added the ability to update an identity provider config via the UpdateIdentityProviderConfig request.
• We have added the ability to list all identity provider configs via the ListIdentityProviderConfigs request.
• We have added the ability to request a specific identity provider config via the GetIdentityProviderConfig request.  
• We have added identity_provider_id field to a number of requests of the Party Management Service.
  
  
  • AllocatePartyRequest, allowing a party to be created within the scope of a particular identity provider
  • GetPartiesRequest, UpdatePartyDetails, in these requests the identity_provider_id field in the request must match the value assigned at creation time
  • ListKnownPartiesRequest, allowing to narrow the query to within the scope of a given participant identity provider
    
    
• We have added identity_provider_id field to the PartyDetails record returned in the AllocatePartyResponse, UpdatePartyDetailsResponse, GetPartiesResponse and ListKnownPartiesResponse messages used in the Party Management Service.
• We have added a new access right type of IdentityProviderAdmin to the User Management Service. It allows designating a particular user as responsible for administering the identity provider that the user is assigned to. The designated user is able to manage other users and parties that are also assigned to the same identity provider.
• We have added an optional identity_provider_id field to the User message within the User Management Service. It signifies the id of the identity provider configured by Identity Provider Config. This message is part of several requests
  
  
  • CreateUserRequest, allowing a user to be created within the scope of a particular identity provider
  • GetUserRequest, UpdateUserRequest, DeleteUserRequest, GrantUserRightsRequest, RevokeUserRightsRequest, ListUserRightsRequest, in these requests the identity_provider_id field in the request must match the value assigned at creation time
  • ListUsersRequest, allowing to narrow the query to within the scope of a given participant identity provider
    
    
• We have added identity_provider_id field to the User record returned in the CreateUserResponse, GetUserResponse, UpdateUserResponse and ListUsersResponse messages used in the User Management Service.

Impact and Migration

This is a purely additive change.

Observability: General Availability of Golden Signals and Key Metrics

Background

Best practice for monitoring a microservices application is an approach known as the Golden Signals or the RED method where metrics monitoring determines whether the application is healthy and, if unhealthy, which service or endpoint (i.e. API) is the root cause of the issue. This release is the General Availability of the Golden Signals for HTTP and gRPC endpoints. It also includes other key metrics. 

Specific Changes

The Golden Signal metrics for each HTTP and gRPC API are available:

• Input request rate as a counter.
• Error rate as a counter. Discussed in the subsection below.
• Latency as a histogram, i.e. the time to process a request.
• Size of the payload as a counter, following the Apache HTTP precedent. 

The labels that accompany each metric can be used for filtering or aggregation.  The instrumentation labels added to each HTTP API metric are:

• http_verb: HTTP verb (e.g., GET, POST).
• http_status: Status code (e.g., 200, 401, 403, 504).
• host: Host identifier. 
• daml_version: Daml release number.
• service: A string to identify what Daml service or Canton component is running in this process (e.g., participant, domain, json_api, etc.). In case several Canton components run in a single process, domain is reported.
• path: The request made to the endpoint (e.g., /v1/create, /v1/exercise).

The gRPC protocol is layered on top of HTTP/2, hence we include certain labels such as the daml_version and service from the above section. The labels that are added by default to each gRPC API metric are:

• canton_version: Canton protocol version. 
• grpc_code: Human readable status code for gRPC (e.g. OK, CANCELLED, DEADLINE_EXCEEDED).
• The type of the client/server gRPC request, under the labels grpc_client_type and grpc_server_type.
• The protobuf package and service names, under the labels grpc_service_name and grpc_method_name.

Other key metrics that are monitored are:

• A binary gauge signaling whether a node is active or passive, for identifying which node is the active node.
• A binary gauge to detect when pruning is going on.
• Each participant node measures the count of the inflight (dirty) requests so the user can see if maxDirtyRequests is close to being hit.  The metrics are:  canton_dirty_requests and canton_max_dirty_requests.
• Each participant node records the distribution of events (updates) received by the participant and allows drill-down by event type (package upload, party creation, transaction), status (success or failure), participant ID and application ID (if available).
   
  • For rejected commands, in addition to the above, drill-down by gRPC status code is also possible.
  • This metric provides insight into whether a particular Daml application is working as expected.
    
    
• A new set of metrics for monitoring usage of JVM execution services used by Daml components.

The updated metrics documentation now provides a categorization as: Latency, Traffic, Errors, Saturation or Debug. As a general rule users are advised to avoid monitoring the Debug metrics as they are considered internal and may disappear or change meaning in subsequent versions.

All other metrics reporters except Prometheus were deprecated (see full list here).

Impact and Migration

The metrics below should instead use the metrics that are now available in common gRPC endpoint metrics: 

The metrics below should instead use the metrics that are now available in common JVM execution service metrics:

• daml.index.db.threadpool.connection.indexer.*
• daml.index.db.threadpool.connection.api_server.*
• daml.lapi.threadpool.in_memory_fan_out.*
• daml.lapi.threadpool.index_bypass_prepare_updates.*
• daml.lapi.threadpool.index_bypass_update_in_memory_state.*
• daml.parallel_indexer.batching.executor.*
• daml.parallel_indexer.inputmapping.executor.*

The following mapping should be used:

For example, to migrate the metric daml.index.db.threadpool.connection.indexer.completed you would instead reference daml_executor_runtime_completed{name=daml.index.db.threadpool.connection.indexer} in your PromQL query.

The following metrics were removed because they were intended for development only and not for public use:

• daml_index_db_get_flat_transactions*
• daml_index_db_get_transaction_trees*

Daml Finance Enhancements

Background

Daml Finance has gained new functionality for financial modeling.

Specific Changes

• New early-access (alpha) packages: Daml.Finance.Instrument.Option and Daml.Finance.Interface.Instrument.Option
  
  
  • Options, as some of the most common financial derivatives, are now represented by a specific packagemaking it easier to implement them (rather than using the generic instrument type). Currently, cash settled European call and put options are supported.
    
    
• FpmL Swaps:
  
  • Financial products Markup Language (FpML) is the open-source XML standard for electronic dealing and processing of OTC derivatives and is supported by Daml Finance. These changes introduce Interest Rate Swap sample trades to test several scenarios. The detailed specification of these scenarios can be found here. 

Impact and Migration

List of packages declared stable, early access, deprecated in 2.6.0 can be found here.

Active Contract Sets at user provided offsets

Background

The Active Contract Service is designed to allow client applications to bootstrap faster by providing a recent state snapshot. Updates to that snapshot can then be streamed through the transaction service.

Until now, the offset at which the snapshot is provided was given by the participant node, with the only constraint being that it is “recent”. That means that in order to get a historic view of the ledger state a client application had to either stream from the ledger beginning or apply transaction effects backwards to the recent snapshot.

With this addition, client applications can request state snapshots for any offsets after the last pruning offset, which means it is much easier to recover historic states.

Specific Changes

Impact and Migration

This is a purely additive change.

[Early Access] Introduction of the Event Query Service

Background

Specific Changes

• We have added the ability to retrieve contract events by contract id via the GetEventsByContractId request. 
• We have added the ability to retrieve contract events by contract key via the GetEventsByContractKey request. 

Impact and Migration

This is a purely additive change.

Minor Improvements

• Java 17 support was added.
  
• An extension was made so that multi-value audiences for the JWT tokens are accepted. This brings the implementation in line with the RFC7519 section 4.1.3 which stipulates that the server should accept tokens when it is able to identify itself with at least one of the values contained in an array of audiences. This is only relevant for the audience-based token format. As an example, a token of the following form will now be accepted:
  

• Daml-LF 1.15 is now the default for new Daml projects. If you want to continue using an older version, set the build-option --target=1.14 or similar in your daml.yaml file.
• The performance of the transaction service has been improved.
• We have extended the Metering Report Service by adding a JSON schema that defines the format of the reports in GetMeteringReportResponse.
• We have extended the Transaction Service by adding a new GetLatestPrunedOffsets request. It allows querying for current pruning offsets.
• Daml Finance - When specifying swap schedules, it is now possible to define a term (T) period which corresponds to one period covering the full term of the instrument. This change is incorporated in the Daml.Finance.Util 2.0.0 and Daml.Finance.Interface.Types.Date 2.0.0 packages.
• Canton protocol version 2 is no longer supported. It had a bug that was fixed in Daml 2.3 and protocol version 3, and protocol version 2 was deprecated at that time. If you have a domain running protocol version 2, please upgrade to protocol version 3 using Daml 2.3-2.5 before moving to Daml 2.6+. Note that this is a rare exception to our backwards compatibility guarantees made only because of the severity of the bug in protocol version 2.
• The following configuration options enable the development/experimental Canton protocol and Daml LF versions.
  
  NOTE: They are not to be used for production, because we do not provide support nor data continuity guarantees for the experimental versions!
  
  The flag will-corrupt-your-system-dev-version-support under canton.domains.<domain>.init.domain-parameters has been removed. Now setting protocol-version to an unstable version (i.e. dev) requires you to explicitly enable canton.parameters.non-standard-config = yes.
  
  The flags will-corrupt-your-system-dev-version-support and unsafe-enable-daml-lf-dev-version under canton.participants.<participant>.parameters have been removed. Now use the new flag canton.participants.<participant>.parameters.dev-version-support to both enable participant connectivity to unstable protocol version domains and to turn on support for unsafe Daml-LF dev versions. Using this option requires you to explicitly enable canton.parameters.non-standard-config = yes.
  
  Refer to the documentation for more details.
• We have improved the rate limiter algorithm that is used by the participant and the sequencer by introducing a new parameter named maxBurstFactor. As a result, the steady-state rate limitation is enforced whenever the burst exceeds maxBurstFactor * maxRate commands above the maxRate. This will reduce the likeliness of undesired backpressure rejects caused by small and temporary bursts.
  
  On the participant node, this factor is part of the normal resource management console commands: resources.set_resource_limits.
  
  On the sequencer node, this factor can be configured using the configuration parameter sequencer.parameters.max-burst-factor = 0.5 (default).
• Canton’s health server's active check is now supported on all nodes.
  
  Active health check configuration field was renamed from canton.monitoring.health.check.participant to canton.monitoring.health.check.node. This is a backwards compatible change.
• A gRPC health checking service is now available in Beta to be configured on each Canton node. This health checking service can be used in Kubernetes deployments for readiness and liveness probes. Refer to the documentation gRPC Health Check.
• Canton’s support for Besu sequencers has been moved from Besu version 21.10 to 22.1. It has also been made safer by checking some required configurations:
  
  • In your Besu node configuration, the RPC HTTP API WEB3 now needs to be enabled ("--rpc-http-api=WEB3,...").
  • Canton checks the version conditions and the sequencer node will throw a BESU_VERSION_MISMATCH error if they are not met.
  • We recommend switching to QBFT consensus.
• We added the ability to run pruning automatically according to a configurable schedule and to track pruning progess via the max-event-age metrics. Refer to the documentation overview and details.
• The replication.set_passive command is now also supported on a remote participant setup.
• Two new error messages have been added: UNKNOWN_SUBMITTERS and SUBMITTERS_NOT_ACTIVE mimic the already existing UNKNOWN_INFORMEES and INFORMEES_NOT_ACTIVE messages, respectively reporting if any submitter is not known at all and if no domain can be found where all submitters are hosted and active.
• Performance improvement by enabling the use of epoll native transport when running on Linux for x86_64, ARM64 and S390_64. In case you want to avoid using the native transport, configure it with the -Dio.netty.transport.noNative=true system property setting.
• The Daml compiler now provides Daml-specific hints for various type errors relating to templates, interfaces, choices, and methods. The original error is preserved and provided for further context.
• Added documentation on how to generate coverage reports using daml test. Refer to the updated tutorial. Improved coverage and counting of interface choices.
• An interface can now list which interfaces it requires. This can improve type information when casting. See the reference documentation for interfaces.
• Scripts running for more than 60s are now interrupted by default, to avoid background processes never being killed. 
• Daml Triggers now batch messages by default. Previously batching occurred during backpressure. 
• Made various Trigger logging improvements:
  
  
  • Trigger metrics logging is INFO by default
  • Trigger TRACE logging now logs the full trigger state and the full message batch
  • The context of a trigger now includes rule evaluation metric information at INFO level and in a .trigger.metrics blocks. The evaluation metrics include statistics such as the number of submissions a rule emits and the time a rule evaluation takes.
    
    
• Triggers now have configurable hard limits. When a hard limit is exceeded the trigger service will stop the trigger instance. The following hard limits have been introduced:
  
  
  • Active contract service (ACS) size has an upper bound defined. This is enabled and set to 20,000 active contracts by default. Daml user code may check maxActiveContracts for warnings.
  • In-flight commands have an upper limit enabled and set to 10,000 by default. Daml user code may check maxInFlightCommands for warnings.
  • Period of time a rule evaluation may take, disabled and set to 7.5s by default.
  • Period of time it takes for a rule step to be performed, disabled and set to 75ms by default.

Security and Bug fixes

• Some remaining security checks to ensure that a ledger is well-authorized according to the ledger model's authorization have been added, and the known limitations documentation has been edited accordingly.
• Fixed Timer resource teardown. This avoids producing confusing error log messages pointing to problems with the database connection pool at application shut down. See https://github.com/digital-asset/daml/pull/16289 
• Fixed the scoping of try and catch in Daml Script. Previously, if one of the actions after a catch block was a bare error/fail/abort , then all actions after the try-catch block were executed in both the try and catch when running the code. See https://github.com/digital-asset/daml/pull/16190 
  Note:
  Due to this structural change in handling the try catch block in Daml Script, please recompile any dependencies that contain script expressions with a try catch block that are using an older SDK version.
• https://nvd.nist.gov/vuln/detail/CVE-2022-26520 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-26520)
  
  
  • This has been closed because we are using postgresql JDBC driver version 42.3.3.
    
    
• https://nvd.nist.gov/vuln/detail/CVE-2022-36944 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-36944)
  
  
  • Upgraded to Scala version 2.13.9 which fixes a Java deserialization issue.

• Daml Finance: fix `DateClock` implementation to avoid key uniqueness violation errors 
  
  
  • Changed the signature of `advance` and `rewind` in the `Reference.Time` interface
    
    
• Daml Finance: fix merge implementation for fungible holdings
  
  
  • Add check that the list of fungibles are non-empty
  • Add check that the account of merged fungibles matches
    
    
• Fixed issues with the KMS integration in replicated node setups.
• Fixed HA fail-over issues for domain manager and mediator.