Release of Daml 2.5.0

 Daml 2.5.0 release candidate has been released. You can install it using:

daml install 2.5.0

Summary

Release 2.5.0 brings a wide range of improvements to operating Canton, system resiliency, and Daml language features.
  • Daml Finance is now stable, offering ready-made building blocks for tokenization and capital markets use cases.
  • Daml Interfaces are now stable as part of Daml-LF 1.15 and Canton protocol 4.0.0, offering more modularization and extensibility for Daml applications.
  • Canton protocol 4.0.0 brings some additional improvements in addition to interfaces. Several previously static Canton configuration parameters are now dynamic, and in some scenarios, performance is improved significantly.
  • The Java bindings for the Ledger API include several usability and safety improvements.
  • The gRPC Ledger API now protects participant nodes more effectively from overload by exerting back pressure when system resources are exhausted.
  • Integration with identity providers is more flexible and adaptable by, amongst other things, adding mutable meta-information to users in the User Management service.
  • By utilizing a KMS system (AWS KMS supported in 2.5.0), Canton nodes can be operated more securely by keeping all cryptographic keys encrypted at rest. 
  • The Canton domain manager now also supports high availability. This works in the same way as for the participant and mediator nodes.
  • The metrics exposed by Daml and Canton are improved and consolidated including updated documentation.
  • [Beta] New “Golden Signals” metrics have been added to support the monitoring of Daml according to industry best practices.

Impact and Migration

  • Daml Finance, Interfaces, and the underlying Daml-LF and and protocol versions are all GA, but need to be activated in the appropriate way:
  • To enable interfaces, the HTTP JSON API service query store database schema has changed and needs to be reset. If you are migrating from a previous version, either reset your database manually or start the HTTP JSON API with one of the options that regenerate the schema (create-only, create-if-needed-and-start, create-and-start).
  • The introduction of Canton Protocol version 4 and the move of static to dynamic configuration in protocol version 4 requires your configuration to be updated if you are upgrading an existing deployment. See detailed migration guide below.
  • In some cases, a type change in the Java bindings may lead to compile errors. If you encounter them, you may need to update type declarations.
  • The new Ledger API resource limits and their default values may lead to backpressure in some high performance environments. In that case, the default limits have to be adjusted as appropriate.
  • To accommodate the KMS feature, there have been some changes to the signatures of console commands related to uploading and downloading cryptographic keys from Canton nodes. See the detailed guide in the KMS section below.
  • The Ledger API metrics have been tidied up and a few detailed debugging metrics have been renamed. If you use metrics that no longer exist, refer to the detailed notes below for migration.

What’s new

Daml Finance is now Generally Available

Background

Developer productivity is a core value driver of Daml. To further support users building applications on their journey to production we are making Daml Finance generally available. Daml Finance is a comprehensive library to support modeling financial use cases in Daml. It allows you to:

  • Record the ownership of assets to represent registries and custodial hierarchies
  • Model any financial contract from simple tokens to exotic derivatives
  • Coordinate complex asset movements and settlement flows
  • Manage the lifecycle and evolution of financial instruments

Being able to take ready-made building blocks for your application allows you to significantly reduce time-to-market and get into production quicker.

We are continuing to improve and expand the library and appreciate feedback from early adopters.

Specific Changes

  • The Daml Assistant now comes with a Daml Finance Getting Started template, which can be installed with `daml new test --template=quickstart-finance`
  • The Getting Started Guide can be found in the Daml Finance Documentation
  • The following packages are marked as Stable with this release:
    • ContingentClaims.Core
    • ContingentClaims.Lifecycle
    • Daml.Finance.Account
    • Daml.Finance.Claims
    • Daml.Finance.Data
    • Daml.Finance.Holding
    • Daml.Finance.Instrument.Generic
    • Daml.Finance.Instrument.Token
    • Daml.Finance.Interface.Account
    • Daml.Finance.Interface.Claims
    • Daml.Finance.Interface.Data
    • Daml.Finance.Interface.Holding
    • Daml.Finance.Interface.Instrument.Base
    • Daml.Finance.Interface.Instrument.Generic
    • Daml.Finance.Interface.Instrument.Token
    • Daml.Finance.Interface.Lifecycle
    • Daml.Finance.Interface.Settlement
    • Daml.Finance.Interface.Types.Common
    • Daml.Finance.Interface.Types.Date
    • Daml.Finance.Interface.Util
    • Daml.Finance.Lifecycle
    • Daml.Finance.Settlement
    • Daml.Finance.Util

Impact and Migration

To use Daml Finance, you need to activate Daml-LF 1.15 by adding this stanza to your daml.yaml project file:

build-options:
  - –target=1.15

Daml Interfaces and Daml-LF 1.15 are now Generally Available

Background

Daml has predictability and safety as two of its core design philosophies in the sense that we want to ensure that both during development and at the point a party signs a contract, all possible consequences of that contract can be predicted. This philosophy could be at odds with a typical business requirement to be able to easily extend functionality in the future. For example, I want to be able to agree on the rules for a marketplace today by signing the marketplace smart contracts. But I want to be able to later define entirely new asset classes that I cannot yet predict. Indeed, building Daml applications that were extensible in such a way has been tricky until now.

The above philosophy also led to a number of design decisions that made Daml code dependencies fairly static and closely coupled on-ledger types with the types downstream components like client applications consumed and processed. This close coupling between different Daml packages and on- and off-ledger types made it more difficult to evolve applications over time.

Daml Interfaces are our answer to extensibility and modularization of applications without breaking with predictability and safety.

Specific Changes

  • Daml Interfaces are now available in the Daml Smart Contract Language with a stable serialization encoding and are supported for production usage with data continuity guarantees.
  • The Daml Ledger API supports the submission of commands through interfaces, and streaming active contract sets and transactions with interface types.
  • The JSON API supports the submission of commands through interfaces, and queries and stream queries for interface types, including support for caching on the query store.
  • Daml Triggers and Daml Script support the submission of commands through interfaces, and queries and stream queries for interface types.
  • The Navigator supports the submission of commands through interfaces.
  • The Java and TypeScript/JavaScript bindings expose the query and command submission capabilities of the APIs.

Impact and Migration

  • To use Daml Interfaces, you no longer need to activate the preview flag on the sandbox but you still must explicitly opt-in to using Daml-LF 1.15 as your compiler target until the next release. Please refer to the documentation on interfaces on their use.
  • Daml-LF is available as of Canton protocol version 4.0.0. Please see under “Canton protocol 4.0.0 is now Generally Available” how to migrate.
  • The HTTP JSON API service query store database schema has changed and needs to be reset. If you are migrating from a previous version, either reset your database manually or start the HTTP JSON API with one of the options that regenerate the schema (create-only, create-if-needed-and-start, create-and-start).

Canton protocol 4.0.0 is now Generally Available

Background

The canton protocol determines the core capabilities and properties of a Canton network. Version 4 brings support for Daml-LF 1.15 and with that Daml Interfaces and Daml Finance. In addition, it moves several static configuration parameters to dynamic configuration so that they can be changed at runtime, and adds an optimization that significantly reduces transaction sizes in some situations.

Specific Changes

  • Starting with protocol version 4, the size of messages exchanged between participants and domains has decreased. More specifically, the subviews of transaction views are now stored in balanced binary Merkle trees, similarly to the way root views are stored. This provides a significant size reduction when many subviews are blinded, as complete subtrees can be reduced to single hashes.
  • The following domain parameters are now dynamic:
    • Reconciliation interval
    • Max request size (renamed from max inbound message size)
    • Max rate per participant

    These parameters can now be changed while the domain is running.  See the documentation for more information.

Impact and Migration

Canton still supports protocol version 3, but protocol version 3 has to be added as a static domain parameter before starting the new binary:

canton.domains.mydomain.init.domain-parameters.protocol-version = 3

If you started the domain node accidentally before changing your configuration, your participants won’t be able to reconnect to the domain, as they will fail with a message like:

DOMAIN_PARAMETERS_CHANGED(9,d5dfa5ce): The domain parameters have changed

To recover from this, you need to force a reset of the stored static domain parameters using:

canton.domains.mydomain.init.domain-parameters.protocol-version = 3
canton.domains.mydomain.init.domain-parameters.reset-stored-static-config = yes

New domains deployed using Canton version 2.5.0 will use protocol version 4. Existing users that want the features of protocol version 4 need to perform a protocol version upgrade.

Config changes need to be adapted to as follows:

  • Moving from static to dynamic

    Specifying the value of a dynamic domain parameter in the static configuration is not allowed and doing so will cause the node to fail during config parsing. As a result, values for the three new dynamic domain parameters need to be removed from the static configuration and set via console commands, either interactively, or through a bootstrap script.

    For example, suppose that the max rate per participant was set using the following config:

    canton {
      domains {
        mydomain {
          domain-parameters {
            max-rate-per-participant = 10000
            reconciliation-interval = 30
            max-inbound-message-size = 10485700
          }
        }
      }
    This config entry should be removed and the value can be set using the following console command:

    mydomain.service.set_max_rate_per_participant(10000)

    mydomain.service.set_max_request_size(10485700)
    // equivalently, one case use
    `mydomain.service.set_max_inbound_message_size(10485700)`
    The other two parameters can bet set using similar console commands (set_reconciliation_interval and set_max_request_size). More examples can be found in the documentation.
  • Increasing maximum request size

    When increasing max request size, the sequencer nodes need to be restarted for the new value to be taken into account. If the domain is not distributed, restart can be done as follows

    participants.all.domains.disconnect(mydomain.name)
    nodes.local.stop()
    nodes.local.start()
    participants.all.domains.reconnect_all()
    If the domain is distributed, then restart can be done as follows:

    participants.all.domains.disconnect(sequencer1.name)
    nodes.local.stop()
    nodes.local.start()
    participants.all.domains.reconnect_all()
  • Other changes around dynamic domain parameters
    Compared to Canton 2.3.0, several changes impact how dynamic domain parameters are updated:

    Command update_dynamic_domain_parameters should be preferred over update_dynamic_parameters.
    The copy method on dynamic domain parameters was renamed to update
    Durations are specified using NonNegativeFiniteDuration rather than with TimeoutDuration. Implicit conversions allow to write these in a human-readable form.
    Migration from 2.3.0 to 2.5.0

    Updating dynamic domain parameters in Canton 2.3.0 was done as follows:

    mydomain.service.update_dynamic_parameters(
      _.copy(
        participantResponseTimeout = TimeoutDuration.ofMinutes(3),
        ledgerTimeRecordTimeTolerance = TimeoutDuration.ofMinutes(1),
        mediatorReactionTimeout = TimeoutDuration.ofMinutes(1),
        transferExclusivityTimeout = TimeoutDuration.ofSeconds(2),
        topologyChangeDelay = TimeoutDuration.ofSeconds(1),
      )
    )
    With Canton 2.5.0, this needs to be rewritten as follows:

    mydomain.service.update_dynamic_domain_parameters(
      _.update(
        participantResponseTimeout = 3.minutes,
        ledgerTimeRecordTimeTolerance = 1.minute,
        mediatorReactionTimeout = 1.minute,
        transferExclusivityTimeout = 2.seconds,
        topologyChangeDelay = 1.second,
      )
    )

Improvements to the Java bindings 

Background

The Java Bindings are the main integration component we offer for use cases where flexibility and performance are of the greatest importance. It provides a low-overhead, non-blocking and rich interface to interact with Ledger API services. Its code generator makes it easy to interact from your application with your on-ledger Daml models without much boilerplate code. Aside from offering full support for interfaces, we added a series of improvements that we expect will improve the overall developer experience when using them, as well as making them safer.


Specific Changes

  • The code generator now also generates metadata for choices, allowing you to refer to them by a handle instead of spelling it out as a string that needs to be kept in sync with your Daml code manually. See https://github.com/digital-asset/daml/issues/14329.
  • As a replacement for fromValue methods that allow to decode a Daml value in its wire encoding into Java codegen equivalent, we introduced ValueDecoders that return the specific type without casting. fromValue methods still exist but are marked as deprecated. See the following section about the impact. This new approach works with interfaces too. See https://github.com/digital-asset/daml/issues/14313.
  • Leveraging the new type-safe ValueDecoders, we also added new methods to the ActiveContractsClient and TransactionClient that allow you to read the active contract set and keep it up to date in a type-safe manner. See https://github.com/digital-asset/daml/issues/14969.
  • You can now retrieve the template or interface identifier of contracts with the new method getContractTypeId. See https://github.com/digital-asset/daml/issues/15350.
  • Methods to produce Ledger API commands do not return Command directly anymore; instead, they return an Update<R>, which is the first step towards a more type-safe interaction with the Ledger API. These types are now also accepted by the clients in our bindings, making the transition to this new approach likely seamless. See the following section about the impact of this change. See https://github.com/digital-asset/daml/issues/14312.
  • In the CommandService, multiple overloaded methods have been deprecated in favor of an equally safe but narrower interface, whereby you build your submission using a builder and submit that. This reduces the surface of the CommandClient to one method for each Ledger API endpoint. See the following section for the impact. See https://github.com/digital-asset/daml/issues/15424.

Impact and Migration

  • Generated methods that produce a command now return a different type, Update<R> instead of Command. Since the types are accepted by the clients, the vast majority of use cases in which these values were passed directly to the client means effectively no change to your code is needed. The cases in which you might need to change the way you are using the code fall in two categories:
    • you were assigning the command to a variable before passing it to the client: in this case, change the type of the variable to the new one.
    • you were calling methods on the command returned by the method: in this case, you can still access the command within the returned Update<R> by calling the commands() method and interacting with the command as you used to.
  • A few existing APIs have been deprecated but will continue working for backwards compatibility.
    • The fromValue(x, f…) API has been deprecated and we recommend you change that to valueDecoder(f…).decode(x), and if you write lambdas over fromValue you can simply call nested valueDecoder functions. Moreover, we now provide such functions for primitive types, in the PrimitiveValueDecoders utility class. Existing calls will continue to work following our deprecation policy but you are strongly advised to use the new approach as soon as possible to avoid future breakages.
    • All methods in the CommandClient have been deprecated in favor of a new builder-based approach. The methods will follow the usual deprecation policy. You are advised to use the new interface, whereby instead of calling a specific overload, you prepare a submission using com.daml.ledger.javaapi.data.CommandsSubmission and issue it. The builders require to take mandatory fields (application ID, command ID and the actual commands) and have a fluent interface that allows to progressively add optional fields (like the workflow ID). The names of the fields are unchanged.

Ledger API resource-based backpressure

Background

We have introduced a backpressure mechanism to the Ledger API by which the client calls are rejected with an ABORT error whenever one of the vital server resources has been exceeded.

Specific Changes

  • When the CPU worker thread pool or the database communication thread pool is overloaded the server responds with a THREADPOOL_OVERLOADED error code.
  • When the server reaches the maximum number of simultaneously open Ledger API gRPC streams, it returns a MAXIMUM_NUMBER_OF_STREAMS error code.
  • When the maximum memory thresholds are exceeded the server responds with a HEAP_MEMORY_OVER_LIMIT error code.
  • The limits in Canton can be modified using the configuration parameters. The defaults have been set to
canton.participants.<participant-id>.ledger-api.rate-limit = {
    max-api-services-queue-size = 20000
    max-streams = 1000
    max-used-heap-space-percentage = 100
    min-free-heap-space-bytes = 0
}

Impact and Migration

If these limits are too low for your environment, you may run into the new errors. In that case you have to set the limits to levels appropriate to your use case.

User management improvements 

Background

User management service allows you to create users on the participant that are associated with a list of actAs and readAs claims that grant that user access rights to parties. An external IAM system can then be used to issue access tokens for a specific user and have the participant resolve the corresponding party claims which then can be used to authorize the calls.

We have made a number of improvements to the user management and party management services that simplify the integration of these services into a larger landscape of access and identity management.

Specific Changes

  • We have added an is_deactivated attribute to participant users. Mutating this attribute provides a superior mechanism for temporarily disabling a user to the earlier delete and create call pair.
  • We have introduced participant server's local metadata for parties and participant users consisting of:
    • a resource version for optional concurrent change control,
    • modifiable key-value based annotations.
  • We have added an ability to update participant users. The modifiable fields are primary_party, is_deactivated, metadata.annotations.
  • We have added an ability to update records associated with the ledger parties. Modifiable field is local_metadata.annotations.
  • With the introduction of the party annotations we discourage the use of party_details.display_name in favor of using party's metadata annotations.

Impact and Migration

This is a purely additive change.

[Enterprise Only] KMS support for envelope encryption of Canton keys 

Background

Canton nodes use a variety of cryptographic keys that they need constant access to during operation. To add a layer of security for node operators, it is now possible to configure Canton nodes such that the keys are always encrypted at rest. This is done by employing a technique called envelope encryption where an external KMS system encrypts the private keys that the node keeps in its database. When the node starts, it communicates with the KMS system to decrypt the keys and keeps the decrypted keys in memory only.

Currently only AWS KMS is supported. 

Specific Changes

Please refer to the feature’s documentation.

Impact and Migration

To accommodate the KMS features, there have been changes to the signatures for console commands related to uploading and downloading for crypto key pairs and public keys. 

`keys.public.upload and `keys.secret.upload` now expect a `ByteString` as input and not a reference to an object.

`keys.public.download` and `keys.secret.download` have been split in two different 
Commands each:

  1. `download` expects a key fingerprint and returns the serialized public key or key pair (`ByteString`) and;
  2. `download_to` additionally expects a filename and stores the `ByteString` directly into a file.

Both methods have the protocol version as an optional input to use for serialization. By default it's the latest protocol version supported by the release.

The return type of `keys.secret.list` has been renamed from `PublicKeyWithName` to `PrivateKeyMetadata` to return additional information related to the stored private keys, such as which wrapper key was used to encrypt the stored private key. The fields on the previous result type are the same as on the new result type for backwards compatibility.

[Enterprise Only] High Availability support for the Canton Domain manager 

Background

Until now, the domain (and/or topology) manager was one type of Canton node that did not support high availability. An intermittent outage of that node was not a total system outage as it is only responsible for handling identity transactions, but it would cause an outage to any operations to do with allocating parties, or adding packages.
As of 2.5.0, it now also supports high availability using an active-passive setup against a shared (HA) database.

Specific Changes

Please refer to the documentation for high availability usage.

Impact and Migration

This is a purely additive change.

Changes to the metrics 

Background

The metrics subsystem has undergone a major revision and is now documented much more comprehensively. As a result, a number of unnecessary or confusing metrics have been removed or rationalized. At the same time all metrics produced have been documented.

Specific Changes

  • Daml metrics have been added to the documentation.
  • The following cache related metrics were removed. They were placed on unreachable code paths and as a result were always showing zeroes:
    • daml.execution.cache.contract_state.load_successes
    • daml.execution.cache.contract_state.load_failures
    • daml.execution.cache.contract_state.load_total_time.*
    • daml.execution.cache.key_state.load_successes
    • daml.execution.cache.key_state.load_failures
    • daml.execution.cache.key_state.load_total_time.*
    • daml.user_management.cache.load_successes
    • daml.user_management.cache.load_failures
    • daml.user_management.cache.load_total_time.*
  • Confusing metrics related to the cache updates have been removed and substituted with pairs of separate metrics for contract and contract key caches respectively
    • removed
      • daml.execution.cache.register_update.*
    • added
      • daml.execution.cache.contract_state.register_update.*
      • .exedamlcution.cache.key_state.register_update.*
  • The following index database-related metrics were removed. They all have counterparts among the daml.services.index.<operation>.* metrics which should be used instead:
    • daml.index.db.get_lf_archive.*
    • daml.index.db.get_parties.*
    • daml.index.db.list_known_parties.*
    • daml.index.db.list_lf_packages.*
    • daml.index.db.lookup_ledger_configuration.*
    • daml.index.db.lookup_ledger_end.*
    • daml.index.db.lookup_ledger_id.*
    • daml.index.db.lookup_participant_id.*
    • daml.index.db.prune.*
    • daml.index.db.store_configuration_entry.*
    • daml.index.db.store_ledger_entry_combined.*
    • daml.index.db.store_package_entry.*
    • daml.index.db.store_party_entry.*
    • daml.index.db.store_rejection.*
  • A number of metrics related to the index service have been renamed:
    • removed:
      • daml.services.index.lookup_contract_after_interpretation.*
    • added:
      • daml.services.index.lookup_contract_state_without_divulgence.*

Impact and Migration

Update your metrics names according to the above.

[Beta] Golden Signal Metric Monitoring 

Background

A best practice for monitoring the services of a distributed system is to use an approach known as the “Golden Signals” or “REDS”. Some helpful references for this are Monitoring Distributed Systems, RED Method, and 4 SRE Golden Signals (What they are and why they matter). Using this methodology, monitored metrics are used to determine if the application is healthy and, if unhealthy, what service or endpoint (i.e., API) is the root cause of the issue.

Specific Changes

In an effort to further improve the operational aspects of the various components that make up the Daml ecosystem, we introduced “Golden Signal” metrics for all services that offer a gRPC or an HTTP-JSON API. These metrics are available when using the Prometheus exporter.
For the specific metrics, see the documentation.

Impact and Migration

This is a purely additive change.

[Labs] Custom Views Library 

Background

While the Ledger API gives full access to any data stored on the ledger and allows users to interact with it, it doesn't provide the full range of querying capabilities that you might expect from an application-specific data store. As such, when developing applications against the Ledger API, it's often the case that the same patterns appear over and over again in many client applications. Even though these patterns are well known and established internally, the newcomer might not see those emerging. As an improvement to how we empower developers to read data from the ledger to build applications around it, we started an initiative to package these patterns to be offered as a facilitation for the Daml off-ledger integrator. Our first release is a Java library that allows to perform a projection of ledger data onto a traditional relational database, around which an application that needs rich querying can be built.

Specific Changes

The new Custom Views Java library is now available in Labs status. You are welcome to play around with the library and provide feedback as to where you would like to see this going in the future.

Impact and Migration

This is a purely additive change.

Minor Improvements

  • Some error messages coming from the HTTP JSON API Service have been corrected to no longer confuse interface IDs with template IDs, based on what varieties of contract type ID the endpoint in question supports. See https://github.com/digital-asset/daml/issues/14844.
  • We made several improvements to the logging of Daml Triggers and how to configure it when running them as part of the Daml Trigger Service (see https://github.com/digital-asset/daml/issues/15489):
    • Support for standard CLI logging options (#15506 and #15508)
    • The trigger context is now logged, with the version, ID, definition, level and a span
    • General overhaul of where triggers log along with more user focused pretty printing of internal values
  • We have introduced a gRPC streams optimization targeting complex Protobuf payloads. In some scenarios, it can cause up to 60-80% throughput increase for GetTransactions, GetTransactionTrees and GetActiveContracts streams. See https://github.com/digital-asset/daml/pull/15408.
  • Daml test coverage: Add interfaces & interface choices to coverage report (#15563)
  • Daml Ledger export:
    • Added support for interface choices (#15189)
    • Export.daml imports are now sorted alphabetically (#15189)
  • Canton node health dumps can now be gathered through a remote console. See the documentation for more information.
  • We have implemented integrity checks that ascertain that the metering reports have not been manipulated between the time of the report generation and its receipt by the DA.
  • Infrastructure transactions are no longer metered by default and will not show up in the participant metering report. To override this there is a new participant node parameter config setting (canton.<participant>.parameters.exclude-infrastructure-transactions) that can be used to meter infrastructure transactions. For more information on participant metering see the Participant Metering documentation. Infrastructure transactions include:
    • Ping transactions, used by the Ping health check
    • Dar Distribution transactions
  • The Ledger API JWT authorization now supports leeway parameters. This allows you to work around authentication failures due to clock-skew by defining a leeway window in which the tokens should still be considered valid. See the documentation for more information.
  • The error category "MaliciousOrFaultyBehaviour" was renamed to "SecurityAlert". The integer code remained unchanged.
    • The error category for INVALID_TOPOLOGY_TX_SIGNATURE_ERROR and UNAUTHORIZED_TOPOLOGY_TRANSACTION has changed to SecurityAlert.
  • Daml Standard Library
    • Added function ``coerceInterfaceContractId`` (#15405)
    • ``show @Text`` now escapes special characters, producing syntactically correct expressions (#15177)
    • Add Semigroup, Foldable, and Traversable instances for Validation (#15080)
    • Generalize the (<?>) operator to support a generic type for Validation (#15244)
    • Remove TemplateOrInterface type class (#15347)
  • Daml compiler error improvements
    • Improve type error for Numeric types with precision > 37 and mistyped/missing field access (#15626)
    • Added error for implementing non-interfaces (#15435)
    • Catch mistyped view implementations via better Daml-specific type error detection (#15426)
    • Various other improvements in displaying warnings/errors, including accessing a nonexistent field.
    • Add warnings to conversion, Warn for tuples size > 5 (#15018). This is enabled by default across customer code. To disable, use flag -disable-warn-large-tuples=yes

Security and Bug fixes