Skip to main content

Change Log

This page documents all releases of the LimePoint Solifi Data Streaming Connector.

Release 2.2.6​

Bug Fixes​

  • Fixed a bug caused by MSSQL JDBC Driver where stale connections were not being invalidated and instead propagated to the consumer application. This in turn caused the application to go in a loop to retry the record. Fixes MINTSD-1577, MINTSD-1578.

Release 2.2.5​

Name Change & Documentation Update​

We are changing the name of the consumer from Solifi Real-Time Reporting Application to LimePoint Solifi Data Streaming Connector. The name change has been updated in the EULA.
There is no change in the name of the docker images.
The new documentation website now resides at https://solifi.limepoint.com. Going forward, we will only publish updates on the main site and no longer on github.

Bug Fixes​

  • Fixed a bug to log CommitFailedException if the consumer fails to update the offset.

Enhancements​

  • Updated platform libraries to latest versions.
  • Added a property (collation-name) to enforce collation for primary key columns. This applies only to MS SQL databases. Default value is empty string and will use whatever is set at the database level.
  • Updated the initial-load-mode to allow loading partitions in parallel per topic. Before this release, the consumer would lock all partitions of a topic and load them into the database 1 at a time, thus blocking other consumer threads to work on that topic. This update allows the partitions to be loaded in parallel, thus saving overall load times.
  • Added a property named records-fetch-size which overrides the value of max.poll.records when running in initial-load-mode and controls how many records will be fetched from the broker. By setting this property you no longer have to change the value of max.poll.records when switching from initial load mode to streaming mode, See Initial Load Mode for details.

Release 2.2.4​

Bug Fixes​

  • Fixed a bug introduced in 2.2.0 which ignored the topic-table-mapping configuration that resulted in tables being created incorrectly.
  • Added exception reporting if the license file is corrupted.

Enhancements​

  • Added more logging to indicate the flow between the consumer and upstream brokers (e.g., consumers now log messages starting with Sending acknowledgement once processing is complete).
  • Updated the default max-poll-records to 5000 and pollTimeOut to 5 seconds.
  • Updated initial-load-mode to be a two-stage process to improve memory footprint. The consumer no longer requires excessive memory to run in initial-load-mode. See Initial Load Mode for details.
  • Added a new property save-full-audit which defaults to false. This property is applicable only during initial-load-mode and forces the consumer to save all audit events in the topics. See Initial Load Mode for details.

Upgrade Notes​

Auditing Users

If any of your existing audit tables have a primary key that does not include all of the following columns, you will need to perform a full data resynchronization:

  • lp_kafka_partition
  • lp_kafka_offset
  • lp_db_insert_user
  • lp_db_insert_date

If you are unable to perform a full resync, please contact LimePoint Support for guidance on correcting the primary keys without dropping tables.

Release 2.2.1​

Bug Fixes​

  • Fixed an issue where the consumer ignored the topic property when running in initial_load mode.

Release 2.2.0​

Enhancements​

  • Introduced a new consumer mode called initial_load. The consumer now supports two modes: streaming and initial_load.
    • By default, the consumer runs in streaming mode (same behavior as previous versions).
    • initial_load mode is recommended for customers handling large data loads (typically over 10 million messages).
    • See Initial Load Mode for details.
  • Updated error handling to clean up messages left behind in the MDC via other threads.

Release 2.1.1​

Enhancements​

  • Updated the datetime precision for the lp_db_insert_date column for all audit tables to precision 6 (helps with MS SQL mirroring issues).
    • Note: Only applicable for new audit tables; does not change existing tables.
  • Improved logging: consumer now appends partition:offset in front of every message.
  • Added msal4j:1.22.0 to help authenticate with MS Azure Identity.

Release 2.1.0​

Enhancements​

  • Platform upgrades including container images, JDK, database drivers, and Spring Boot.
  • Improved error logging with better messaging for database, network, or business exceptions.
  • Added insert_date to the audit logging tables.
  • Updated MSSQL driver support to not hold locks when using merge.

Release 2.0.8​

Enhancements​

  • Added support for unicode characters. By default, the consumer stores data without unicode. To enable unicode, set support-unicode: true in application.yml.
    • Note: Enabling unicode reduces column sizes as unicode occupies twice the space.
    • See Sample Configuration for details.
  • Updated the auditing function to add the consumer group id in audit tables. This enables customers to keep audit log history during data refreshes.
  • Added logging for delete events. The consumer now logs Received 'Delete' ConsumerRecord when it finds a record to be deleted.

Release 2.0.7​

Bug Fixes​

  • Fixed an issue with date columns not being recognized correctly if the schema has long type marked against the column.
    • This only affects clients who have not yet migrated to the v2 version of topics.
    • You may notice Updating Optional DATE schema from 'long' to 'int' for <column_name> messages in logs (can be safely ignored).

Release 2.0.6​

Enhancements​

  • Added support for monitoring via Prometheus endpoint. The consumer now exposes a /actuator/prometheus endpoint.
  • See Health Monitoring for details.

Release 2.0.5​

Enhancements​

  • Added support for using Regex to manage table-to-topic mappings via the solifi.topic-table-mapping property.
  • This enables clients to remove prefixes in topic names when creating tables.
  • See Sample Configuration for details.

Release 2.0.4​

Enhancements​

  • Added support for overriding the default schema used when creating tables in MSSQL server.
  • The property solifi.default-db-schema-name defaults to dbo when not specified.
  • See Sample Configuration for details.

Release 2.0.3​

Enhancements​

  • Introduced a new property topics-regex which uses regex pattern matching to automatically look up topics.
  • Default value is .* (reads all available topics).
  • The consumer periodically checks (every 5 mins) for new topics.
  • With this feature, you no longer need to restart the consumer when adding new topics.
  • If topics-regex and topics are both specified, topics takes precedence.

Release 2.0.2​

Changes​

  • Updated auditing tables to add a prefix of lp_ for metadata columns to avoid conflicts with database column names.

Release 2.0.1​

Architecture Changes​

Version 1 of the consumer relied on SQL files to manage database tables. This approach had limitations as any schema changes in the upstream Solifi broker required a new consumer release.

From version 2.x onward, the architecture was restructured:

  • ❌ No longer ships SQL files
  • ✅ Detects required schemas from the Schema Registry
  • ✅ Dynamically creates or updates tables in the backend database
First Run Behavior

When running the consumer for the first time, you might see errors in the logs related to table creation. These can be safely ignored as they are caused by multiple threads attempting to create the same table simultaneously.

Getting the Latest Version​

Contact LimePoint Support to obtain the latest version of the consumer.