Release Notes

Release Date: October 16th, 2023

What’s New

We're excited to announce the initial integration of Discovery with OpenAI in this release. By integrating with OpenAI, we've incorporated robust embedding capabilities for vector search, enhancing data retrieval efficiency and accuracy. Furthermore, we've harnessed GPT's capabilities for chat completions. It's important to note that this integration is currently in its alpha stage, and we're eager to explore the transformative possibilities that GPT offers. We've also dedicated efforts to upgrading dependencies and implementing minor improvements. Should you have any questions, suggestions, or feedback, please don't hesitate to reach out to our development team. We value your input and are committed to continuously improving our services. Continue to read on for all new features, bug fixes, and improvements.

New Features

Bugs Fixed

• Trim UUIDs on API

Improvements

• Udemy Connector uses single proccess/scan

• Add schedule id to seed-execution table

• [ocr-processor] Remove conditional action based on Tika

• [bert-service-processor] Make model name to use configurable

Breaking changes

• If upgrading from a version prior to 1.7.0, export the existing configuration, delete the configuration indices (ingestion: seed, processor, and pipeline / discovery: endpoint and processor), and re-import it after upgrade has been performed.

Others

• Graalvm updated to version 23.0.1

• Micronaut BOM updated to version 3.10.0

• Micronaut OpenAPI updated to version 4.10.0

• Discovery 1.9.0 was tested in AKS v1.25.11, v1.26.6, and v1.27.1

Supported Versions

Discovery provides support and bug fixing for the following versions:

• Version 1.10.0

  • If upgrading from a version prior to 1.7.0, export the existing configuration, delete the configuration indices (ingestion: seed, processor, and pipeline / discovery: endpoint and processor), and re-import it after upgrade has been performed.

  • Json exception when script processor compilation fails

  • BatchId is stored as null

  • OCR component records don't continue in the pipeline

  • Internal server error importing a zip file

  • [Search UI] Queries with special regex characters causes page to fail

  • Scheduler entity is ignoring some properties from the configuration

  • Website Connector requires a mountpoint to work in k8s

  • Hugging Face service creates outdated file structure

• Version 1.9.0

  • If upgrading from a version prior to 1.7.0, export the existing configuration, delete the configuration indices (ingestion: seed, processor, and pipeline / discovery: endpoint and processor), and re-import it after upgrade has been performed.

  • Improvements in JsonUtils.substitute

  • Token scanner writes checksums to the scan index.

  • Staging Connector - Aggregation by group doesn't support Arrays

• Version 1.8.0 (If you are in this version, plan to upgrade soon)

  • If upgrading from a version prior to 1.7.0, export the existing configuration, delete the configuration indices (ingestion: seed, processor, and pipeline / discovery: endpoint and processor), and re-import it after upgrade has been performed.

Deprecated Versions

Releases that are no longer recommended for use and their deprecation dates are listed below.

• Version 1.7.0 (October 2023)

  • Close record collector in AbstractRequestActionExecutor for errored jobs

  • Incorrect log message from Staging Hydrator

  • Error importing zip

  • [Discovery API] Feedback Atlas default template

  • [Ingestion] Possible memory leakage in Elasticsearch connector

  • ES Connector: Checksum is not hashed before adding a record.

  • Search UI showing more search result pages than required

  • Job in the wrong state should be retried

• Version 1.6.0 (August 2023)

  • Pipeline configuration should not allow null action

  • Ingestion Admin should allow deep cloning multiple times

  • Single seed schedules should not be enqueued if the same seed is already running

  • Discovery API - Endpoints should handle an empty body request

  • Discovery API - Mongo Component should store the response body as JsonNode

  • Discovery API - Snap Component not casting error message when facets field points to an array

  • Breaking changes:

    • When configuring the S3 connector, the region must be in the format AWS requires it.

      • e.g. “us-east-1” instead of “"US_EAST_1"

• Version 1.5.0 (July 2023)

  • Internal server should not error when adding item with no "body" to Staging repo

  • JsonUtils can not substitute value properties of intNode objects when are into an array