.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
.. use this file except in compliance with the License. You may obtain a copy of
.. the License at
..
..   http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
.. License for the specific language governing permissions and limitations under
.. the License.

.. _release/3.5.x:

============
3.5.x Branch
============

.. contents::
    :depth: 1
    :local:

.. _release/3.5.1:

Version 3.5.1
=============

Features
--------

* :ghissue:`5626`, :ghissue:`5665`: Debian Trixie support
* :ghissue:`5709`: Automatic Nouveau and Clouseau index cleanup
* :ghissue:`5697`: Add UUID v7 as a ``uuid`` algorithm option. The default is
  still the default ``sequential`` algorithm.
* :ghissue:`5713`, :ghissue:`5697`, :ghissue:`5701`, :ghissue:`5704`: Purge
  improvements and fixes. Optimize it up to ~30% faster for large batches.
  ``max_document_id_number`` setting was removed and ``max_revisions_number``
  set to ``unlimited`` by default to match ``_bulk_docs`` and ``_bulk_get``
  endpoints.
* :ghissue:`5611`: Implement the ability to downgrade CouchDB versions
* :ghissue:`5588`: Populate zone from ``COUCHDB_ZONE`` env variable in Docker
* :ghissue:`5563`: Set Erlang/OTP 26 as minimum supported version
* :ghissue:`5546`, :ghissue:`5641`: Improve Clouseau service checks in
  ``clouseau_rpc`` module.
* :ghissue:`5639`: Use OS certificates for replication
* :ghissue:`5728`: Configurable reduce limit threshold and ratio

Performance
-----------

* :ghissue:`5625`: BTree engine term cache
* :ghissue:`5617`: Optimize Nouveau searches when index is fresh
* :ghissue:`5598`: Use HTTP/2 for Nouveau
* :ghissue:`5701`: Optimize revid parsing: 50-90% faster. Should help purge
  requests as well as ``_bulk_docs`` and ``_bulk_get`` endpoints.
* :ghissue:`5564`: Use the built-in binary hex encode
* :ghissue:`5613`: Improve scanner performance
* :ghissue:`5545`: Bump process limit to 1M

Bugfixes
--------

* :ghissue:`5722`, :ghissue:`5683`, :ghissue:`5678`, :ghissue:`5646`,
  :ghissue:`5630`, :ghissue:`5615`, :ghissue:`5696`: Scanner fixes. Add write
  limiting and switch to traversing documents by sequence IDs instead of by
  document IDs.
* :ghissue:`5707`, :ghissue:`5706`, :ghissue:`5706`, :ghissue:`5694`,
  :ghissue:`5691`, :ghissue:`5669`, :ghissue:`5629`, :ghissue:`5574`,
  :ghissue:`5573`, :ghissue:`5566`, :ghissue:`5553`, :ghissue:`5550`,
  :ghissue:`5534`, :ghissue:`5730`: QuickJS Updates. Optimized string operations,
  faster context creation, a lot of bug fixes.
* :ghissue:`5719`: Use "all" ring options for purged_infos
* :ghissue:`5649`: Retry call to dreyfus index on noproc errors
* :ghissue:`5663`: More informative error if epochs out of order
* :ghissue:`5649`: Dreyfus retries on error
* :ghissue:`5643`: Fix reduce_limit = log feature
* :ghissue:`5620`: Use copy_props in the compactor instead of set_props
* :ghissue:`5632`, :ghissue:`5627`, :ghissue:`5607`: Nouveau fixes. Enhance
  ``_nouveau_cleanup``. Improve security on http/2.
* :ghissue:`5614`: Stop replication jobs to nodes which are not part of the cluster
* :ghissue:`5596`: Fix query args parsing during cluster upgrades
* :ghissue:`5595`: Make replicator shutdown a bit more orderly
* :ghissue:`5595`: Avoid making a mess in the logs when stopping replicator app
* :ghissue:`5588`: Fix ``couch_util:set_value/3``
* :ghissue:`5587`: Improve ``mem3_rep:find_source_seq/4`` logging
* :ghissue:`5586`: Don't wait indefinitely for replication jobs to stop
* :ghissue:`5578`: Use ``[sync]`` option in ``couch_bt_engine:commit_data/1``
* :ghissue:`5556`: Add guards to ``fabric:design_docs/1`` to prevent
  ``function_clause`` error
* :ghissue:`5555`: Improve replicator client mailbox flush
* :ghissue:`5551`: Handle ``bad_generator`` and ``case_clause`` in ``ken_server``
* :ghissue:`5552`: Improve cluster startup logging
* :ghissue:`5552`: Improve mem3 supervisor
* :ghissue:`5552`: Handle shard opener tables not being initializes better
* :ghissue:`5549`: Don't spawn more than one ``init_delete_dir`` instance
* :ghissue:`5535`: Disk monitor always allows ``mem3_rep`` checkpoints
* :ghissue:`5536`: Fix ``mem3_util`` overlapping shards
* :ghissue:`5533`: No cfile support for 32bit systems
* :ghissue:`5688`: Handle timeout in ``dreyfus_fabric_search``
* :ghissue:`5548`: Fix config key typo in mem3_reshard_dbdoc
* :ghissue:`5540`: Ignore extraneous cookie in replicator session plugin

Cleanups
--------

* :ghissue:`5717`: Do not check for Dreyfus. It's part of the tree now.
* :ghissue:`5715`: Remove Hastings references
* :ghissue:`5714`: Cleanup fabric r/w parameter handling
* :ghissue:`5693`: Remove explicit erlang module prefix for auto-imported functions
* :ghissue:`5686`: Remove ``erlang:`` prefix from ``erlang:error()``
* :ghissue:`5686`: Fix ``case_clause`` when got ``missing_target`` error
* :ghissue:`5690`: Fix props caching in mem3
* :ghissue:`5680`: Implement db doc updating
* :ghissue:`5666`: Replace ``gen_server:format_status/2`` with ``format_status/1``
* :ghissue:`5672`: Cache and store mem3 shard properties in one place only
* :ghissue:`5644`: Remove redundant ``*_to_list`` / ``list_to_*`` conversion
* :ghissue:`5633`: Use ``config:get_integer/3`` in couch_btree
* :ghissue:`5618`: DRY out ``couch_bt_engine`` header pointer term access
* :ghissue:`5614`: Stop replication jobs to nodes which are not part of the cluster
* :ghissue:`5610`: Add a ``range_to_hex/1`` utility function
* :ghissue:`5565`: Use maps comprehensions and generators in a few places
* :ghissue:`5649`: Remove pointless message
* :ghissue:`5649`: Remove obsolete clauses from dreyfus
* :ghissue:`5621`: Minor couch_btree refactoring

Docs
----

* :ghissue:`5705`: Docs: Update the /_up endpoint docs to include status responses
* :ghissue:`5653`: Document that _all_dbs endpoint supports inclusive_end query param
* :ghissue:`5575`: Document how to mitigate high memory usage in docker
* :ghissue:`5600`: Avoid "master" wording at setup cluster
* :ghissue:`5381`: Change unauthorized example to 401 for replication
* :ghissue:`5682`: Update install instructions
* :ghissue:`5674`: Add setup documentation for two factor authentication
* :ghissue:`5562`: Add AI policy
* :ghissue:`5548`: Fix reshard doc section name
* :ghissue:`5543`: Add ``https`` to allowed replication proxy protocols

Tests/CI/Builds
---------------

* :ghissue:`5720`: Update deps: Fauxton, meck and PropEr
* :ghissue:`5708`: Improve search test
* :ghissue:`5702`: Increase timeout for `process_response/3` to fix flaky tests
* :ghissue:`5703`: Use deterministic doc IDs in Mango key test
* :ghissue:`5692`: Implement 'assert_on_status' macro
* :ghissue:`5684`: Sequester docker ARM builds and fail early
* :ghissue:`5679`: Add ``--disable-spidermonkey`` to ``--dev[-with-nouveau]``
* :ghissue:`5671`: Print request/response body on errors from mango test suite
* :ghissue:`5670`: Fix ``make clean`` after ``dev/run --enable-tls``
* :ghissue:`5668`: Update xxHash
* :ghissue:`5667`: Update mochiweb to v3.3.0
* :ghissue:`5664`: Disable ppc64le and s390x builds
* :ghissue:`5604`: Use ASF fork of ``gun`` for ``cowlib`` dependency
* :ghissue:`5636`: Reduce btree prop test count a bit
* :ghissue:`5633`: Fix and improve couch_btree testing
* :ghissue:`5572`: Remove a few more instances of Ubuntu Focal
* :ghissue:`5571`: Upgrade Erlang for CI
* :ghissue:`5570`: Skip macos CI for now and remove Ubuntu Focal
* :ghissue:`5488`: Bump Clouseau to 2.25.0
* :ghissue:`5541`: Enable Clouseau for the Windows CI
* :ghissue:`5537`: Add retries to native full CI stage
* :ghissue:`5531`: Fix Erlang cookie configuration in ``dev/run``
* :ghissue:`5662`: Remove old Jenkinsfiles
* :ghissue:`5661`: Unify CI jobs

.. _release/3.5.0:

Version 3.5.0
=============

Highlights
----------

* :ghissue:`5399`, :ghissue:`5441`, :ghissue:`5443`, :ghissue:`5460`,
  :ghissue:`5461`: Implement parallel ``pread`` calls: lets clients issue
  concurrent ``pread`` calls without blocking each other or having to wait for
  all writes and ``fsync`` calls. This is enabled by default and can be
  disabled with ``[couchdb] use_cfile = false`` in the configuration.

  CouchDB already employs a multiple-parallel-read and concurrent serial-write
  design at the database engine layer, but below that in the storage engine,
  each file representing a database shard is required to route all read / write
  / sync requests through a single Erlang process that owns the file descriptor
  (fd).

  An Erlang process can at most execute at the speed of a single CPU core. Two
  scenarios can lead to a starvation of the fd-owning Erlang process message
  inbox:

    - 1000s of concurrent read requests with a constant stream of writes per shard,
      or:
    - a high latency storage devices like network block storage.

  Parallel preads re-implements the Erlang ``file`` module in parts as
  ``couch_cfile`` to allow multiple ``dup()``’d file descriptors to be used for
  reading and writing. Writes continue to be serialised at the database engine
  layer, but reads now are no longer blocked by writes waiting to commit.

  Comes with an extensive test suite that includes property testing to ensure
  ``couch_cfile`` behaves exactly like Erlang’s ``file`` in all other cases.

  Performance is always equal or better than before. These scenarios show
  preliminary improvements:

  - random document reads: 15% more throughput
  - read ``_all_docs``: 15% more throughput
  - read ``_all_docs`` with ``include_docs=true``: 25% more throughput
  - read ``_changes``: 4% more throughput
  - single document writes: 8% more throughput
  - 2000x concurrent clients, random document reads on a 12 node cluster: 30%
    more throughput
  - concurrent constant document writes and concurrent single document reads on
    a single shard: 40% more throughput.

    This feature is not available on Windows.

* :ghissue:`5435`: Improve default ``chttpd_server`` options. This helps with
  faster processing of many concurrent TCP connections.

* :ghissue:`5347`: Fix attachment size calculation. This could lead to shards
  not being scheduled for compaction correctly.

* :ghissue:`5494`: Implement ``_top_N`` and ``_bottom_N`` reducers. These
  reducers return the top or bottom values from a map-reduce view. ``N`` is a
  number between ``1`` and ``100``. For instance, a ``_top_5`` reducer will
  return the top 5 highest values for a group level.

* :ghissue:`5498`: Implement ``_first`` and ``_last`` reducers. These reducers
  return the first, and respectively, the last view row for a given group
  level. For example, ``_last`` can be used to retrieve the last timestamp
  update for each ``device_id`` if we had a key like ``[device_id, timestamp]``
  if we query the view with ``group_level=1``.

* :ghissue:`5466`: Conflict finder plugin. Enable the
  ``couch_scanner_plugin_conflict_finder`` plugin to find conflicting docs
  across all the databases. It can be configured to report individual revisions
  or aggregated statistics.

Performance
-----------

* :ghissue:`5499`: QuickJS rope based string implementation.
* :ghissue:`5451`: Optimize config system to use persistent terms.
* :ghissue:`5437`: Fix ``atts_since`` functionality for document ``GET``
  requests. Avoids re-replicating attachment bodies on doc updates.
* :ghissue:`5398`: Save 1 ``write`` for each committing data to disk by using
  ``fdatasync`` while keeping the same level of storage reliability.

Features
--------

* :ghissue:`5526`: Default ``upgrade_hash_on_auth`` to ``true``. Downgrading to
  ``3.4.1``, ``3.4.2``, or ``3.4.3`` is safe. Those versions know how to verify
  these new password hashes.
* :ghissue:`5517`: Enable xxHash file checksums by default. Downgrading to
  ``3.4.x`` versions should be safe. Those versions know how to read and verify
  xxHash checksums.
* :ghissue:`5527`: Update Fauxton and xxHash dependencies.
* :ghissue:`5525`: Array opcode optimization for QuickJS.
* :ghissue:`5518`: More precise error location reporting for QuickJS.
* :ghissue:`5507`, :ghissue:`5510`, :ghissue:`5509`: Erlang 28 compatibiliity.
* :ghissue:`5516`: Detailed node membership info for Prometheus.
* :ghissue:`5489`: Allow TLS client certs for Nouveau requests.
* :ghissue:`5452`: ``BigInt`` support for QuickJS.
* :ghissue:`5439`: Nouveau: upgrade ``dropwizard`` to 4.0.12.
* :ghissue:`5429`: Add ``simple+pbkdf2`` migration password scheme.
* :ghissue:`5424`: Scanner: reduce log noise, fix QuickJS plugin mocks,
  gracefully handle broken search indexes.
* :ghissue:`5421`: Nouveau: upgrade Lucene to 9.12.1.
* :ghissue:`5414`: Remove unused ``multi_workers`` option from
  ``couch_work_queue``.
* :ghissue:`5402`: Remove unused, undocumented and detrimental idle check
  timeout feature.
* :ghissue:`5395`: Remove unused, undocumented and unreliabele ``pread_limit``
  feature from ``couch_file``.
* :ghissue:`5385`: Clean up ``fabric_doc_update`` by introducing an ``#acc``
  record.
* :ghissue:`5372`: Upgrade to Elixir 1.17.
* :ghissue:`5351`: Clouseau: show version in ``/_version`` endpoint.
* :ghissue:`5338`: Scanner: add Nouveau and Clouseau design doc validation.
* :ghissue:`5335`: Nouveau: support reading older Lucene 9x indexes.
* :ghissue:`5327`, :ghissue:`5329`, :ghissue:`5419`: Allow switching JavaScript
  engines at runtime.
* :ghissue:`5326`, :ghissue:`5328`: Allow clients to specify HTTP request ID,
  including UUIDs.
* :ghissue:`5321`, :ghissue:`5366`, :ghissue:`5413`: Add support for
  SpiderMonkey versions 102, 115 and 128.
* :ghissue:`5317`: Add `quickjs` to the list of welcome features.

Bugfixes
--------

* :ghissue:`5515`: Make sure ``query_limit`` config takes effect. Raise default
  limit from ``2^28`` to ``2^59``. Allow ``infinity`` as a more ergonomic
  config value.
* :ghissue:`5522`: Reopen indexes closed by Lucene in Nouveau.
* :ghissue:`5508`: Fix array ``from()`` and ``at()`` for QuickJS.
* :ghissue:`5502`: Buffer overflow and segfault fix in QuickJS.
* :ghissue:`5469`, :ghissue:`5471`: Retry closed connections in Nouveau.
* :ghissue:`5463`: Fix ``badarith`` in Nouveau index query.
* :ghissue:`5447`: Fix arithmetic mean in ``_prometheus``.
* :ghissue:`5440`: Fix ``_purged_infos`` when exceeding ``purged_infos_limit``.
* :ghissue:`5431`: Restore the ability to return ``Error`` objects from `map()`.
* :ghissue:`5417`: Clouseau: add a version check to ``connected()`` function to
  reliably detect if a Clouseau node is ready to be used.
* :ghissue:`5416`: Ensure we always map the documents in order in
  ``couch_mrview_updater``. While views still built correctly, this behaviour
  simplifies debugging.
* :ghissue:`5373`: Fix checksumming in ``couch_file``, consolidate similar
  functions and bring test coverage from 66% to 90%.
* :ghissue:`5367`: Scanner: be more resilient in the face of non-deterministic
  functions.
* :ghissue:`5345`: Scanner: be more resilient in the face of incomplete sample
  data.
* :ghissue:`5344`: Scanner: allow empty doc fields.
* :ghissue:`5341`: Improve Mango test reliability.
* :ghissue:`5337`: Prevent a broken ``mem3`` app from permanently failing
  replication.
* :ghissue:`5334`: Fix QuickJS scanner ``function_clause`` error.
* :ghissue:`5332`: Skip deleted documents in the scanner.
* :ghissue:`5331`: Skip validation for design docs in the scanner.
* :ghissue:`5330`: Prevent inserting illegal design docs via Mango.

Docs
----

* :ghissue:`5433`: Mango: document Nouveau index type.
* :ghissue:`5432`: Add conceptual docs for Mango.
* :ghissue:`5428`: Fix wrong link in example in ``CONTRIBUTING.md``.
* :ghissue:`5400`: Clarify RHEL9 installation caveats.
* :ghissue:`5380`, :ghissue:`5404`: Fix various typos.
* :ghissue:`5338`: Clouseau: document version in ``/_version`` endpoint.
* :ghissue:`5340`, :ghissue:`5412`: Nouveau: document search cleanup API.
* :ghissue:`5316`, :ghissue:`5325`, :ghissue:`5426`, :ghissue:`5442`,
  :ghissue:`5445`: Document various JavaScript engine incompatibilities,
  including SpiderMonkey 1.8.5 vs. newer SpiderMonkey and SpiderMonkey vs.
  QuickJS.
* :ghissue:`5320`, :ghissue:`5374`: Improve auto-lockout feature documentation.
* :ghissue:`5323`: Nouveau: improve install instructions.

Tests
_____

* :ghissue:`5492`: Enable Clouseau testing for FreeBSD
* :ghissue:`5490`: Enable Clouseau testing for MacOS
* :ghissue:`5397`: Fix negative-steps error in Elixir tests.

Builds
------
* :ghissue:`5360`: Use ``brew --prefix`` to find ICU paths on macOS.

Other
-----

There’s always IOPS in the banana stand.
