# packages

pkgId: 8958114a392e5af6d7d90988ec73fbfe654b534109099048d4a2f50ed3061bb9
name: python-bx-py-utils
arch: src
version: 105
epoch: 0
release: 1
summary: Various Python utility functions
url: https://pypi.org/project/bx-py-utils/
rpm_license: MIT
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bx-py-utils-105-1.src.rpm
checksum_type: sha256
description:
  Various Python utility functions

  [![tests](https://github.com/boxine/bx_py_utils/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/boxine/bx_py_utils/actions/workflows/tests.yml)
  [![codecov](https://codecov.io/github/boxine/bx_py_utils/branch/master/graph/badge.svg)](https://app.codecov.io/github/boxine/bx_py_utils)
  [![bx_py_utils @ PyPi](https://img.shields.io/pypi/v/bx_py_utils?label=bx_py_utils%20%40%20PyPi)](https://pypi.org/project/bx_py_utils/)
  [![Python Versions](https://img.shields.io/pypi/pyversions/bx_py_utils)](https://github.com/boxine/bx_py_utils/blob/master/pyproject.toml)
  [![License MIT](https://img.shields.io/pypi/l/bx_py_utils)](https://github.com/boxine/bx_py_utils/blob/master/LICENSE)




  ```bash
  pip install bx_py_utils
  ```




  Here only a simple list about existing utilities.
  Please take a look into the sources and tests for deeper informations.


  [comment]: <> (✂✂✂ auto generated start ✂✂✂)



  * [`anonymize()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/anonymize.py#L17-L47) - Anonymize the given string with special handling for eMail addresses and the possibility to truncate the output.
  * [`anonymize_dict()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/anonymize.py#L50-L75) - Returns a new dict with anonymized values for keys containing one of the given keywords.



  * [`FnmatchExclude()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L203-L212) - Helper for auto doc `exclude_func` that exclude files via fnmatch pattern.
  * [`assert_readme()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L174-L200) - Check and update README file with generate_modules_doc()
  * [`assert_readme_block()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L141-L171) - Check and update README file: Asset that "text_block" is present between the markers.
  * [`generate_modules_doc()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L46-L138) - Generate a list of function/class information via pdoc.
  * [`get_code_location()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L38-L43) - Return start and end line number for an object via inspect.



  * [`ClientSideCertManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/aws/client_side_cert_manager.py#L6-L72) - Helper to manage client-side TLS certificate via AWS Secrets Manager by



  * [`SecretsManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/aws/secret_manager.py#L4-L34) - Access AWS Secrets Manager values



  * [`removeprefix()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/compat.py#L1-L7) - Backport of `removeprefix` from PEP-616 (Python 3.9+)
  * [`removesuffix()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/compat.py#L10-L16) - Backport of `removesuffix` from PEP-616 (Python 3.9+)



  * [`dict_get()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L8-L29) - nested dict `get()`
  * [`dict_list2markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L47-L77) - Convert a list of dictionaries into a markdown table.
  * [`pluck()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L32-L44) - Extract values from a dict, if they are present



  Doc-Write, see: https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/doc_write/README.md




  * [`OverrideEnviron()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/environ.py#L31-L54) - Context manager to change 'os.environ' temporarily.
  * [`cgroup_memory_usage()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/environ.py#L12-L28) - Returns the memory usage of the cgroup the Python interpreter is running in.



  * [`print_exc_plus()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/error_handling.py#L14-L72) - Print traceback information with a listing of all the local variables in each frame.



  * [`EmptyFileError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L18-L21) - Will be raised from get_and_assert_file_size() if a 0-bytes file was found.
  * [`FileError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L12-L15) - Base error class for all 'file_utils' exceptions.
  * [`FileHasher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L79-L104) - Context Manager for generate different hashes from file content while processing a file.
  * [`FileSizeError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L24-L38) - File size is not the same as the expected size.
  * [`NamedTemporaryFile2()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L53-L76) - Generates a temp file with the given filename **without** any random name sequence.
  * [`OverlongFilenameError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L188-L191) - cut_filename() error: The file name can not be shortened, because sterm is to short.
  * [`TempFileHasher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L107-L178) - File like context manager that combines NamedTemporaryFile2 and FileHasher.
  * [`cut_filename()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L194-L234) - Short the file name (and keep the last suffix). Raise OverlongFilenameError if it can't fit.
  * [`get_and_assert_file_size()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L41-L50) - Check file size of given file object. Raise EmptyFileError for empty files or return size
  * [`safe_filename()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L181-L185) - Makes an arbitrary input suitable to be used as a filename.



  * [`filename_matcher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/filename_matcher.py#L6-L25) - Enhance fnmatch that accept a list of patterns.



  * [`introspection_query()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/graphql_introspection.py#L5-L25) - Generate GraphQL introspection query with variable nested depth.



  * [`url_safe_encode()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/hash_utils.py#L13-L22) - Encode bytes into a URL safe string.
  * [`url_safe_hash()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/hash_utils.py#L25-L46) - Generate a URL safe hash with `max_size` from given string/bytes.



  * [`ElementsNotFoundError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L44-L48) - Happens if requested HTML elements cannot be found
  * [`InvalidHtml()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L18-L41) - XMLSyntaxError with better error messages: used in validate_html()
  * [`get_html_elements()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L95-L106) - Returns the selected HTML elements as string
  * [`pretty_format_html()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L87-L92) - Pretty format given HTML document via BeautifulSoup (Needs 'beautifulsoup4' package)
  * [`validate_html()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L51-L73) - Validate a HTML document via XMLParser (Needs 'lxml' package)



  * [`pformat()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/humanize/pformat.py#L5-L16) - Format given object: Try JSON fist and fallback to pformat()



  * [`human_timedelta()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/humanize/time.py#L14-L52) - Converts a time duration into a friendly text representation.



  * [`import_string()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/import_utils.py#L18-L31) - Import a dotted module path and return the attribute/class designated by the last name in the path.



  * [`chunk_iterable()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/iteration.py#L4-L21) - Returns a generator that yields slices of iterable of the given `chunk_size`.



  * [`ChangeCurrentWorkDir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L57-L72) - Context Manager change the "CWD" to an other directory.
  * [`MockCurrentWorkDir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L75-L94) - Context Manager to move the "CWD" to a temp directory.
  * [`assert_is_dir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L7-L15) - Check if given path is a directory
  * [`assert_is_file()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L18-L28) - Check if given path is a file



  * [`processify()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/processify.py#L12-L53) - Decorator to run a function as a process.



  * [`get_pyproject_config()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/pyproject_toml.py#L17-L41) - Get a config section from "pyproject.toml". The path can be optional specify.



  * [`rison_dumps()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/rison.py#L4-L31) - Encode as RISON, a URL-safe encoding format.



  * [`FrameNotFound()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/stack_info.py#L8-L13) - Base class for lookup errors.
  * [`last_frame_outside_path()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/stack_info.py#L16-L44) - Returns the stack frame that is the direct successor of given "file_path".



  * [`compare_sentences()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L68-L89) - Calculates the Levenshtein distance between text1 and text2. With filter functionality.
  * [`ensure_lf()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L130-L140) - Replace line endings to unix-style.
  * [`get_words()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L41-L65) - Extract words from a text. With filter functionality.
  * [`is_uuid()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L105-L127) - Returns True if text is a valid UUID (https://www.rfc-editor.org/rfc/rfc9562#name-uuid-format).
  * [`levenshtein_distance()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L9-L38) - Calculates the Levenshtein distance between two strings.
  * [`startswith_prefixes()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L143-L159) - >>> startswith_prefixes('foobar', prefixes=('foo','bar'))
  * [`truncate()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L162-L187) - Truncates the given string to the given length
  * [`uuid_from_text()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L92-L102) - Generate a UUID instance from the given text in a determinism may via SHA224 hash.



  * [`assert_equal()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L64-L77) - Check if the two objects are the same. Display a nice diff, using `pformat()`
  * [`assert_text_equal()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L80-L95) - Check if the two text strings are the same. Display an error message with a diff.
  * [`pformat_ndiff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L16-L26) - Generate a `ndiff` from two objects, using `pformat()`
  * [`pformat_unified_diff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L47-L61) - Generate a unified diff from two objects, using `pformat()`
  * [`text_ndiff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L8-L13) - Generate a `ndiff` between two text strings.
  * [`text_unified_diff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L40-L44) - Generate a unified diff between two text strings.



  * [`MassContextManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/context_managers.py#L4-L25) - A context manager / decorator that enter/exit a list of mocks.



  * [`parse_dt()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/datetime.py#L5-L25) - Helper for easy generate a `datetime` instance via string.



  * [`DenyAnyRealRequestContextManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/deny_requests.py#L24-L38) - Context manager that denies any request via docket/urllib3. Will raise DenyCallError.
  * [`deny_any_real_request()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/deny_requests.py#L41-L50) - Deny any request via docket/urllib3. Useful for tests, because they should mock all requests.



  * [`FileWatcher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/filesystem_utils.py#L6-L49) - Helper to record which new files have been created.



  * [`NoLogs()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/log_utils.py#L14-L31) - Context manager to Suppress all logger outputs
  * [`RaiseLogUsage()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/log_utils.py#L4-L11) - A log handler, that raise an error on every log output.



  * [`SecretsManagerMock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_aws_secret_manager.py#L1-L16) - Mock for `bx_py_utils.aws.secret_manager.SecretsManager()`



  * [`MockedBoto3Session()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_boto3session.py#L4-L46) - Mock for `boto3.session.Session()`



  * [`MockUUIDGenerator()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_uuid.py#L5-L25) - Helper to mock `uuid.uuid4()` with reproducible results (e.g. for snapshot tests)



  A simple mock for Boto3's S3 modules.

  * [`PseudoS3Client()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mocks3.py#L61-L258) - Simulates a boto3 S3 client object in tests



  * [`RedirectOut()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/redirect.py#L7-L34) - Redirect stdout + stderr into a buffer (with optional strip the output)



  * [`assert_json_requests_mock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L29-L44) - Check the requests mock history. In this case all requests must be JSON.
  * [`assert_json_requests_mock_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L47-L54) - Check requests mock history via snapshot. Accepts only JSON requests.
  * [`assert_requests_mock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L57-L77) - Check the requests mock history. Accept mixed "text" and "JSON".
  * [`assert_requests_mock_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L80-L85) - Check requests mock history via snapshot. Accept mixed "text" and "JSON".



  Assert complex output via auto updated snapshot files with nice diff error messages.

  * [`SnapshotChanged()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L50-L51) - Assertion failed.
  * [`assert_binary_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L366-L407) - Assert binary data via snapshot file
  * [`assert_html_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L301-L350) - Assert "html" string via snapshot file with validate and pretty format
  * [`assert_py_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L252-L298) - Assert complex python objects vio PrettyPrinter() snapshot file.
  * [`assert_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L202-L249) - Assert given data serialized to JSON snapshot file.
  * [`assert_text_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L154-L199) - Assert "text" string via snapshot file
  * [`get_snapshot_file()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L124-L151) - Generate a file path use stack information to fill not provided path components.



  * [`MockTimeMonotonicGenerator()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/time.py#L1-L19) - Helper to mock `time.monotonic()` in tests.



  * [`BaseDocTests()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/unittest_utils.py#L40-L91) - Helper to include all doctests in unittests, without change unittest setup. Just add a normal TestCase.
  * [`assert_no_flat_tests_functions()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/unittest_utils.py#L16-L30) - Check if there exists normal test functions (That will not be executed by normal unittests)



  * [`FreezeXlsxTimes()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L84-L97) - Context manager / decorator intended to freeze timestamps of xlsx files creation by e.g.: openpyxl.
  * [`generate_xlsx_md_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L68-L81) - Generate a markdown snapshot of a XLSX: Display ZIP info + Sheets content as Markdown.
  * [`xlsx2dict()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L27-L51) - Convert a XLSX file content into a dictionary: Every sheet is a key, and the value is a list of dictionaries.
  * [`xlsx2markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L54-L65) - Convert all Sheets of a XLSX into markdown tables.



  * [`FreezeZipFileDatetime()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L15-L30) - Context manager / decorator to freezes the modification time of files written to a zip file.
  * [`zip_info()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L44-L68) - Generates similar information than `unzip -v`: Yields ZipFileInfo for each file in the zip file.
  * [`zip_info_markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L71-L75) - Generates a markdown representation of the zip file content. Similar to `unzip -v` output.



  * [`cutout()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/text_tools.py#L1-L36) - Mark a point in a long text by line no + column with context lines around.

  [comment]: <> (✂✂✂ auto generated end ✂✂✂)




  Quick hint about snapshot. If you have many snapshots in your project and you need to change many with a code change, then you can run the tests without a snapshot change leading to an error, by set `RAISE_SNAPSHOT_ERRORS=0` in your environment.

  e.g.:

  ```bash
  RAISE_SNAPSHOT_ERRORS=0 python3 -m unittest
  ```

  Renew all snapshot files with:
  ```bash
  make update-test-snapshot-files
  ```





  We split `bx_py_utils` and moved all Django related utilities into the separated project:

  * https://github.com/boxine/bx_django_utils

  So, `bx_py_utils` is better usable in non-Django projects, because Django will not installed as decency of "bx_py_utils"




  To start developing e.g.:

  ```bash
  ~$ git clone https://github.com/boxine/bx_py_utils.git
  ~$ cd bx_py_utils
  ~/bx_py_utils$ make
  help                       List all commands
  install-base-req           Install needed base packages via apt
  install                    Install the project in a Python virtualenv
  update-requirements        Update requirements
  lint                       Run code formatters and linter
  fix-code-style             Fix code formatting
  tox-listenvs               List all tox test environments
  tox                        Run tests via tox with all environments
  test                       Run tests
  coverage                   Run tests with coverage
  update-test-snapshot-files Update all snapshot files (by remove and recreate all snapshot files)
  mypy                       Run mypy
  safety                     Run https://github.com/pyupio/safety
  publish                    Release new version to PyPi
  clean                      Remove created files from the test project
  ```




  * Increase verion number in `bx_py_utils/__init__.py`
  * Create pull request
  * After merge, call: `make publish`




  [MIT](https://github.com/boxine/bx_py_utils/blob/master/LICENSE). Patches welcome!



  We’ve been rethinking the listening experience for kids and have created an ecosystem where haptic and listening experience are combined via smart technology - the Toniebox.

  We are constantly looking for engineers to join our team in different areas. If you’d be interested in contributing to our platform, have a look at: https://tonies.com/jobs/



  * https://pypi.org/project/bx-py-utils/


pkgId: 7d84ed033e35ed7e3131df806cb136dc043e10906a5496452993bc64c6cd7940
name: python-bx-py-utils
arch: src
version: 106
epoch: 0
release: 1
summary: Various Python utility functions
url: https://pypi.org/project/bx-py-utils/
rpm_license: MIT
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bx-py-utils-106-1.src.rpm
checksum_type: sha256
description:
  Various Python utility functions

  [![tests](https://github.com/boxine/bx_py_utils/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/boxine/bx_py_utils/actions/workflows/tests.yml)
  [![codecov](https://codecov.io/github/boxine/bx_py_utils/branch/master/graph/badge.svg)](https://app.codecov.io/github/boxine/bx_py_utils)
  [![bx_py_utils @ PyPi](https://img.shields.io/pypi/v/bx_py_utils?label=bx_py_utils%20%40%20PyPi)](https://pypi.org/project/bx_py_utils/)
  [![Python Versions](https://img.shields.io/pypi/pyversions/bx_py_utils)](https://github.com/boxine/bx_py_utils/blob/master/pyproject.toml)
  [![License MIT](https://img.shields.io/pypi/l/bx_py_utils)](https://github.com/boxine/bx_py_utils/blob/master/LICENSE)




  ```bash
  pip install bx_py_utils
  ```




  Here only a simple list about existing utilities.
  Please take a look into the sources and tests for deeper informations.


  [comment]: <> (✂✂✂ auto generated start ✂✂✂)



  * [`anonymize()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/anonymize.py#L17-L47) - Anonymize the given string with special handling for eMail addresses and the possibility to truncate the output.
  * [`anonymize_dict()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/anonymize.py#L50-L75) - Returns a new dict with anonymized values for keys containing one of the given keywords.



  * [`FnmatchExclude()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L203-L212) - Helper for auto doc `exclude_func` that exclude files via fnmatch pattern.
  * [`assert_readme()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L174-L200) - Check and update README file with generate_modules_doc()
  * [`assert_readme_block()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L141-L171) - Check and update README file: Asset that "text_block" is present between the markers.
  * [`generate_modules_doc()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L46-L138) - Generate a list of function/class information via pdoc.
  * [`get_code_location()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/auto_doc.py#L38-L43) - Return start and end line number for an object via inspect.



  * [`ClientSideCertManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/aws/client_side_cert_manager.py#L6-L72) - Helper to manage client-side TLS certificate via AWS Secrets Manager by



  * [`SecretsManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/aws/secret_manager.py#L4-L34) - Access AWS Secrets Manager values



  * [`removeprefix()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/compat.py#L1-L7) - Backport of `removeprefix` from PEP-616 (Python 3.9+)
  * [`removesuffix()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/compat.py#L10-L16) - Backport of `removesuffix` from PEP-616 (Python 3.9+)



  * [`dict_get()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L8-L29) - nested dict `get()`
  * [`dict_list2markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L47-L77) - Convert a list of dictionaries into a markdown table.
  * [`pluck()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/dict_utils.py#L32-L44) - Extract values from a dict, if they are present



  Doc-Write, see: https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/doc_write/README.md




  * [`OverrideEnviron()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/environ.py#L31-L54) - Context manager to change 'os.environ' temporarily.
  * [`cgroup_memory_usage()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/environ.py#L12-L28) - Returns the memory usage of the cgroup the Python interpreter is running in.



  * [`print_exc_plus()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/error_handling.py#L14-L72) - Print traceback information with a listing of all the local variables in each frame.



  * [`EmptyFileError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L18-L21) - Will be raised from get_and_assert_file_size() if a 0-bytes file was found.
  * [`FileError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L12-L15) - Base error class for all 'file_utils' exceptions.
  * [`FileHasher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L79-L104) - Context Manager for generate different hashes from file content while processing a file.
  * [`FileSizeError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L24-L38) - File size is not the same as the expected size.
  * [`NamedTemporaryFile2()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L53-L76) - Generates a temp file with the given filename **without** any random name sequence.
  * [`OverlongFilenameError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L188-L191) - cut_filename() error: The file name can not be shortened, because sterm is to short.
  * [`TempFileHasher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L107-L178) - File like context manager that combines NamedTemporaryFile2 and FileHasher.
  * [`cut_filename()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L194-L234) - Short the file name (and keep the last suffix). Raise OverlongFilenameError if it can't fit.
  * [`get_and_assert_file_size()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L41-L50) - Check file size of given file object. Raise EmptyFileError for empty files or return size
  * [`safe_filename()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/file_utils.py#L181-L185) - Makes an arbitrary input suitable to be used as a filename.



  * [`filename_matcher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/filename_matcher.py#L6-L25) - Enhance fnmatch that accept a list of patterns.



  * [`introspection_query()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/graphql_introspection.py#L5-L25) - Generate GraphQL introspection query with variable nested depth.



  * [`url_safe_encode()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/hash_utils.py#L13-L22) - Encode bytes into a URL safe string.
  * [`url_safe_hash()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/hash_utils.py#L25-L46) - Generate a URL safe hash with `max_size` from given string/bytes.



  * [`ElementsNotFoundError()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L44-L48) - Happens if requested HTML elements cannot be found
  * [`InvalidHtml()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L18-L41) - XMLSyntaxError with better error messages: used in validate_html()
  * [`get_html_elements()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L95-L106) - Returns the selected HTML elements as string
  * [`pretty_format_html()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L87-L92) - Pretty format given HTML document via BeautifulSoup (Needs 'beautifulsoup4' package)
  * [`validate_html()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/html_utils.py#L51-L73) - Validate a HTML document via XMLParser (Needs 'lxml' package)



  * [`pformat()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/humanize/pformat.py#L5-L16) - Format given object: Try JSON fist and fallback to pformat()



  * [`human_timedelta()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/humanize/time.py#L14-L52) - Converts a time duration into a friendly text representation.



  * [`import_string()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/import_utils.py#L18-L31) - Import a dotted module path and return the attribute/class designated by the last name in the path.



  * [`chunk_iterable()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/iteration.py#L4-L21) - Returns a generator that yields slices of iterable of the given `chunk_size`.



  * [`ChangeCurrentWorkDir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L57-L72) - Context Manager change the "CWD" to an other directory.
  * [`MockCurrentWorkDir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L75-L94) - Context Manager to move the "CWD" to a temp directory.
  * [`assert_is_dir()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L7-L15) - Check if given path is a directory
  * [`assert_is_file()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/path.py#L18-L28) - Check if given path is a file



  * [`processify()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/processify.py#L12-L53) - Decorator to run a function as a process.



  * [`get_pyproject_config()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/pyproject_toml.py#L17-L41) - Get a config section from "pyproject.toml". The path can be optional specify.



  * [`rison_dumps()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/rison.py#L4-L31) - Encode as RISON, a URL-safe encoding format.



  * [`FrameNotFound()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/stack_info.py#L8-L13) - Base class for lookup errors.
  * [`last_frame_outside_path()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/stack_info.py#L16-L44) - Returns the stack frame that is the direct successor of given "file_path".



  * [`compare_sentences()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L68-L89) - Calculates the Levenshtein distance between text1 and text2. With filter functionality.
  * [`ensure_lf()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L130-L140) - Replace line endings to unix-style.
  * [`get_words()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L41-L65) - Extract words from a text. With filter functionality.
  * [`is_uuid()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L105-L127) - Returns True if text is a valid UUID (https://www.rfc-editor.org/rfc/rfc9562#name-uuid-format).
  * [`levenshtein_distance()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L9-L38) - Calculates the Levenshtein distance between two strings.
  * [`startswith_prefixes()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L143-L159) - >>> startswith_prefixes('foobar', prefixes=('foo','bar'))
  * [`truncate()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L162-L187) - Truncates the given string to the given length
  * [`uuid_from_text()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/string_utils.py#L92-L102) - Generate a UUID instance from the given text in a determinism may via SHA224 hash.



  * [`assert_equal()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L64-L77) - Check if the two objects are the same. Display a nice diff, using `pformat()`
  * [`assert_text_equal()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L80-L95) - Check if the two text strings are the same. Display an error message with a diff.
  * [`pformat_ndiff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L16-L26) - Generate a `ndiff` from two objects, using `pformat()`
  * [`pformat_unified_diff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L47-L61) - Generate a unified diff from two objects, using `pformat()`
  * [`text_ndiff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L8-L13) - Generate a `ndiff` between two text strings.
  * [`text_unified_diff()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/assertion.py#L40-L44) - Generate a unified diff between two text strings.



  * [`MassContextManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/context_managers.py#L9-L36) - A context manager / decorator that enter/exit a list of mocks.
  * [`MassContextManagerExceptions()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/context_managers.py#L4-L6) - Common base class for all non-exit exceptions.



  * [`parse_dt()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/datetime.py#L5-L25) - Helper for easy generate a `datetime` instance via string.



  * [`DenyAnyRealRequestContextManager()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/deny_requests.py#L24-L38) - Context manager that denies any request via docket/urllib3. Will raise DenyCallError.
  * [`deny_any_real_request()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/deny_requests.py#L41-L50) - Deny any request via docket/urllib3. Useful for tests, because they should mock all requests.



  * [`FileWatcher()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/filesystem_utils.py#L6-L49) - Helper to record which new files have been created.



  * [`NoLogs()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/log_utils.py#L14-L31) - Context manager to Suppress all logger outputs
  * [`RaiseLogUsage()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/log_utils.py#L4-L11) - A log handler, that raise an error on every log output.



  * [`SecretsManagerMock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_aws_secret_manager.py#L1-L16) - Mock for `bx_py_utils.aws.secret_manager.SecretsManager()`



  * [`MockedBoto3Session()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_boto3session.py#L4-L46) - Mock for `boto3.session.Session()`



  * [`MockUUIDGenerator()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mock_uuid.py#L5-L25) - Helper to mock `uuid.uuid4()` with reproducible results (e.g. for snapshot tests)



  A simple mock for Boto3's S3 modules.

  * [`PseudoS3Client()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/mocks3.py#L61-L258) - Simulates a boto3 S3 client object in tests



  * [`RedirectOut()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/redirect.py#L7-L34) - Redirect stdout + stderr into a buffer (with optional strip the output)



  * [`assert_json_requests_mock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L29-L44) - Check the requests mock history. In this case all requests must be JSON.
  * [`assert_json_requests_mock_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L47-L54) - Check requests mock history via snapshot. Accepts only JSON requests.
  * [`assert_requests_mock()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L57-L77) - Check the requests mock history. Accept mixed "text" and "JSON".
  * [`assert_requests_mock_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/requests_mock_assertion.py#L80-L85) - Check requests mock history via snapshot. Accept mixed "text" and "JSON".



  Assert complex output via auto updated snapshot files with nice diff error messages.

  * [`SnapshotChanged()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L50-L51) - Assertion failed.
  * [`assert_binary_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L366-L407) - Assert binary data via snapshot file
  * [`assert_html_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L301-L350) - Assert "html" string via snapshot file with validate and pretty format
  * [`assert_py_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L252-L298) - Assert complex python objects vio PrettyPrinter() snapshot file.
  * [`assert_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L202-L249) - Assert given data serialized to JSON snapshot file.
  * [`assert_text_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L154-L199) - Assert "text" string via snapshot file
  * [`get_snapshot_file()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/snapshot.py#L124-L151) - Generate a file path use stack information to fill not provided path components.



  * [`MockTimeMonotonicGenerator()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/time.py#L1-L19) - Helper to mock `time.monotonic()` in tests.



  * [`BaseDocTests()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/unittest_utils.py#L40-L91) - Helper to include all doctests in unittests, without change unittest setup. Just add a normal TestCase.
  * [`assert_no_flat_tests_functions()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/unittest_utils.py#L16-L30) - Check if there exists normal test functions (That will not be executed by normal unittests)



  * [`FreezeXlsxTimes()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L84-L97) - Context manager / decorator intended to freeze timestamps of xlsx files creation by e.g.: openpyxl.
  * [`generate_xlsx_md_snapshot()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L68-L81) - Generate a markdown snapshot of a XLSX: Display ZIP info + Sheets content as Markdown.
  * [`xlsx2dict()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L27-L51) - Convert a XLSX file content into a dictionary: Every sheet is a key, and the value is a list of dictionaries.
  * [`xlsx2markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/xlsx.py#L54-L65) - Convert all Sheets of a XLSX into markdown tables.



  * [`FreezeZipFileDatetime()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L15-L30) - Context manager / decorator to freezes the modification time of files written to a zip file.
  * [`zip_info()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L44-L68) - Generates similar information than `unzip -v`: Yields ZipFileInfo for each file in the zip file.
  * [`zip_info_markdown()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/test_utils/zip_file_utils.py#L71-L75) - Generates a markdown representation of the zip file content. Similar to `unzip -v` output.



  * [`cutout()`](https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/text_tools.py#L1-L36) - Mark a point in a long text by line no + column with context lines around.

  [comment]: <> (✂✂✂ auto generated end ✂✂✂)




  Quick hint about snapshot. If you have many snapshots in your project and you need to change many with a code change, then you can run the tests without a snapshot change leading to an error, by set `RAISE_SNAPSHOT_ERRORS=0` in your environment.

  e.g.:

  ```bash
  RAISE_SNAPSHOT_ERRORS=0 python3 -m unittest
  ```

  Renew all snapshot files with:
  ```bash
  make update-test-snapshot-files
  ```





  We split `bx_py_utils` and moved all Django related utilities into the separated project:

  * https://github.com/boxine/bx_django_utils

  So, `bx_py_utils` is better usable in non-Django projects, because Django will not installed as decency of "bx_py_utils"




  To start developing e.g.:

  ```bash
  ~$ git clone https://github.com/boxine/bx_py_utils.git
  ~$ cd bx_py_utils
  ~/bx_py_utils$ make
  help                       List all commands
  install-base-req           Install needed base packages via apt
  install                    Install the project in a Python virtualenv
  update-requirements        Update requirements
  lint                       Run code formatters and linter
  fix-code-style             Fix code formatting
  tox-listenvs               List all tox test environments
  tox                        Run tests via tox with all environments
  test                       Run tests
  coverage                   Run tests with coverage
  update-test-snapshot-files Update all snapshot files (by remove and recreate all snapshot files)
  mypy                       Run mypy
  safety                     Run https://github.com/pyupio/safety
  publish                    Release new version to PyPi
  clean                      Remove created files from the test project
  ```




  * Increase verion number in `bx_py_utils/__init__.py`
  * Create pull request
  * After merge, call: `make publish`




  [MIT](https://github.com/boxine/bx_py_utils/blob/master/LICENSE). Patches welcome!



  We’ve been rethinking the listening experience for kids and have created an ecosystem where haptic and listening experience are combined via smart technology - the Toniebox.

  We are constantly looking for engineers to join our team in different areas. If you’d be interested in contributing to our platform, have a look at: https://tonies.com/jobs/



  * https://pypi.org/project/bx-py-utils/


pkgId: 70f1f3a45c4a9cc66cfc091027ad39b36b4dc2a786d656f40857674a51f89557
name: python-byAd12-library
arch: src
version: 1.2.1
epoch: 0
release: 1
summary: Multi function library.
url: https://github.com/byAd12/byAd12-Library.git
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byAd12-library-1.2.1-1.src.rpm
checksum_type: sha256
description:
  **byAd12-library** helps you doing a ICMP (ping) attack to a specific target.
  Feel free to open an issue or a pull request in the repository.



  - **PyPi**: [https://pypi.org/project/byAd12-library/](https://pypi.org/project/byAd12-library/)
  - **Installation**:
    ```bash
    pip install byAd12-library
    ```



  - **Website**: [https://byAd12.pages.dev](https://byAd12.pages.dev)
  - **Email**: [adgimenezp@gmail.com](mailto:adgimenezp@gmail.com)



  - `Ping_Flood_(IPv4)`
  Sends massive pings (ICMP messages) to a target (local or server) using IPv4.

  - `byAd12_Info_()`
  Provides information about the library.

  - `david_()`
  Adds 2 points ("..") at the end of the string.



  - **pip install threading**
  - **pip install ping3**



  ```python
  from byAd12_library import Ping_Flood_, byAd12_Info_, david_


  Ping_Flood_("192.168.1.1") # Send massive pings

  byAd12_Info_() # Get information about the library

  david_("text") # Adds 2 points ("..") at the end of the string
  ```


pkgId: 83ea8857c0090add31598d5c924189c26cf82a50f6518320f5535bb7cd0a4c8e
name: python-bydhvs
arch: src
version: 0.3.2
epoch: 0
release: 1
summary: A module for communicating with the BYD HVS Battery system
url: https://pypi.org/project/bydhvs/
rpm_license: GPL-3.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bydhvs-0.3.2-1.src.rpm
checksum_type: sha256
description:
  The module is available from the [Python Package Index](https://pypi.python.org/pypi)
      $ pip3 install bydhvs
  On a Fedora-based system or a CentOS/RHEL host with EPEL.
      $ sudo dnf -y install python3-bydhvs


pkgId: a045f7fa7310090104e60c1c62ad4e8b2320f73e9c4daa66d1febf6a052ab696
name: python-byma
arch: src
version: 0.1.11
epoch: 0
release: 1
summary: A Numerical Mathematics Packages with an easier and clean user interface.
url: https://gitlab.com/ByteMath/python/byma
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byma-0.1.11-1.src.rpm
checksum_type: sha256
description:
  ![PyPI - Version](https://img.shields.io/pypi/v/byma?style=plastic&label=ByMa&labelColor=green&color=blue&link=https%3A%2F%2Fbytemath.gitlab.io%2Fpython%2FByMa%2Findex.html)
  [![PyPI Downloads](https://img.shields.io/pypi/dm/label=PyPI%20downloads)](
  https://pypi.org/project/byma/)
  ![Libraries.io dependency status for specific release](https://img.shields.io/librariesio/release/pypi/numpy/1.1.1?style=plastic&logo=numpy&label=NumPy&labelColor=blue&link=https%3A%2F%2Fnumpy.org%2F)
  ![Libraries.io dependency status for specific release](https://img.shields.io/librariesio/release/pypi/scipy/1.0.0?style=plastic&logo=scipy&label=SciPy&labelColor=light%20blue&link=https%3A%2F%2Fscipy.org%2F)
  ![GitLab Last Commit](https://img.shields.io/gitlab/last-commit/ByteMath%2Fpython%2FByMa?gitlab_url=https%3A%2F%2Fgitlab.com%2FByteMath%2Fpython%2FByMa&style=plastic)
  ![Static Badge](https://img.shields.io/badge/Docs-Read?style=plastic&label=Read&color=purple&link=https%3A%2F%2Fbytemath.gitlab.io%2Fpython%2FByMa%2Findex.html)



  ByMa is a Python package designed to facilitate numerical mathematics tasks by implementing a range of standard methods, from iterative techniques to bifurcation methods. Noted for its simplicity, clarity, and efficiency, ByMa aims to enhance the learning experience for newcomers to numerical mathematics while streamlining the implementation and utilization of popular scientific libraries such as NumPy, SciPy, and Matplotlib.




  ByMa is best installed in a [virtual environment](https://docs.python.org/3/library/venv.html).
  We state the most common steps for creating and using a virtual environment here.
  Refer to the documentation for more details.

  To create a virtual environment run
  ```
  python3 -m venv /path/to/new/virtual/environment
  ```

  and to activate the virtual environment, run
  ```
  source /path/to/new/virtual/environment/bin/activate
  ```

  After this, we can install ByMa from the pip package by using
  ```
  pip install byma
  ```

  In case the dependencies are not installed, you can run
  ```
  pip install -e .
  ```



  ByMa consists of several subpackages, each serving a distinct purpose:

  - **nuby**: Numerical Bifurcation Analysis Tools
  - **iteral**: Tools for iterative algorithms
  - **pyplot**: Tools for plotting functions
  - **interface**: Interface functionalities



  * Lorenzo Zambelli [website](https://lorenzozambelli.it)


pkgId: a48e32e4a8199fe821630a9d980641e01649b1d2bff33604a17cfab1ee383556
name: python-bynder-sdk
arch: src
version: 2.0.0
epoch: 0
release: 1
summary: Bynder SDK can be used to speed up the integration of Bynder in Python
url: https://bynder.com
rpm_license: MIT
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bynder-sdk-2.0.0-1.src.rpm
checksum_type: sha256
description:
  ![Tests](https://github.com/Bynder/bynder-python-sdk/workflows/Tests/badge.svg)
  ![Publish](https://github.com/Bynder/bynder-python-sdk/workflows/Publish/badge.svg)
  [![Coverage Status](https://coveralls.io/repos/github/Bynder/bynder-python-sdk/badge.svg?branch=master)](https://coveralls.io/github/Bynder/bynder-python-sdk?branch=master)
  ![PyPI](https://img.shields.io/pypi/v/bynder-sdk)
  ![PyPI - Downloads](https://img.shields.io/pypi/dm/bynder-sdk?color=orange)
  The main goal of this SDK is to speed up the integration of Bynder
  customers who use Python. Making it easier to connect to the Bynder API
  (<https://bynder.docs.apiary.io>) and execute requests on it.
  _**Note:** As of version 1.0.0 this SDK now uses OAuth 2.0. For the last
  version using OAuth 1.0a please refer to
  [version 0.0.6](https://github.com/Bynder/bynder-python-sdk/tree/0.0.6)_.


pkgId: 547554284fc3bc4a8f95149cc4a0dadad663317b05ff5c51914e444a706c1a7e
name: python-byotrack
arch: src
version: 1.1.0
epoch: 0
release: 1
summary: Biological particle tracking with Python
url: https://github.com/raphaelreme/byotrack
rpm_license: MIT
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byotrack-1.1.0-1.src.rpm
checksum_type: sha256
description:
  [![Lint and Test](https://github.com/raphaelreme/byotrack/actions/workflows/tests.yml/badge.svg)](https://github.com/raphaelreme/byotrack/actions/workflows/tests.yml)
  [![Documentation Status](https://readthedocs.org/projects/byotrack/badge/?version=latest)](https://byotrack.readthedocs.io/en/latest/?badge=latest)

  ![pipeline](docs/source/images/tracking.svg)

  **ByoTrack** is a Python library that enables tracking of biological object in videos (2D or 3D).

  Many bioimage informatics tools already implement their own tracking tools (Icy [1], ImageJ [6], ...) but most of them are implemented in Java which makes it difficult for non-Java developers to experiment with the code. It is also difficult to integrate deep learning algorithms (mainly developed in Python) into these software.

  We provide a unified python API for tracking that can be easily extended with new (and old) algorithms. We also provide implementations of well-known algorithms following our API. ByoTrack is based on numpy, pytorch and numba allowing fast computations with the access to the full python ecosystem.

  Overview:
  * Video
      * Able to read most classical format (supported by opencv) + tiff
      * Supports 2D and 3D.
  * Particle Tracking
      * MultiStepTracker (Detect / Link / Refine)
  * Particle Detections
      * Wavelet Detector [2] (Similar as the one in Icy [1] but coded in pytorch)
      * Stardist [3] (Inference only. Training should be done with the [official implementation](https://github.com/stardist/stardist))
  * Particle Linking
      * Nearest neighbors using optical flow, kalman filters or both (KOFT) [9]
      * EMHT [4] (Wraps the implementation in Icy [1], requires Icy to be installed)
      * u-track / TrackMate [7] (Wraps the TrackMate [6, 8] implementation in ImageJ/Fiji, requires Fiji to be installed)
  * Tracks Refining
      * Cleaning
      * EMC2 [5]: Track stitching (gap closing)
      * Interpolate missing positions
  * Optical Flow
      * Support for Open-CV and Scikit-Image algorithms. Can be used for particle linking, track stitching
      and interpolations.




  ```bash
  $ pip install byotrack
  ```

  Some tracker implementations require additional dependencies that are not installed with the library, to use them you need to install their dependencies on your own.
  Here is the complete list:


  - StarDistDetector
      - stardist (+ tensorflow): [Install stardist](https://github.com/stardist/stardist#installation)
  - KalmanLinker & KOFTLinker
      - torch_kf: [Install torch-kf](https://github.com/raphaelreme/torch-kf#install)
  - IcyEMHTLinker
      - Icy: [Download Icy](https://icy.bioimageanalysis.org/download/)
      - Spot Tracking Blocks plugin: [Install an Icy plugin](https://icy.bioimageanalysis.org/tutorial/how-to-install-an-icy-plugin/)
  - TrackMateLinker
      - Fiji: [Download Fiji](https://imagej.net/downloads)
      - tifffile: [Install tifffile](https://github.com/cgohlke/tifffile#quickstart)
  - SkimageOpticalFlow
      - scikit-image: [Install scikit-image](https://scikit-image.org/docs/stable/user_guide/install.html)

  For visualization, with `byotrack.visualize` module you need to [install matplotlib](https://matplotlib.org/stable/install/index.html).



  ```python
  import byotrack


  from byotrack.implementation.detector.wavelet import WaveletDetector
  from byotrack.implementation.linker.icy_emht import IcyEMHTLinker
  from byotrack.implementation.refiner.cleaner import Cleaner
  from byotrack.implementation.refiner.stitching import EMC2Stitcher


  video = byotrack.Video(video_path)
  transform_config = VideoTransformConfig(aggregate=True, normalize=True, q_min=0.01, q_max=0.999)
  video.set_transform(transform_config)





  detector = WaveletDetector(scale=1, k=3.0, min_area=5)



  linker = IcyEMHTLinker(icy_path)



  refiners = []
  if True:
      refiners.append(Cleaner(5, 3.5))  # Split tracks on position jumps and drop small ones
      refiners.append(EMC2Stitcher())  # Merge tracks if they track the same particle

  tracker = byotrack.MultiStepTracker(detector, linker, refiners)


  tracks = tracker.run(video)


  byotrack.Track.save(tracks, output_path)
  ```

  Please refer to the [official documentation](https://byotrack.readthedocs.io/en/latest/).



  In coming...



  ```bibtex
  @article{hanson2024automatic,
    title={Automatic monitoring of neural activity with single-cell resolution in behaving Hydra},
    author={Hanson, Alison and Reme, Raphael and Telerman, Noah and Yamamoto, Wataru and Olivo-Marin, Jean-Christophe and Lagache, Thibault and Yuste, Rafael},
    journal={Scientific Reports},
    volume={14},
    number={1},
    pages={5083},
    year={2024},
    publisher={Nature Publishing Group UK London}
  }
  ```




  * [1] F. De Chaumont, S. Dallongeville, N. Chenouard, et al., "Icy:
        an open bioimage informatics platform for extended reproducible
        research", Nature methods, vol. 9, no. 7, pp. 690–696, 2012.
  * [2] J.-C. Olivo-Marin, "Extraction of spots in biological images
        using multiscale products", Pattern Recognition, vol. 35, no. 9,
        pp. 1989–1996, 2002.
  * [3] U. Schmidt, M. Weigert, C. Broaddus, and G. Myers, "Cell detection
        with star-convex polygons", in Medical Image Computing and
        Computer Assisted Intervention–MICCAI 2018: 21st International
        Conference, Granada, Spain, September 16-20, 2018, Proceedings,
        Part II 11. Springer, 2018, pp. 265–273.
  * [4] N. Chenouard, I. Bloch, and J.-C. Olivo-Marin, "Multiple hypothesis
        tracking for cluttered biological image sequences",
        IEEE transactions on pattern analysis and machine intelligence,
        vol. 35, no. 11, pp. 2736–3750, 2013.
  * [5] T. Lagache, A. Hanson, J. Perez-Ortega, et al., "Tracking calcium
        dynamics from individual neurons in behaving animals",
        PLoS computational biology, vol. 17, pp. e1009432, 10 2021.
  * [6] J. Schindelin, I. Arganda-Carreras, E. Frise, et al., "Fiji:
        an open-source platform for biological-image analysis", Nature
        Methods, 9(7), 676–682, 2012.
  * [7] K. Jaqaman, D. Loerke, M. Mettlen, et al., "Robust single-particle
        tracking in live-cell time-lapse sequences.", Nature Methods, 5(8),
        695–702, 2008.
  * [8] J.-Y. Tinevez, N. Perry, J. Schindelin, et al., "TrackMate: An
        open and extensible platform for single-particle tracking.",
        Methods, 115, 80–90, 2017.
  * [9] R. Reme, A. Newson, E. Angelini, J.-C. Olivo-Marin and T. Lagache,
        "Particle tracking in biological images with optical-flow enhanced
        kalman filtering", in International Symposium on Biomedical Imaging
        (ISBI2024).


pkgId: 60147efa9d986344293dacda1de424ec7fdf3d2d71b5974e0d9ac28dd3e86a73
name: python-bypass-url-parser
arch: src
version: 0.4.4
epoch: 0
release: 1
summary: Default template for PDM package
url: https://pypi.org/project/bypass-url-parser/
rpm_license: AGPL-3.0-or-later
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bypass-url-parser-0.4.4-1.src.rpm
checksum_type: sha256
description:
  [![PyPI - Version](https://img.shields.io/pypi/v/bypass-url-parser)](https://pypi.org/project/bypass-url-parser/) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/bypass-url-parser)](https://pypi.org/project/bypass-url-parser/) [![PyPI - License](https://img.shields.io/pypi/l/bypass-url-parser)](https://github.com/laluka/bypass-url-parser/blob/main/LICENSE) [![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm.fming.dev)

  Tool that tests `MANY` url bypasses to reach a `40X protected page`.

  If you wonder why this code is `nothing but a dirty curl wrapper`, here's why:

  - Most of the python requests do url/path/parameter encoding/decoding, and I hate this.
  - If I submit raw chars, I want raw chars to be sent.
  - If I send a weird path, I want it weird, not normalized.

  This is `surprisingly hard` to achieve in python without loosing all of the lib goodies like parsing, ssl/tls encapsulation and so on. \
  So, be like me, use `curl as a backend`, it's gonna be just fine.

  Also, this tool can be used as a library, see [lib_sample_usage.py](lib_sample_usage.py)



  We recommend using `pipx` to install this tool:

  ```bash
  pipx install bypass-url-parser

  pipx install git+https://github.com/laluka/bypass-url-parser
  ```

  Alternatively, you can use `pip`:

  ```bash
  pip install bypass-url-parser
  ```



  ```
  Bypass Url Parser, made with love by @TheLaluka
  A tool that tests MANY url bypasses to reach a 40X protected page.

  Usage:
      bypass-url-parser (-u <URL> | -R <file>) [-m <mode>] [-o <outdir>] [-S <level>] [ (-H <header>)...] [-r <num>]
                        [-s <ip>] [--spoofip-replace] [-p <port>] [--spoofport-replace] [-t <threads>] [-T <timeout>]
                        [--request-tls] [--jsonl] [--dump-payloads] [-x <proxy_url>] [-v | -d | -dd]

  Program options:
      -u, --url <URL>           URL (path is optional) to run bypasses against
      -R, --request <file>      Load HTTP raw request from a file
      -H, --header <header>     Header(s) to use, format: "Cookie: can_i_haz=fire"
      -m, --mode <mode>         Bypass modes. See 'Bypasser.BYPASS_MODES' in code [Default: all]
      -o, --outdir <outdir>     Output directory for results
      -x, --proxy <proxy_url>   Set a proxy in the format http://proxy_ip:port.
      -S, --save-level <level>  Save results level. From 0 (DISABLE) to 3 (FULL) [Default: 2]
      -s, --spoofip <ip>        IP(s) to inject in ip-specific headers
      -p, --spoofport <port>    Port(s) to inject in port-specific headers
      -r, --retry <num>         Retry attempts of failed requests. Set 0 to disable all retry tentatives [Default: 1]
      -t, --threads <threads>   Scan with N parallel threads [Default: 1]
      -T, --timeout <timeout>   Request times out after N seconds [Default: 5]

  General options:
      -h, --help                Show help, you are here :)
      -v, --verbose             Verbose output
      -d, --debug               Show more details like curl commands generated by this tool
      -dd, --debug              Print Debug level 2 (with all classes debug_class output)
      -V, --version             Show version info

  Misc options:
      --spoofip-replace         Disable list of default internal IPs in 'http_headers_ip' bypass mode
      --spoofport-replace       Disable list of default internal ports in 'http_headers_port' bypass mode
      --request-tls             Force usage of TLS/HTTPS for the request load with the '-R, --request' option
      --dump-payloads           Print all payloads (curls) generated by this tool.
      --jsonl                   Print results in JSON lines format (pipe command output)

  Examples:
      bypass-url-parser -u "http://127.0.0.1/juicy_403_endpoint/" -s 8.8.8.8 -d
      bypass-url-parser -u /path/urls -t 30 -T 5 -H "Cookie: me_iz=admin" -H "User-agent: test"
      bypass-url-parser -R /path/request_file --request-tls -m "mid_paths, end_paths"
  ```



  ```bash
  bypass-url-parser -u http://127.0.0.1:8000/foo/bar
  2022-08-09 14:52:40 lalu-perso bup[361559] WARNING Trying to bypass 'http://127.0.0.1:8000/foo/bar' url (3213 payloads)...
  2022-08-09 14:52:40 lalu-perso bup[361559] INFO Doing: 50 / 3213
  [...]
  2022-08-09 14:52:54 lalu-perso bup[361559] INFO Doing: 3200 / 3213
  2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (1/3) the '16' failed curl commands with 10 threads and 10s timeout
  2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (2/3) the '16' failed curl commands with 5 threads and 20s timeout
  2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (3/3) the '16' failed curl commands with 1 threads and 30s timeout
  2022-08-09 14:52:55 lalu-perso bup[361559] INFO
  [#####] [bypass_method] [payload] => [status_code] [content_type] [content_length] [lines_count] [word_counts] [title] [server] [redirect_url]
  [GROUP (1587)] [original_request] [http://127.0.0.1:8000/foo/bar] => [404] [text/html] [469] [14] [95] [Error response] [SimpleHTTP/0.6 Python/3.8.10] []
  [GROUP (10)] [http_methods] [-X CONNECT http://127.0.0.1:8000/foo/bar] => [501] [text/html] [500] [14] [96] [Error response] [SimpleHTTP/0.6 Python/3.8.10] []
  [SINGLE] [mid_paths] [http://127.0.0.1:8000/???foo/bar] => [200] [text/html] [913] [26] [27] [Directory listing for /???foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
  [SINGLE] [mid_paths] [http://127.0.0.1:8000//???foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/???foo/bar]
  [SINGLE] [mid_paths] [http://127.0.0.1:8000/??foo/bar] => [200] [text/html] [911] [26] [27] [Directory listing for /??foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
  [SINGLE] [mid_paths] [http://127.0.0.1:8000//??foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/??foo/bar]
  [SINGLE] [mid_paths] [http://127.0.0.1:8000/?foo/bar] => [200] [text/html] [909] [26] [27] [Directory listing for /?foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
  [SINGLE] [mid_paths] [http://127.0.0.1:8000//?foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/?foo/bar]
  [SINGLE] [mid_paths] [http://127.0.0.1:8000///?anythingfoo/bar] => [200] [text/html] [929] [26] [27] [Directory listing for ///?anythingfoo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
  [SINGLE] [mid_paths] [http://127.0.0.1:8000////?anythingfoo/bar] => [200] [text/html] [931] [26] [27] [Directory listing for ////?anythingfoo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
  [GROUP (2)] [mid_paths] [http://127.0.0.1:8000/#?foo/bar] => [200] [text/html] [893] [26] [27] [Directory listing for /] [SimpleHTTP/0.6 Python/3.8.10] []
  [GROUP (2)] [mid_paths] [http://127.0.0.1:8000//#?foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/]
  ```





  ```bash

  sudo apt install -y bat curl virtualenv python3

  virtualenv -p python3 .py3
  source .py3/bin/activate
  PDM_BUILD_SCM_VERSION="$(git describe --abbrev=0)-dev" pip install .

  python src/bypass_url_parser/__init__.py -u https://thinkloveshare.com/juicy_403_endpoint/

  bypass-url-parser -u https://thinkloveshare.com/juicy_403_endpoint/
  cat /tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json  | jq -r '.results[].request_curl_cmd'
  cat /tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json  | jq -r '.results[].response_data'
  ```



  ```bash
  docker run --rm -it -v "$PWD:/host" -w /host ghcr.io/laluka/bypass-url-parser:latest bash -il


  ```





  Bypass_url_parser allows to define some arguments in many ways:

  - `-m, --mode`, `-s, --spoofip` and `-p, --spoofport` arguments can be a filename, a string, a comma-separated string list or a list (when `Bypasser` is used as a library);
  - `-u, --url` argument can be a filename, a string or a list (when `Bypasser` is used as a library);
  - `stdin` (with `-`) is supported for all these arguments.

  For example, if you want to define several target urls (`-u, --url`), all the following commands produce the same result:

  ```bash
  bypass-url-parser -u http://thinkloveshare.com/test
  bypass-url-parser -u /path/urls
  cat /path/urls | bypass-url-parser -u -
  echo 'http://thinkloveshare.com/test' | bypass-url-parser -u -
  ```



  A target must be defined for the tool to work. 2 options:

  - `-u, --url`: URL(s), in GET
  - `-R, --request`: Request file. The protocol can't be guessed from file, so `http` by default or `https` if `--request-tls` option is present.



  If `-m, --mode` is specified, you can select the desired bypass mode to run a specific test (or tests) and reduce the number of requests sent by the tool.

  For now, the following bypass mode(s) are supported:

  ```
  all, mid_paths, end_paths, case_substitution, char_encode, http_methods, http_versions, http_headers_method, http_headers_scheme, http_headers_ip, http_headers_port, http_headers_url, user_agent
  ```

  Example:

  ```bash
  bypass-url-parser -u /path/urls -m "case_substitution, char_encode, http_headers_scheme"
  ```



  In order to customize the ip addresses and ports used in bypass attempts, the tool supports the following options:

  - With `-s, --spoofip` you can set some IP(s) to inject into `ip-specific` headers (`X-Forwarded-For`, `X-Real-Ip`, etc.)
  - With `-p, --spoofport` you can set some ports to inject into `port-specific` headers (`X-Forwarded-Port`)

  By default, these custom entries are added to the internal IP/port lists. If you want to use only your IP(s)/port(s), you can use `--spoofip-replace` and/or `--spoofport-replace` arguments.

  Example:

  ```bash
  bypass-url-parser -u /path/urls -s /path/custom_ip --spoofip-replace
  bypass-url-parser -u /path/urls -p "3000, 9443, 10443"
  ```



  With the `--jsonl` option, it's possible to print the results on `stdout` in `JSON-Lines` format. The standard tool's output and results are displayed with a logger on `stderr`, so it is possible to pipe the `JSON-Line` output format with other tools:

  ```bash
  bypass-url-parser -u "https://thinkloveshare.com/juicy_403_endpoint/" -t 20 -S 0 -m case_substitution,char_encode --jsonl | jq
  ```

  ***Notes:** With `-S 2` ou `-S 3`, the JSON-Lines output also includes the path and the name of saved html files.*



  By default, if target url is unique, the tool saves a copy of the results in `/tmp/tmpXXX-bypass-url-parser/` directory.

  ***Notes:** If multiple target urls are passed to `-u`, results are prefixed with the url as directory (`/tmp/tmpXXX-bypass-url-parser/http-target-com-8080-api-users/`).*

  There are two arguments to customize this behavior:

  - `-o, --outdir` to set a custom output directory
  - `-S, --save-level` to choose a saving level

  The saving levels are:

  - `0` (NONE): Disable output saving and output directory creation;
  - `1` (MINIMAL): Only save the program log file which contains the results: `triaged-bypass.log`;
  - `2` (PERTINENT): Save the program log file `triaged-bypass.log` and **pertinent (results)** curl responses in `triaged-bypass.json` file and separate html files (Default);
  - `3` (FULL): Save the program log file `triaged-bypass.log` and **all** curl responses in `triaged-bypass.json` file and separate html files.



  ```bash
  bypass-url-parser -S 0
  bypass-url-parser -S 1 -o /tmp/bypass-res
  bypass-url-parser -S 2 -o /tmp/bypass-res2 -H "User-Agent: curl 7.74.0" -u http://thinkloveshare.com/juicy_403_endpoint/
  tree /tmp/bypass-res2/
  ├── bypass-2469eecf6c38b5817d2248e911ad4382.html
  ├── bypass-6f7cce7caf0a0a4b440859fa189d496d.html
  ├── bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html
  ├── bypass-93079abffe63d34f79ac4a511cd6b5e6.html
  ├── bypass-945822230d58d1ad4680d5dfbc470ecb.html
  ├── bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html
  ├── triaged-bypass.json
  └── triaged-bypass.log

  0 directories, 8 files
  ```



  Starting from `MINIMAL` level, the results displayed by the program are saved in the `triaged-bypass.log` file.



  With `PERTINENT` and `FULL` saving levels, the program additionally exports all results in the `triaged-bypass.json` file:

  ```json
  {
    "url": "http://thinkloveshare.com/juicy_403_endpoint/",
    "bypass_modes": "all",
    "results": [
      {
        "request_curl_cmd": "/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -H 'X-BlueCoat-Via: localhos[...SNIP...]",
        "request_curl_payload": "-H X-BlueCoat-Via: localhost http://thinkloveshare.com/juicy_403_endpoint/",
        "response_headers": "HTTP/1.1 301 Moved Permanently\nConnection: keep-alive\nContent-Length: 162\nServer: GitHub.c[...SNIP...]",
        "response_data": "<html>\n<head><title>301 Moved Permanently</title></head>\n<body>\n<center><h1>301 Moved Permane[...SNIP...]",
        "response_status_code": 301,
        "response_content_type": "text/html",
        "response_content_length": 162,
        "response_lines_count": 7,
        "response_words_count": 4,
        "response_title": "301 Moved Permanently",
        "response_server_type": "GitHub.com",
        "response_redirect_url": "https://thinkloveshare.com/juicy_403_endpoint/",
        "response_html_filename": "bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html"
      },
      {
        "request_curl_cmd": "/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -X PROPFIND http://thinklove[...SNIP...]",
        "request_curl_payload": "-X PROPFIND http://thinkloveshare.com/juicy_403_endpoint/",
        "response_headers": "HTTP/1.1 405 Method Not Allowed\nConnection: close\nContent-Length: 131\nServer: Varnish\nRet[...SNIP...]",
        "response_data": "<html>\n<head><title>405 Not Allowed</title></head>\n<body bgcolor=\"white\">\n<center><h1>405 N[...SNIP...]",
        "response_status_code": 405,
        "response_content_type": "",
        "response_content_length": 131,
        "response_lines_count": 5,
        "response_words_count": 5,
        "response_title": "405 Not Allowed",
        "response_server_type": "Varnish",
        "response_redirect_url": "",
        "response_html_filename": "bypass-945822230d58d1ad4680d5dfbc470ecb.html"
      },
      {  "...": "[...SNIP...]"},
      {
        "request_curl_cmd": "/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is 'http://thinkloveshare.com/j[...SNIP...]",
        "request_curl_payload": "http://thinkloveshare.com/juicy_403_endpoint/\u00b0//",
        "response_headers": "HTTP/1.1 400 Bad request\nConnection: keep-alive\nContent-Length: 90\nCache-Control: no-cache[...SNIP...]",
        "response_data": "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n",
        "response_status_code": 400,
        "response_content_type": "text/html",
        "response_content_length": 90,
        "response_lines_count": 3,
        "response_words_count": 7,
        "response_title": "",
        "response_server_type": "",
        "response_redirect_url": "",
        "response_html_filename": "bypass-2469eecf6c38b5817d2248e911ad4382.html"
      }
    ]
  }
  ```

  Making them easier to handle with `jq`:

  ```bash
  $ jq -r '.results[] | [.request_curl_payload, .response_status_code, .response_content_type, .response_content_length] | join("|")' /tmp/bypass-res2/triaged-bypass.json

  -H X-BlueCoat-Via: localhost http://thinkloveshare.com/juicy_403_endpoint/|301|text/html|162
  -X PROPFIND http://thinkloveshare.com/juicy_403_endpoint/|405||131
  http://thinkloveshare.com/%3b%2f%2e%2e%2f%2e%2e%2f%2fjuicy_403_endpoint/|400|text/html|9121
  -H Host: 8.8.8.8 http://thinkloveshare.com/juicy_403_endpoint/|404|text/html|9115
  -X CONNECT http://thinkloveshare.com/juicy_403_endpoint/|400|text/plain|15
  http://thinkloveshare.com/juicy_403_endpoint/°//|400|text/html|90
  ```



  With `PERTINENT` and `FULL` saving levels, curl commands and full HTTP responses are also stored in pseudo `.html` files:

  ```bash
  $ echo /tmp/bypass-res2/*.html | xargs batcat
  ───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
         │ File: /tmp/bypass-res2/bypass-2469eecf6c38b5817d2248e911ad4382.html
  ───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1   │ /usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is 'http://thinkloveshare.com/juicy_403_endpoint/°//'
     2   │
     3   │ HTTP/1.1 400 Bad request
     4   │ Connection: keep-alive
     5   │ Content-Length: 90
     6   │ Cache-Control: no-cache
     7   │ Content-Type: text/html
     8   │ Accept-Ranges: bytes
     9   │ Date: Tue, 25 Apr 2023 23:51:38 GMT
    10   │ Via: 1.1 varnish
    11   │ X-Served-By: cache-par-lfpg1960025-PAR
    12   │ X-Cache: MISS
    13   │ X-Cache-Hits: 0
    14   │ X-Timer: S1682466698.230664,VS0,VE10
    15   │ Vary: Accept-Encoding
    16   │ X-Fastly-Request-ID: b6bbb82302420db4f101a316dca39cc283a4fd44
    17   │
    18   │ <html><body><h1>400 Bad request</h1>
    19   │ Your browser sent an invalid request.
    20   │ </body></html>
  ───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
         │ File: /tmp/bypass-res2/bypass-6f7cce7caf0a0a4b440859fa189d496d.html
  ───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1   │ /usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -X CONNECT http://thinkloveshare.com/juicy_403_endpoint/
     2   │
     3   │ HTTP/1.1 400 Bad Request
     4   │ Connection: close
     5   │ Content-Length: 15
     6   │ content-type: text/plain; charset=utf-8
     7   │ x-served-by: cache-par-lfpg1960083
     8   │
     9   │ invalid request
  ───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
         │ File: /tmp/bypass-res2/bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html
  ───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1   │ /usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is http://thinkloveshare.com/%3b%2f%2e%2e%2f%2e%2e%2f%2fjuicy_403_endpoint/
     2   │
     3   │ HTTP/1.1 400 Bad Request
     4   │ Connection: keep-alive
     5   │ Content-Length: 9121
     6   │ Server: GitHub.com
     7   │ Content-Type: text/html; charset=utf-8
     8   │ ETag: "64417b9f-23a1"
     9   │ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self'
    10   │ X-GitHub-Request-Id: 598E:F13E:26EE27D:284F5D2:64486731
    11   │ Accept-Ranges: bytes
    12   │ Date: Tue, 25 Apr 2023 23:50:11 GMT
    13   │ Via: 1.1 varnish
    14   │ X-Served-By: cache-par-lfpg1960046-PAR
    15   │ X-Cache: MISS
    16   │ X-Cache-Hits: 0
    17   │ X-Timer: S1682466611.396077,VS0,VE101
    18   │ Vary: Accept-Encoding
    19   │ X-Fastly-Request-ID: 30bd6af5892c40da130ee49bbeacd147a1a6b3c3
    20   │
    21   │ <!DOCTYPE html>
    22   │ <html>
    23   │   <head>
    24   │     <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    25   │     <meta http-equiv="Content-Security-Policy" content="default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self'">
    26   │     <title>Bad request &middot; GitHub Pages</title>
    27   │     <style type="text/css" media="screen">
  [...SNIP...]
  ```



  - Initial release by [@TheLaluka](https://twitter.com/TheLaluka)
  - Huge refactoring & lib-mode with thanks to [@jtop_fap](https://twitter.com/jtop_fap)
  - Support for `Docker` & `Pypi` builds with the kind work of [@DugnyG](https://twitter.com/DugnyG)



  Copyright (C) 2022 Laluka

  This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

  This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

  You should have received a copy of the GNU Affero General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.


pkgId: 8547eef81410acfa34b9105f6bf400c89bef7127483b251e265bf84f7063836d
name: python-byrdocs-cli
arch: src
version: 0.1.2
epoch: 0
release: 1
summary: A simple command line tool for BYR Docs
url: https://pypi.org/project/byrdocs-cli/
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byrdocs-cli-0.1.2-1.src.rpm
checksum_type: sha256
description:
  Using pip:
  ```bash
  pip3 install byrdocs
  ```



  ```bash
  usage: byrdocs [-h] [--version] [--token TOKEN] [command] [file]

  positional arguments:
    command        Command to execute
    file           Path to the file to upload

  options:
    -h, --help     show this help message and exit
    --version, -v  show program's version number and exit
    --token TOKEN  Token for login command
  ```



  Build:

  ```bash
  python3 -m build
  ```


  Push to PyPI:
  ```bash
  python3 -m twine upload --repository pypi dist/* --verbose
  ```


pkgId: 0e75e8073e831ad9a7d5a5c7a89188d75f4bd4fe8e77e23d4ee53a7100721518
name: python-byrdocs-cli
arch: src
version: 0.3.2
epoch: 0
release: 1
summary: A simple command line tool for BYR Docs
url: https://github.com/byrdocs/byrdocs-cli
rpm_license: MIT License
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byrdocs-cli-0.3.2-1.src.rpm
checksum_type: sha256
description:
  ```bash
  pip3 install byrdocs-cli
  ```



  Choose one of the following commands according to your package manager:
  ```bash
  sudo apt install pipx
  sudo dnf install pipx
  sudo pacman -S pipx
  ```

  Then:
  ```
  pipx install byrdocs-cli
  ```



  ```
  usage: byrdocs [-h] [--token TOKEN] [command] [file]

  Commands:
    upload <file>    Upload a file. If no command is specified, it defaults to upload.
    login            Authenticate with BYR Docs and obtain a token.
    logout           Remove the locally stored authentication token.

  positional arguments:
    command        Command to execute
    file           Path to the file to upload

  options:
    -h, --help     show this help message and exit
    --token TOKEN  Token for login command

  Examples:
   $ byrdocs login
   $ byrdocs /home/exam_paper.pdf
   $ byrdocs logout
  ```



  Build:

  ```bash
  python3 -m build
  ```


  Push to PyPI:
  ```bash
  python3 -m twine upload --repository pypi dist/* --verbose
  ```

  Test:
  ```bash
  python test.py [arguments]
  ```


pkgId: 71b681a9be6f4ea9f6044234f68ef814faaed49cf0a0f16fb05d6172e577e362
name: python-byrdocs-cli
arch: src
version: 0.5.0
epoch: 0
release: 1
summary: A simple command line tool for BYR Docs
url: https://github.com/byrdocs/byrdocs-cli
rpm_license: MIT License
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byrdocs-cli-0.5.0-1.src.rpm
checksum_type: sha256
description:
  ```bash
  pip3 install byrdocs-cli
  ```



  根据包管理器选择对应的命令安装 `pipx`：
  ```bash
  sudo apt install pipx
  sudo dnf install pipx
  sudo pacman -S pipx
  ```

  再使用 `pipx` 进行安装：
  ```
  pipx install byrdocs-cli
  ```



  使用 `pip`:
  ```bash
  pip install byrdocs-cli --upgrade
  ```

  使用 `pipx`:
  ```bash
  pipx upgrade byrdocs-cli
  ```



  ```
  用法: byrdocs [-h] [--token TOKEN] [command] [file]

  命令：
    upload <文件路径>    上传文件 [默认命令]
    login               登录到 BYR Docs
    logout              退出登录
    init                交互式生成文件元信息文件
    validate            (待实现) 验证元信息文件的合法性

  参数:
    command        要执行的命令
    file           要上传的文件路径

  选项:
    -h, --help     输出该帮助信息并退出
    --token TOKEN  指定登录时使用的 token

  示例：
    $ byrdocs login
    $ byrdocs /home/exam_paper.pdf
    $ byrdocs logout
    $ byrdocs init
  ```



  构建:

  ```bash
  python3 -m build
  ```


  发布到 PyPI:
  ```bash
  python3 -m twine upload --repository pypi dist/* --verbose
  ```

  测试:
  ```bash
  python test.py [arguments]
  ```


pkgId: f7d3ee22b7a0915e79e20161f0fb58fe69b5992a14ca26656ddaf41ab976b931
name: python-byron
arch: src
version: 0.8a1.dev66
epoch: 0
release: 1
summary: An evolutionary source-code fuzzer
url: https://cad-polito-it.github.io/byron/
rpm_license: Apache-2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byron-0.8a1.dev66-1.src.rpm
checksum_type: sha256
description:
  [![Codename](https://img.shields.io/badge/codename-Don_Juan-pink.svg)](https://en.wikipedia.org/wiki/Don_Juan_(poem))
  [![GitHub License](https://img.shields.io/github/license/squillero/byron)](https://opensource.org/licenses/)
  [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/byron)](https://www.python.org/)
  [![GitHub Workflow Status (with event)](https://img.shields.io/github/actions/workflow/status/cad-polito-it/byron/pytest.yml?label=tests)
  ](https://github.com/cad-polito-it/byron/actions)
  [![Documentation Status](https://readthedocs.org/projects/byron/badge/?version=pre-alpha)](https://byron.readthedocs.io/en/pre-alpha/?badge=pre-alpha)
  [![GitHub last commit (alpha)](https://img.shields.io/github/last-commit/cad-polito-it/byron/alpha?label=last+github+commit)](https://github.com/cad-polito-it/byron/pulse)
  ![PyPI - Downloads](https://img.shields.io/pypi/dm/byron?label=downloads)

  Byron is a source code [fuzzer](https://en.wikipedia.org/wiki/Fuzzing) designed to support assembly and high-level languages. It starts by generating a set of random programs, which are then iteratively improved by an [evolutionary algorithm](https://cad-polito-it.github.io/byron/evolution). Internally, it encodes candidate solutions as [typed](https://rcor.me/papers/typed-graph-theory.pdf), [directed](https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)#Directed_graph) [multigraphs](https://en.wikipedia.org/wiki/Multigraph), and can effectively handle complex, realistic structures containing local and global variables, conditional and looping statements, and subroutines.

  Programs can be evaluated using a user-defined Python function or an external tool, such as an interpreter or a simulator. Different types of parallelization are supported out of the box, from simple multithreading to the creation of temporary directories where multiple subprocesses are concurrently [spawned](https://en.wikipedia.org/wiki/Spawn_(computing)).

  **⚠️ Byron is currently in [alpha](https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha) and under active development**



  As simple as

  ```
  pip install --upgrade byron
  ```

  A few dependencies can enhance Byron functionalities, but are not strictly required. You can get them all with

  ```
  pip install --upgrade "byron[full]"
  ```



  None yet, but some HOWTO's and examples [are available](https://github.com/cad-polito-it/byron/tree/alpha/examples) in the development repo.



  * Giovanni Squillero — <giovanni.squillero@polito.it>
  * Alberto Tonda — <alberto.tonda@inrae.fr>



  Copyright (c) 2023-24 [Giovanni Squillero](https://github.com/squillero) and [Alberto Tonda](https://github.com/albertotonda/)
  Byron is [free and open-source software](https://en.wikipedia.org/wiki/Free_and_open-source_software), and it is distributed under the permissive [Apache License 2.0](https://opensource.org/license/apache-2-0/).


pkgId: bc2c66517700dac8b1a7aee1e7cc5cb3b119407ca45cb621058f777517575103
name: python-bytebuffer
arch: src
version: 0.1.2
epoch: 0
release: 1
description: UNKNOWN
url: https://github.com/alon-sage/python-bytebuffer
rpm_license: Apache License 2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytebuffer-0.1.2-1.src.rpm
checksum_type: sha256
summary:
  High performance library for reading and writing values of primitive types. Primitive values are translated to (or from) sequences of bytes.


pkgId: 47838a800d1009a0c59beb94500c0e2fe463369454100dbbb57fcbfd1b81b9bf
name: python-bytecode
arch: src
version: 0.15.1
epoch: 0
release: 1
summary: Python module to generate and modify bytecode
url: https://pypi.org/project/bytecode/
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytecode-0.15.1-1.src.rpm
checksum_type: sha256
description:
  ********
  bytecode
  ********

  .. image:: https://img.shields.io/pypi/v/bytecode.svg
     :alt: Latest release on the Python Cheeseshop (PyPI)
     :target: https://pypi.python.org/pypi/bytecode

  .. image:: https://github.com/MatthieuDartiailh/bytecode/workflows/Continuous%20Integration/badge.svg
      :target: https://github.com/MatthieuDartiailh/bytecode/actions
      :alt: Continuous integration

  .. image:: https://github.com/MatthieuDartiailh/bytecode/workflows/Documentation%20building/badge.svg
      :target: https://github.com/MatthieuDartiailh/bytecode/actions
      :alt: Documentation building

  .. image:: https://img.shields.io/codecov/c/github/MatthieuDartiailh/bytecode/master.svg
     :alt: Code coverage of bytecode on codecov.io
     :target: https://codecov.io/github/MatthieuDartiailh/bytecode

  .. image:: https://img.shields.io/badge/code%20style-black-000000.svg
     :alt: Code formatted using Black
     :target: https://github.com/psf/black

  ``bytecode`` is a Python module to generate and modify bytecode.

  * `bytecode project homepage at GitHub
    <https://github.com/MatthieuDartiailh/bytecode>`_ (code, bugs)
  * `bytecode documentation
    <https://bytecode.readthedocs.io/>`_
  * `Download latest bytecode release at the Python Cheeseshop (PyPI)
    <https://pypi.python.org/pypi/bytecode>`_

  Install bytecode: ``python3 -m pip install bytecode``. It requires Python 3.8
  or newer. The latest release that supports Python 3.7 and 3.6 is 0.13.0.
  The latest release that supports Python 3.5 is 0.12.0. For Python 2.7 support,
  have a look at `dead-bytecode <https://github.com/p403n1x87/dead-bytecode>`_
  instead.

  Example executing ``print('Hello World!')``:

  .. code:: python

      from bytecode import Instr, Bytecode

      bytecode = Bytecode([Instr("LOAD_NAME", 'print'),
                           Instr("LOAD_CONST", 'Hello World!'),
                           Instr("CALL_FUNCTION", 1),
                           Instr("POP_TOP"),
                           Instr("LOAD_CONST", None),
                           Instr("RETURN_VALUE")])
      code = bytecode.to_code()
      exec(code)

rpm_license:
  The MIT License (MIT) Copyright Contributors to the bytecode project.  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


pkgId: 4eb795ec817f81b432f5b22659063afb0b6353dcd4a31f32a0c6d606d55d5906
name: python-bytecodemanipulation
arch: src
version: 0.3.5
epoch: 0
release: 1
summary: High level python bytecode manipulation
url: https://github.com/uuk0/PyBytecodeManipulator
rpm_license: MIT License
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytecodemanipulation-0.3.5-1.src.rpm
checksum_type: sha256
description:
  A high level cross-version python bytecode manipulation library build ontop
  of 'dis' and 'inspect'

  Supports code inlining, branch removing and arbitrary code injection into
  existing functions.

  WARNING: using bytecode manipulation on a so low level as we do can break
  the python runtime at any point without a warning. We circumvent a lot of
  safety checks normally done.

  WARNING: We cannot make sure that everything works as it should, expect broken code
  at runtime!


  Supported python versions:

  - 3.10 (main development)
  - 3.11.0[b3] (forward porting; WIP; Currently not working)

  Other versions will not work as a lot of config is stored in .json files per-version,
  so you may need to provide your own .json config files for the version you need.

  (Some versions might be plug-and-play, but most will require code changes additionally)



  Due to the breaking nature of anything this code touches, and the absents of any traces
  in the function itself, we decided to add a lot of "debug" statements indicating
  mostly the who-has-done-what-to-which-method for the runtime.
  This makes debugging broken code easier, as it is more clear what happened to each transformed function.

  If you want them removed, create your own Fork of this and remove them, on your own risk.

  We may use in the future the logging library, so you can disable our logger instance, but we
  are currently in an inter-stage of the code, so other stuff has priority.



  - Nuitka (https://github.com/Nuitka/Nuitka): Incompatible; will break as nuitka removes the \_\_code__ attribute
    we modify
  - Other bytecode modification / analysers: Should work as long as you as the user does NOT expose intermediate
    results which might contain internal instructions




  There is the possibility to "debug" functions using the execution emulator.
  It will be able to give you more information about a crash than the python core interpreter,
  but will be a lot slower than it.

  It comes also with the possibility to run your bytecode in another interpreter version, so
  you can experiment with some stuff.
  In theory, it is also possible to run in python versions not supported by the
  bytecode manipulation system, but it is not recommended.

  TransformationHandler() takes as an arg debug_code and debug_further_calls
  for activating it for all accessed methods.

  BytecodePatchHelper() contains a method enable_verbose_exceptions() for activating it on
  that exact method.




  Replacing global access with constant value

  ```python
  from bytecodemanipulation.Optimiser import cache_global_name, apply_now

  @cache_global_name("min", lambda: min)
  def test(x, y):
      return min(x, y)

  apply_now(test)
  ```

  Inlining method calls

  ```python
  from bytecodemanipulation.Optimiser import cache_global_name, inline_calls, apply_now

  @inline_calls
  def call(a, b):
      return a + b

  @cache_global_name("call", lambda: call)
  def test(x, y):
      return call(x, y)

  apply_now(test)
  ```

  will result in bytecode which could be represented with:

  ```python
  def test(x, y):
      return x + y
  ```

  (there might be extra local variables generated for parameters)



  - Constant Expression inlining (can be declared for custom functions to be constant)
  - LOAD_GLOBAL for builtins (if enabled)
  - standard library inlining (if enabled)
  - specialization of methods based on arguments, e.g. constant arguments (when already resolved before, requires one of above options)
  - branch elimination when jumping on a constant (TODO: also if condition can be inferred ahead-of-time, see specialization)
  - local variable elimination (TODO: add more cases)




  - Line Numbers get mixed up, we need some way to assign meaningful line numbers
  - With python 3.11, exception table exists, and this breaks our current concept of one big flow diagram,
    as exception handling code might exist outside the default flow
  - Also python 3.11 introduces CACHE-s for instructions, which will require some work in order to work
  - During optimization, a lot of stuff is being recomputed each optimization pass, we need to cache that drastically
  - Method inlining is not working properly and needs a lot more testing (WIP)
  - If the exact type is known at optimisation time (e.g. object creation via class call, or type annotation), we can try to
    inline method accesses for further optimisation
  - Python 3.12 will likely break how certain operations are stored in instructions, combing opcodes-without-args into a single
    parent instruction, using the arg to switch between modes




  - The library provides also a way to use some "python-assembly" for writing code
  - This is only python bytecode, so no fancy stuff can be done
  - See ASSEMBLY.md for more information on instructions
  - We provide an import system hook for importing .pyasm files via bytecodemanipulation.assembler.hook
  - You may use the functions from bytecodemanipulation.assembler.target for creating inline-assembly
      - assembly(\<code>) can be used for inline assembly
      - this also includes macro, which may be also created from python-defined methods




  We use the python formatting library "black" on our code



  - abstract opcode affect away into a .json file describing all opcodes
  - create a json file for defining certain bytecode sequences
  - write more library-specific optimisations
  - write generating bytecode system for emulator, constructing a function pointing to the
    .json file for exception printing, and optimizing wherever possible
  - add optimisation: JUMP on condition for some operators (e.g. NOT)


pkgId: 6e01e4ae3161deea71a936c93fdcc9942b40d1817a1ba1d150b028fbab9042c0
name: python-bytecrypt
arch: src
version: 0.0.2
epoch: 0
release: 1
summary: Easily encrypt/decrypt bytes with password
url: https://pypi.org/project/bytecrypt/
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytecrypt-0.0.2-1.src.rpm
checksum_type: sha256
description:
  Bytecript is a python package for easy data encryption / decryption with password.


  ```py



  from bytecrypt.bytecrypt import encrypt_bytes
  from bytecrypt.bytecrypt import decrypt_bytes

  encrypted_data = encrypt_bytes(b"secret", b"password")
  decrypted_data = decrypt_bytes(encrypted_data, b"password")

  print("\nEncrypted data: " + str(encrypted_data.decode("utf-8")))
  print("\nDecrypted data: " + str(decrypted_data.decode("utf-8")))


  ```

rpm_license:
  MIT License  Copyright (c) 2024 sm  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


pkgId: 35e2321ce550d752ee14697ff2a5f9b2f7dab6f4fb3e34a2765452174b9c2ba9
name: python-bytecrypt
arch: src
version: 0.3.1
epoch: 0
release: 1
summary: Easily encrypt/decrypt data with password
url: https://pypi.org/project/bytecrypt/
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytecrypt-0.3.1-1.src.rpm
checksum_type: sha256
description:
  Bytecrypt is a python package for easy data encryption / decryption with password.


  ```sh

  pip install bytecrypt

  ```





  ```py



  from bytecrypt import *


  encrypted_data = encrypt_bytes(b"secret", b"password")
  decrypted_data = decrypt_bytes(encrypted_data, b"password")

  print("\nEncrypted data: " + str(encrypted_data.decode("utf-8")))
  print("\nDecrypted data: " + str(decrypted_data.decode("utf-8")))

  ```

  ```sh


  $ python main.py

  Encrypted data: gAAAAABnO7wzHm-WLv-s_fQgHRe_-0Al_CmzUU7XfZcRaRSBXbLy1j8Z97KhiY8nZbaHETyKSO_NuGQH1f73MMs58nrT7pxWJg==

  Decrypted data: secret


  ```




  ```py

  encrypt_file("path/to/file/test.docx", b"testPassword")
  decrypt_file("path/to/file/test.docx", b"testPassword")

  encrypt_file("path/to/file/test.docx", b"testPassword", encrypt_filename=True)
  decrypt_file("path/to/file/test.docx", b"testPassword", decrypt_filename=True)


  encrypt_directory("path/to/directory", b"testPassword")
  decrypt_directory("path/to/directory", b"testPassword")

  encrypt_directory("path/to/directory", b"testPassword", encrypt_filename=True)
  decrypt_directory("path/to/directory", b"testPassword", decrypt_filename=True)


  ```




  ```sh


  bytecrypt -e -f "test_file.txt" -p "test123"


  bytecrypt -d -f "test_file.txt" -p "test123"


  bytecrypt -e -f "test_file.txt" -efn -p "test123"


  bytecrypt -d -f "EJHF2_1bf...FHJ=" -dfn -p "test123"


  bytecrypt -e -str "test_string-1234" -p "test123"
  bytecrypt -d -str "tYWHbf_...2dHSL=" -p "test123"


  bytecrypt -e -dir "test/directory1" -p "test123"
  bytecrypt -d -dir "test/directory1" -p "test123"
  bytecrypt -e -dir . -p "test123"
  bytecrypt -e -dir . -p -efn "test123"
  bytecrypt -d -dir . -p -dfn "test123"


  bytecrypt -e -dir "test/directory1" -r -p "test123"

  ```




  ```sh

  -e      ;   --encrypt
  -d      ;   --decrypt
  -f      ;   --file
  -dir    ;   --directory
  -efn    ;   --encrypt_filename
  -dfn    ;   --decrypt_filename
  -str    ;   --string
  -r      ;   --recursive
  -p      ;   --password

  ```

rpm_license:
  MIT License  Copyright (c) 2024 sm  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


pkgId: 6579a6074d60c07547e910e985eea2b2cfe25568be5bb4a4993a478f44cc3231
name: python-byteplus
arch: src
version: 1.0.33
epoch: 0
release: 1
summary: SDK for byteplus
url: https://github.com/byteplus-sdk
rpm_license: Apache 2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byteplus-1.0.33-1.src.rpm
checksum_type: sha256
description:
  ```shell
  pip install --upgrade byteplus
  ```

  * [example](https://github.com/byteplus-sdk/example-python)


  * [code sample](https://docs.byteplus.com/recommend/docs/code-sample)


pkgId: 0561812dd33fde57e3fac7b108645755cc69f25bce01e2448820538e4aa282b0
name: python-byterat
arch: src
version: 0.1.0
epoch: 0
release: 1
summary: Python library to export data from and interact with the Byterat Platform
url: https://pypi.org/project/byterat/
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byterat-0.1.0-1.src.rpm
checksum_type: sha256
rpm_license:
  Certain libraries contained in this file are subject to open source license requirements. Specifically: GQL: Copyright (c) 2016 GraphQL Python  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.  Pandas: Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team All rights reserved.  Copyright (c) 2011-2024, Open source contributors.  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:  * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.  * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.  * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


pkgId: 9e27560084f60d81a1f35f44e27ebbb4bef1ab0ebfa111b7051ec42e7ddbd0d4
name: python-bytesparse
arch: src
version: 1.0.1
epoch: 0
release: 1
summary: Library to handle sparse bytes within a virtual memory space
url: https://pypi.org/project/bytesparse/
rpm_license: BSD-2-Clause
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytesparse-1.0.1-1.src.rpm
checksum_type: sha256
description:
  This library aims to provide utilities to work with a `virtual memory`, which
  consists of a virtual addressing space where sparse `chunks` of data can be
  stored.
  In order to be easy to use, its interface should be close to that of a
  ``bytearray``, which is the closest pythonic way to store dynamic data.
  The main downside of a ``bytearray`` is that it requires a contiguous data
  allocation starting from address 0. This is not good when sparse data have to
  be stored, such as when emulating the addressing space of a generic
  microcontroller.
  The main idea is to provide a ``bytearray``-like class with the possibility to
  internally hold the sparse `blocks` of data.
  A `block` is ideally a tuple ``(start, data)`` where `start` is the start
  address and `data` is the container of data items (e.g. ``bytearray``).
  The length of the block is ``len(data)``.
  Those blocks are usually not overlapping nor contiguous, and sorted by start
  address.


pkgId: 8db3e3c446c305c03c433279f21020d2e258106dd52b3212936165c2b846b4f6
name: python-bytesviewnotifiactionclient
arch: src
version: 1.1.4
epoch: 0
release: 1
summary: A plugin for Notification
url: https://github.com/bytesview/bytesview-notifiaction-client
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytesviewnotifiactionclient-1.1.4-1.src.rpm
checksum_type: sha256
description:
  BytesView Notification Client is a centalized way of managing notifications using fcm

  <!-- [![Build Status](https://img.shields.io/github/actions/workflow/status/bytesview/python-client/python-publish.yml)](https://github.com/bytesview/python-client/actions/workflows/python-publish.yml)
  [![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/newsdataapi/python-client/blob/main/LICENSE)
  [![PyPI](https://img.shields.io/pypi/v/newsdataapi?color=084298)](https://pypi.org/project/newsdataapi)
  [![PyPI - Downloads](https://img.shields.io/pypi/dm/newsdataapi)](https://pypi.org/project/newsdataapi)
  [![Supported Python versions](https://img.shields.io/pypi/pyversions/newsdataapi)](https://pypi.org/project/newsdataapi) -->


  ```
  pip install bytesviewnotifiactionclient

  ```



  About BytesView click [here](https://www.bytesview.com/)


pkgId: 5b21b96cff6c839a72e36e1aaad13d81649a4904c7a445300d2218608e5a558f
name: python-bytewax-duckdb
arch: src
version: 0.1.22
epoch: 0
release: 1
summary: Bytewax custom sink for DuckDB and MotherDuck
url: https://github.com/bytewax/bytewax-duckdb
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytewax-duckdb-0.1.22-1.src.rpm
checksum_type: sha256
description:
  [![Actions Status](https://github.com/bytewax/bytewax-duckdb/workflows/CI/badge.svg)](https://github.com/bytewax/bytewax-duckdb/actions)
  [![PyPI](https://img.shields.io/pypi/v/bytewax-duckdb.svg?style=flat-square)](https://pypi.org/project/bytewax-duckdb/)
  [![Bytewax User Guide](https://img.shields.io/badge/user-guide-brightgreen?style=flat-square)](https://docs.bytewax.io/projects/bytewax-duckdb/en/stable/)

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/6073079/195393689-7334098b-a8cd-4aaa-8791-e4556c25713e.png" width="350">
    <source media="(prefers-color-scheme: light)" srcset="https://user-images.githubusercontent.com/6073079/194626697-425ade3d-3d72-4b4c-928e-47bad174a376.png" width="350">
    <img alt="Bytewax">
  </picture>



  Bytewax sink and operator for DuckDB and MotherDuck



  We've made installation as easy as Py by making it pip-installable:

  ```bash
  pip install bytewax-duckdb
  ```

  This will also install the latest DuckDB and Bytewax modules.



  When working with this integration in Bytewax, you can use it to process data in batch and write data to a target database or file in a structured way. However, there’s one essential assumption you need to know, the sink expects data in a specific tuple format, structured as:

  ```python
  ("key", List[Dict])
  ```
  Where

  `"key"`: The first element is a string identifier for the batch. Think of this as a “batch ID” that helps to organize and keep track of which group of entries belong together. Every batch you send to the sink has a unique key or identifier.

  `List[Dict]`: The second element is a list of dictionaries. Each dictionary represents an individual data record, with each key-value pair in the dictionary representing fields and their corresponding values.

  Together, the tuple tells the sink: “Here is a batch of data, labeled with a specific key, and this batch contains multiple data entries.”

  This format is designed to let the sink write data efficiently in batches, rather than handling each entry one-by-one. By grouping data entries together with an identifier, the sink can:

  * Optimize Writing: Batching data reduces the frequency of writes to the database or file, which can dramatically improve performance, especially when processing high volumes of data.

  * Ensure Atomicity: By writing a batch as a single unit, we minimize the risk of partial writes, ensuring either the whole batch is written or none at all. This is especially important for maintaining data integrity.

  Here is an example for a local DuckDB file.

  ```python
  import bytewax.duckdb.operators as duck_op
  import bytewax.operators as op
  from bytewax.dataflow import Dataflow
  from bytewax.testing import TestingSource, run_main

  flow = Dataflow("duckdb")


  def create_dict(value: int) -> Tuple[str, Dict[str, Union[int, str]]]:
      return ("1", {"id": value, "name": "Alice"})


  inp = op.input("inp", flow, TestingSource(range(50)))
  dict_stream = op.map("dict", inp, create_dict)

  duck_op.output(
      "out",
      dict_stream,
      "sample.duckdb",
      "example_table",
      "CREATE TABLE IF NOT EXISTS example_table (id INTEGER, name TEXT)",
  )

  run_main(flow)
  ```

  **Important**
  To connect to a MotherDuck instance, ensure to [create an account](https://app.motherduck.com/?auth_flow) and [generate a token](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/#creating-an-access-token). You can store this token into your environment variables.

  ```python
  import os
  import random
  from typing import Dict, Tuple, Union


  md_token = os.getenv("MOTHERDUCK_TOKEN")


  flow = Dataflow("duckdb-names-cities")


  names = ["Alice", "Bob", "Charlie", "Diana", "Eve"]
  locations = ["New York", "Los Angeles", "Chicago", "Houston", "Phoenix"]



  def create_dict(value: int) -> Tuple[str, Dict[str, Union[int, str]]]:
      name = random.choice(names)
      age = random.randint(20, 60)  # Random age between 20 and 60
      location = random.choice(locations)
      return ("batch_1", {"id": value, "name": name, "age": age, "location": location})



  inp = op.input("inp", flow, TestingSource(range(50)))
  dict_stream = op.map("dict", inp, create_dict)
  db_path = f"md:my_db?motherduck_token={md_token}"

  duck_op.output(
      "out",
      dict_stream,
      db_path,
      "names_cities",
      "CREATE TABLE IF NOT EXISTS names_cities (id INTEGER, name TEXT, age INTEGER, location TEXT)",
  )


  run_main(flow)
  ```





  We use [`just`](https://just.systems/man/en/) as a command runner for
  actions / recipes related to developing Bytewax. Please follow [the
  installation
  instructions](https://github.com/casey/just?tab=readme-ov-file#installation).
  There's probably a package for your OS already.



  I suggest using [`pyenv`](https://github.com/pyenv/pyenv)
  to manage python versions.
  [the installation instructions](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation).

  You can also use your OS's package manager to get access to different
  Python versions.

  Ensure that you have Python 3.12 installed and available as a "global
  shim" so that it can be run anywhere. The following will make plain
  `python` run your OS-wide interpreter, but will make 3.12 available
  via `python3.12`.

  ```console
  $ pyenv global system 3.12
  ```



  We use [`uv`](https://github.com/astral-sh/uv) as a virtual
  environment creator, package installer, and dependency pin-er. There
  are [a few different ways to install
  it](https://github.com/astral-sh/uv?tab=readme-ov-file#getting-started),
  but I recommend installing it through either
  [`brew`](https://brew.sh/) on macOS or
  [`pipx`](https://pipx.pypa.io/stable/).



  We use [`just`](https://just.systems/man/en/) as a command runner for
  actions / recipes related to developing Bytewax. Please follow [the
  installation
  instructions](https://github.com/casey/just?tab=readme-ov-file#installation).
  There's probably a package for your OS already.



  We have a `just` recipe that will:

  1. Set up a venv in `venvs/dev/`.

  2. Install all dependencies into it in a reproducible way.

  Start by adding any dependencies that are needed into [pyproject.toml](pyproject.toml) or into
  [requirements/dev.in](requirements/dev.in) if they are needed for development.

  Next, generate the pinned set of dependencies with

  ```console
  > just venv-compile-all
  ```



  Once you have compiled your dependencies, run the following:

  ```console
  > just get-started
  ```

  Activate your development environment and run the development task:

  ```console
  > . venvs/dev/bin/activate
  > just develop
  ```



  `bytewax-duckdb` is commercially licensed with
  publicly available source code. You are welcome to prototype using
  this module for free, but any use on business data requires a paid
  license. See https://modules.bytewax.io/ for a license. Please see the
  full details in [LICENSE](./LICENSE.md).

rpm_license:
  # Software License Agreement  The terms and conditions of this Software License Agreement (this "License Agreement") govern your use of the accompanying Bytewax Module software (the "Software") and constitute a binding legal contract between you and Bytewax, Inc. ("Bytewax"). Your download, installation or use of the Software constitutes acceptance of this License Agreement. If you do not agree to any part of this License Agreement, then you may not use the Software and must delete all copies of the Software in your possession or control.  If you are accepting the terms of this Agreement on behalf of a legal entity, you represent and warrant that you have the authority to bind that legal entity to the terms of this License Agreement, and, in such event, "you" and "your" will refer to that legal entity. If you do not have such authority, you must not accept this License Agreement, in which case you may not download or use the Software.  1.  Subject to all terms and conditions of this License Agreement and your timely payment of all subscription fees due hereunder, Bytewax hereby grants you a limited, nonexclusive, nontransferable license (without right to sublicense) to: (i) install and use the object code version of the Software internally for the purpose of ingesting and processing data for your own internal business purposes; and (ii) incorporate the Software, solely in object code form or code artifact, as a component of software-as-a-service offering that is hosted by you ("Application"), provided that such Application provides material additional functionality and features in addition to that provided by the Software.  2.  Subject to all terms and conditions of this License Agreement and your timely payment of all subscription fees due hereunder, Bytewax hereby grants you, a limited, nonexclusive, nontransferable license to internally use any documentation that may be provided by Bytewax regarding the Software ("Documentation") as reasonably necessary for your authorized use of the Software. You may make copies of the Documentation in connection with exercising the foregoing license. Without limiting any other terms and conditions of this License Agreement, you may not transfer or provide the Documentation to any third party.  3.  You shall be entitled to download and use any updated versions of the Software that Bytewax may make available to similarly situated licensees in its sole discretion. Any such updated versions constitute the "Software" for all purposes hereunder. In addition, Bytewax will use commercially reasonable efforts to answer questions regarding use of the Software during Bytewax's normal business hours through such support channels as Bytewax may make available from time to time.  4.  You have no rights or licenses with respect to the Software or Documentation except as expressly provided in this License Agreement. Without limiting the generality of the foregoing, you may not, except to the extent expressly provided for in Sections 1 and 2 above: (a) copy, distribute, rent, lease, lend, sublicense or transfer the Software or Documentation or use the Software or Documentation on a service bureau basis; (b) decompile, reverse engineer, or disassemble the Software or otherwise attempt to discover the source code of the Software, except solely to the extent such acts are authorized under applicable law notwithstanding this prohibition; (c) create derivative works based on the Software or Documentation; (d) modify, remove, or obscure any copyright, trademark, patent or other notices or legends that may appear on the Software or Documentation or during the use and operation thereof; (e) cause the Software to become subject to the terms of any open source license agreement; or (f) deploy your Application in a manner that allows the end user to directly access, download or use the Software other than as incorporated within your Application. acknowledge that certain components of the Software may be covered by open source software licenses. To the extent the terms of such open source licenses prohibit any of the restrictions in this License Agreement with respect to the applicable open source software, such restrictions do not apply. To the extent the terms of such open source licenses require Bytewax to make an offer to provide source code or related information in connection with the open source software, such offer is hereby made. You are not required to provide any ideas, feedback or suggestions regarding the Software (collectively, "Feedback") to Bytewax. To the extent you do provide any Feedback to Bytewax, you acknowledge that Bytewax may freely use, reproduce, modify, distribute, make, have made, sell, offer for sale, import and otherwise exploit in any manner such Feedback without payment of any royalties or other consideration to you.  6.  As part of its operation, the Software may collect and send to Bytewax data regarding the system on which the Software is installed and relating to usage of the Software ("Collected Data"). Collected Data will not include the data that you ingest or process through use of the Software. Collected Data may be used by or on behalf of Bytewax for any lawful purpose, including without limitation to develop and improve Bytewax's products and services and for statistical analysis purposes. In addition, Bytewax will disclose Collected Data where Bytewax, in good faith, believes that the law or legal process (such as a court order, search warrant or subpoena) requires Bytewax to do so or in other circumstances where Bytewax believes it is necessary to protect the rights or property of Bytewax or third parties.  7.  For the avoidance of doubt, you may not directly or indirectly utilize the Software as an embedded component of any device or system or as part of any software application that is distributed. The Software is licensed solely for your internal use or use in an Application as set forth in Section 1. If you wish to request a license for any other purpose, please contact Bytewax at sales@bytewax.io.  8.  You may only make your Application available under terms and conditions that: (i) do not conflict with any provision of this License Agreement or Bytewax's rights in and to the Software; (ii) are not materially less protective of the Software and Bytewax's rights therein and thereto than the provisions of this License Agreement; (iii) do not make any representations, warranties or covenants regarding the Software or on behalf of Bytewax; (iv) state that the Application incorporates the Software and that Bytewax owns all copyright and other intellectual property rights in and to the Software; (v) disclaim any and all warranties and liability on behalf of Bytewax to the maximum extent allowable under applicable law; and (vi) are presented to and accepted by the Application end user in a manner sufficient to create a binding legal agreement under applicable law.  9.  The Software and Documentation are licensed, not sold. The Software and Documentation provided hereunder are the property of Bytewax or its third-party licensors. Bytewax shall retain all right, title and ownership interest and all worldwide patent, copyright, trade secret, trademark and other intellectual property rights and other proprietary rights in and to the Software and Documentation and any copies, derivative works, upgrades, updates, improvements and modifications thereof, in, regardless of the form or media in which, or on which, the original and any other copies, derivative works, upgrades, updates, improvements or modifications may exist.  10.  You agree to pay all subscription fees for the Software for the applicable subscription term you select ("Subscription Term"), as set forth on our website. Each Subscription Term shall automatically renew for a Subscription Term of equal length until you cancel the subscription on or before the end of the then current Subscription Term, unless otherwise terminated in accordance with this License Agreement. You agree to provide a valid credit card or other approved payment mechanism for paying subscription fees and other charges. You authorize us to charge your credit card or other payment mechanism for any subscription fees and charges that you may incur in connection with your account or use of the Software. We reserve the right to change the fee structure and fees charged for the Software at any time. We will provide you with thirty days advance notice (which may be by email) of changes to the fee structure and fees. By continuing use of the Software following that thirty day notice period, you agree to the updated fee structure and fees. You acknowledge that if you object to the proposed change(s), your sole remedy is to cancel your subscription. All fees are exclusive of any taxes required by applicable law, and you agree to pay or reimburse Bytewax for any all taxes relating to this License Agreement, other than taxes based on Bytewax's net income.  11. If you breach any provision of this License Agreement or cancel your subscription, this License Agreement (including all of your rights and licenses with respect to the Software and Documentation) shall immediately terminate without further notice from or action by Bytewax. Bytewax may decline to renew your subscription and terminate this License Agreement at the end of the Subscription Term for any or no reason upon notice (including by email) to you. Upon termination or expiration of this License Agreement, you agree to cease all use of the Software and Documentation (including by removing it from your Application) and to delete all copies thereof in your possession or control. The provisions of Sections 4, 5, 6, 8, 9 (with respect to accrued but unpaid fees), 11 through 22 and this Section 10 shall survive any termination or expiration of this Agreement according to their terms.  12. You agree that Bytewax may state in the public domain that you are a user of the Software, and may include references to and screenshots of your Applications in Bytewax's marketing and publicity materials.  13. THE SOFTWARE AND DOCUMENTATION ARE FURNISHED "AS IS". BYTEWAX MAKES NO WARRANTIES, WHETHER EXPRESS OR IMPLIED WITH RESPECT TO THE SOFTWARE AND THE DOCUMENTATION, INCLUDING WITH RESPECT TO THEIR QUALITY, PERFORMANCE, MERCHANTABILITY, NON-INFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE AND DOCUMENTATION IS WITH YOU. BYTEWAX HAS NO OBLIGATION TO PROVIDE ANY SUPPORT OR ASSISTANCE RELATING TO YOUR USE OF THE SOFTWARE.  14. IN NO EVENT SHALL BYTEWAX BE LIABLE FOR ANY CONSEQUENTIAL, INDIRECT, SPECIAL, OR INCIDENTAL DAMAGES ARISING OUT OF OR RELATING TO THIS LICENSE AGREEMENT OR YOUR USE OF THE SOFTWARE, EVEN IF BYTEWAX HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH POTENTIAL LOSS OR DAMAGE AND REGARDLESS OF THE THEORY OF LIABILITY. IN NO EVENT SHALL BYTEWAX 'S AGGREGATE LIABILITY TO YOU IN RELATION TO THIS LICENSE AGREEMENT EXCEED FEES PAID BY YOU HEREUNDER DURING THE SIX MONTH PERIOD PRECEDING THE DATE OF THE CLAIM. THE FOREGOING LIMITATIONS OF LIABILITY AND EXCLUSIONS OF CERTAIN DAMAGES SHALL APPLY REGARDLESS OF THE EXISTENCE OF MULTIPLE CLAIMS OR THE THEORY OF LIABILITY, AND REGARDLESS OF THE SUCCESS OR EFFECTIVENESS OF OTHER REMEDIES.  15. You agree to indemnify and hold Bytewax and its affiliated companies, and each of their directors, officers, employees, contractors, suppliers and partners, harmless from any claims, losses, damages, liabilities, costs and expenses, including attorney's fees, arising out of or relating to your use or misuse of the Software (including without limitation your commercialization of any Application), breach of this Agreement or violation of the rights of any other person or entity, except solely to the extent any of the foregoing arise out of the willful misconduct or gross negligence of Bytewax or an IP Claim for which Bytewax is obligated under Section 15 below. Bytewax reserves the right, at your expense, to assume the exclusive defense and control of any matter for which you are required to indemnify the Bytewax and you agree to cooperate with Bytewax's defense of these claims.  16. Subject to the terms and conditions hereof, Bytewax agrees to, at its own expense, defend and/or settle any claim, action or suit brought by a third party against you alleging that Software infringes such third party's intellectual property rights (an "IP Claim"), and Bytewax will pay those amounts finally awarded by a court of competent jurisdiction against the Company Indemnitees, or payable pursuant to a settlement agreed to by Bytewax with respect to the IP Claim, provided that you promptly notify Bytewax in writing of the IP Claim, tender sole control of the defense and settlement thereof to Bytewax and provide Bytewax with all reasonably requested information and assistance in connection with the IP Claim. If Bytewax, in its sole discretion, believes an IP Claim or an adverse judgment in connection with an IP Claim is likely, then Bytewax may, at its option, (a) obtain a license from such third party claimant that allows you to continue the use of the Software, (b) modify the Software so as to be non-infringing, or (c) terminate this License Agreement. Bytewax will have no obligation or liability relating to any IP Claim that: (x) is based on modification or customization of the Software by you or any person or entity other than Bytewax; (y) is based on the combination or use of the Software (or any component of either) with any software, hardware, system, method, device or materials not provided by Bytewax; or (z) results from your use of the Software in a manner that is inconsistent with its intended use or is in breach of this Agreement. This Section 15 sets forth the entire liability of Bytewax and your sole and exclusive remedy in the event of any claim that the Software infringes any third party intellectual property right.  17. Bytewax may make modifications, deletions and/or additions to this License Agreement ("Changes") at any time. Changes will be effective: (i) thirty (30) days after Bytewax sends notice of the Changes to the e-mail address associated with your account; or (ii) when you opt-in or otherwise expressly agree to the Changes or a version of this License Agreement incorporating the Changes, whichever comes first. In addition, this License Agreement may be modified by an amendment signed by authorized representatives of you and Bytewax. This License Agreement may not be modified or amended other than as set forth in this Section 16.  18. To the extent that you are an agency or instrumentality of the U.S. government, the parties agree that the Software and Documentation are commercial computer software and commercial computer software documentation, respectively, and that your rights therein are as specified in this License Agreement, per FAR 12.212 and DEARS 227.7202-3, as applicable, or in the case of NASA, subject to NFS 1852.227-86.  19. You may not assign your rights under this License Agreement without the express prior consent of Bytewax. If you are a legal entity, any merger involving you, acquisition of all or substantially all of your assets or change of control shall be deemed an assignment of this Agreement for which prior written consent is required. Bytewax may freely assign this License Agreement.  20. The Software may be subject to export laws and regulations. You agree to comply with any United States and international export laws and regulations that may apply.  21. This License Agreement shall be governed by and construed in accordance with the laws of the state of California, without giving effect to any principles of conflicts of law. Any action arising out of or relating to this License Agreement shall be filed only in the state or federal courts of the Northern District of California and you and Bytewax hereby consent and submit to the personal jurisdiction of such courts for the purposes of litigating any such action.  22. Notwithstanding anything to the contrary, Bytewax may apply to any court of competent jurisdiction for injunctive or other equitable relief.  23. This License Agreement and the User Agreement for Bytewax is the complete and final agreement of the parties with respect to the Software and Documentation. If any part of this License Agreement is found to be void, unenforceable or invalid, that part will be deemed stricken and will not affect the validity of the other provisions. Failure by Bytewax to enforce any provision of this License Agreement will not be deemed a waiver of future enforcement of that or any other provision.


pkgId: 650ccf057e8eeca2757e2a586075f0bc4321be579a8e2c2a4d19d7bf7bc10207
name: python-bytez
arch: src
version: 0.2.28
epoch: 0
release: 1
summary: Python API client for Bytez service
url: https://github.com/bytez-com/docs
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bytez-0.2.28-1.src.rpm
checksum_type: sha256
description:
  - [Basic Usage](#basic-usage)
  - [Installation](#installation)
  - [Authentication and Getting Your Key](#authentication-and-getting-your-key)
  - [List Available Models](#list-available-models)
  - [Initialize the Model Api](#initialize-the-model-api)
  - [Load a Model](#load-a-model)
  - [Check Model Status](#check-model-status)
  - [Run a Model](#run-a-model)
  - [Run a Model with HuggingFace Params](#run-a-model-with-huggingface-params)
  - [Stream the Response](#stream-the-response)
  - [Shutdown a Model](#shutdown-a-model)
  - [List Your Running Instances](#list-your-running-instances)
  - [Request a Huggingface model not yet on Bytez](#request-a-huggingface-model-not-yet-on-bytez)
  - [Request a model not on Huggingface or Bytez](#request-a-model-not-on-huggingface-or-bytez)
  - [Examples](#examples)
    - [Token Classification](#token-classification)
    - [Depth Estimation](#depth-estimation)
    - [Image Classification](#image-classification)
    - [Sentence Similarity](#sentence-similarity)
    - [Image to Text](#image-to-text)
    - [Image Feature Extraction](#image-feature-extraction)
    - [Mask Generation](#mask-generation)
    - [Summarization](#summarization)
    - [Text Classification](#text-classification)
    - [Feature Extraction](#feature-extraction)
    - [Translation](#translation)
    - [Question Answering](#question-answering)
    - [Text to Video](#text-to-video)
    - [Fill Mask](#fill-mask)
    - [Audio Classification](#audio-classification)
    - [Image Segmentation](#image-segmentation)
    - [Visual Question Answering](#visual-question-answering)
    - [Text to Speech](#text-to-speech)
    - [Video Classification](#video-classification)
    - [Object Detection](#object-detection)
    - [Text to Text Generation](#text-to-text-generation)
    - [Zero-Shot Image Classification](#zero-shot-image-classification)
    - [Zero-Shot Classification](#zero-shot-classification)
    - [Document Question Answering](#document-question-answering)
    - [Text Generation](#text-generation)
    - [Unconditional Image Generation](#unconditional-image-generation)
    - [Automatic Speech Recognition](#automatic-speech-recognition)
    - [Zero-Shot Object Detection](#zero-shot-object-detection)
    - [Text to Image](#text-to-image)
    - [Chat Models](#chat-models)
    - [Models with Function Calling](#models-with-function-calling)
  - [Feedback](#feedback)



  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Qwen/Qwen2-7B-Instruct")

  model.load()

  input_text = "Once upon a time there was a beautiful home where"

  model_params = {"max_new_tokens": 20, "max_new_tokens": 5, "temperature": 0.5}

  result = model.run(input_text, model_params=model_params)

  output = result["output"]

  generated_text = output[0]["generated_text"]

  print(generated_text)
  ```

  Streaming usage (only text-generation models support streaming currently)

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Qwen/Qwen2-7B-Instruct")

  model.load()

  input_text = "Once upon a time there was a beautiful home where"

  model_params = {"max_new_tokens": 20, "max_new_tokens": 5, "temperature": 0.5}

  stream = model.run(
      input_text,
      stream=True,
      model_params=model_params,
  )

  for chunk in stream:
      print(f"Output: {chunk}")
  ```



  ```bash
  pip install bytez
  ```



  To use this API, you need an API key. Obtain your key by visiting the [Bytez Settings Page](https://bytez.com/settings)

  ![Bytez Settings Page](https://github.com/user-attachments/assets/884b92b1-021a-4aa4-a150-312ae89f80d0)

  To then use it in code:

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")
  ```



  Lists the currently available models and provides basic information about each one, such as the RAM required to run an instance.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")


  models = client.list_models()

  println(models)


  models_by_task = client.list_models("object-detection")

  println(models_by_task)
  ```



  Initialize a model, so you can check its status, load, run, or shut it down.

  ```py
  model = client.model("openai-community/gpt2")
  ```



  Convenience method for `model.start()`. Automatically waits for the instance to become ready.

  Progress is printed as it executes.

  ```py
  model.load()
  ```

  The options argument is _optional_ and has two properties, concurrency, and timeout.

  ```py
  model.load({
    "concurrency": 1,
    "timeout": 300,
  })
  ```

  ```
  /** concurrency
   * Number of serverless instances.
   *
   * For example, if you set to `3`, then you can do 3 parallel inferences.
   *
   * If you set to `1`, then you can do 1 inference at a time.
   *
   * Default: `1`
   */
  /** timeout
   * Seconds to wait before serverless instance auto-shuts down.
   *
   * By default, if an instance doesn't receive a request after `300` seconds, then it shuts down.
   *
   * Receiving a request resets this timer.
   *
   * Default: `300`
   */
  ```



  Check on the status of the model, to see if it's deploying, running, or stopped.

  ```py
  status = model.status()

  print(status)
  ```



  Run inference.

  ```py
  output = model.run("Once upon a time there was a")

  print(output)
  ```



  Run inference with HuggingFace parameters.

  ```py
  input_text = "Once upon a time there was a small little man who"

  model_params = {"max_new_tokens": 20, "temperature": 2}

  result = model.run(input_text, model_params=model_params)

  print(result)
  ```



  Note: This is only supported for `text-generation` models.

  ```py
  input_text = "Once upon a time there was a beautiful home where"

  model_params = {"max_new_tokens": 20, "max_new_tokens": 5, "temperature": 0.5}

  stream = model.run(
      input_text,
      stream=True,
      model_params=model_params,
  )

  for chunk in stream:
      print(f"Output: {chunk}")
  ```



  By default, models will shut down based on their timeout (seconds) when loaded via `model.start()` or `model.load`.

  To shut down and save costs early, run the following:

  ```py
  model.stop()
  ```



  ```py
  instances = client.list_instances()

  print(instances)
  ```



  To request a model that exists on Huggingface but not yet on Bytez, you can do the following:

  ```py
  model_id = "openai-community/gpt2"

  job_status = client.process(model_id)

  print(job_status)
  ```

  This sends a job to an automated queue. When the job completes, you'll receive an email indicating the model is ready for use with the models API.



  Please reach out to us and we'll do what's necessary to make other models available!

  Please join our [Discord](https://discord.gg/Zrd5UbMEBA) or contact us via email at [help@bytez.com](mailto:help@bytez.com)



  Below are examples of using various models with the Bytez API in Python.

  All examples are also located [here](https://github.com/Bytez-com/docs/tree/main/examples/python) under the `tasks` directory.



  Token classification involves identifying and categorizing tokens in a text. Common use cases include Named Entity Recognition (NER), Part-of-Speech tagging, and other NLP tasks.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("dslim/bert-base-NER")

  model.load()

  input_text = "John Doe is a software engineer at Google."

  result = model.run(input_text)

  word_objects = result["output"]

  for word_object in word_objects:
      
      print(word_object)

      word = word_object.get("word")
      entity = word_object.get("entity")
      score = word_object.get("score")
      index = word_object.get("index")
      start = word_object.get("start")
      end = word_object.get("end")

      print(
          {
              "word": word,
              "entity": entity,
              "score": score,
              "index": index,
              "start": start,
              "end": end,
          }
      )
  ```



  Depth estimation involves predicting the distance of objects from the camera. Use cases include robotics, augmented reality, and autonomous vehicles.

  ```py
  import os
  import requests
  import base64
  from bytez import Bytez

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))


  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("vinvino02/glpn-nyu")

  model.load()

  input_image_url = "https://as1.ftcdn.net/v2/jpg/03/03/55/82/1000_F_303558268_YNUQp9NNMTE0X4zrj314mbWcDHd1pZPD.jpg"


  result = model.run(input_image_url)

  output = result.get("output")

  depth_png = output["depth_png"]
  formatted_predicted_depth_array = output["formatted_predicted_depth_array"]


  img_buffer = base64.b64decode(depth_png)


  image_path = f"{WORKING_DIR}/testImage.png"
  with open(image_path, "wb") as f:
      f.write(img_buffer)


  original_image_path = f"{WORKING_DIR}/originalImage.jpg"
  response = requests.get(input_image_url)

  with open(original_image_path, "wb") as f:
      f.write(response.content)

  print("Wrote the original image to:", original_image_path)
  print("Wrote the inference image to:", image_path)


  rows = formatted_predicted_depth_array
  for j, row in enumerate(rows):
      for i, pixel in enumerate(row):
          
          pass
  ```



  Image classification involves categorizing images into predefined classes. Use cases include object recognition, medical imaging, and security systems.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("google/vit-base-patch16-224")

  model.load()

  input_image_url = (
      "https://www.padoniavets.com/sites/default/files/field/image/cats-and-dogs.jpg"
  )

  result = model.run(input_image_url)

  labelObjects = result.get("output")

  for labelObject in labelObjects:
      
      print(labelObject)

      label = labelObject["label"]
      score = labelObject["score"]

      print({"label": label, "score": score})
  ```



  Sentence similarity involves measuring how similar two sentences are. Use cases include duplicate question detection, paraphrase detection, and text clustering.

  ```py
  from bytez import Bytez
  import numpy as np

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("sentence-transformers/all-MiniLM-L6-v2")

  model.load()

  sentences = [
      "What is the weather like today?",
      "Is it sunny today?",
      "The e39 BMW M5 was one of the best production sport sedans ever produced.",
  ]

  results = []


  for sentence in sentences:
      result = model.run(sentence)
      embedding = result.get("output")

      print({"embedding": embedding})

      results.append({"embedding": embedding, "sentence": sentence})


  original_sentence_with_embedding = results[0]
  sentences_to_compare = results[1:]


  def cosine_similarity(embedding1, embedding2):
      embedding1 = np.array(embedding1)
      embedding2 = np.array(embedding2)

      dot_product = np.dot(embedding1, embedding2)
      magnitude1 = np.linalg.norm(embedding1)
      magnitude2 = np.linalg.norm(embedding2)

      similarity = dot_product / (magnitude1 * magnitude2)

      return similarity



  for sentence_object in sentences_to_compare:
      similarity = cosine_similarity(
          original_sentence_with_embedding["embedding"], sentence_object["embedding"]
      )
      print(
          f'Cosine similarity between "{original_sentence_with_embedding["sentence"]}" and "{sentence_object["sentence"]}": {similarity}'
      )
  ```



  Image to text involves generating textual descriptions of images. Use cases include image captioning, content generation, and accessibility features.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  input_image_url = "https://as1.ftcdn.net/v2/jpg/03/03/55/82/1000_F_303558268_YNUQp9NNMTE0X4zrj314mbWcDHd1pZPD.jpg"

  model = client.model("Salesforce/blip-image-captioning-base")

  model.load()

  result = model.run(input_image_url)


  output = result.get("output")

  print(output)

  generated_text = output[0]["generated_text"]

  print(generated_text)
  ```



  Image feature extraction involves extracting features from images for tasks like object detection, image classification, and image retrieval.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("nomic-ai/nomic-embed-vision-v1")

  model.load()

  input_image_url = "https://as1.ftcdn.net/v2/jpg/03/03/55/82/1000_F_303558268_YNUQp9NNMTE0X4zrj314mbWcDHd1pZPD.jpg"

  result = model.run(input_image_url)

  output = result.get("output")


  print(output)

  embedding = output[0]

  print(embedding)
  ```



  Mask generation involves generating masks for objects in images. Use cases include image segmentation, medical imaging, and computer vision tasks.

  ```py
  import os
  import requests
  import base64
  import json
  from PIL import Image
  from io import BytesIO
  import numpy as np
  from bytez import Bytez

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))


  def get_base64_image(url):
      response = requests.get(url)
      response.raise_for_status()  # Ensure the request was successful
      image_bytes = response.content
      return base64.b64encode(image_bytes).decode("utf-8")


  input_image_base64 = get_base64_image(
      "https://huggingface.co/datasets/huggingfacejs/tasks/resolve/main/mask-generation/mask-generation-input.png"
  )

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("facebook/sam-vit-base")

  model.load()

  result = model.run({"b64ImageBufferPng": input_image_base64})

  output = result.get("output")

  input_image_dimensions = output["input_image_dimensions"]
  masks = output["masks"]
  scores = output["scores"]

  print(
      {
          "input_image_dimensions": input_image_dimensions,
          "masks": masks,
          "scores": scores,
      }
  )


  test_masks_path = f"{WORKING_DIR}/testMasks.json"
  with open(test_masks_path, "w") as file:
      file.write(json.dumps(masks, indent=2))



  with open(test_masks_path, "r") as file:
      test_masks = json.loads(file.read())


  def write_masks_to_image(input_image_base64, masks):
      
      src_img_bytes = base64.b64decode(input_image_base64)
      src_img = Image.open(BytesIO(src_img_bytes)).convert("RGBA")
      src_img_np = np.array(src_img)

      
      colors = [
          (255, 0, 0, 50),  # Red
          (0, 255, 0, 50),  # Green
          (0, 0, 255, 50),  # Blue
          (255, 255, 0, 50),  # Yellow
      ]

      
      for mask_index, mask in enumerate(masks):
          color = colors[mask_index % len(colors)]
          for i in range(len(mask)):
              for j in range(len(mask[i])):
                  if mask[i][j]:
                      src_img_np[i, j] = color

      
      img_with_masks = Image.fromarray(src_img_np, "RGBA")
      image_path = f"{WORKING_DIR}/testImage.png"
      img_with_masks.save(image_path)


  write_masks_to_image(input_image_base64, test_masks)
  ```



  Summarization involves creating concise summaries of longer texts. Use cases include news summarization, document summarization, and generating abstracts.

  ```py
  from bytez import Bytez


  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("ainize/bart-base-cnn")

  model.load()

  input_text = """
  The Big Bang is a physical theory that describes how the universe expanded from an initial state of high density and temperature.[1] The notion of an expanding universe was first scientifically originated by physicist Alexander Friedmann in 1922 with the mathematical derivation of the Friedmann equations.[2][3][4][5]

  Independent of Friedmann's work, the Big Bang was first proposed in 1931 by Roman Catholic priest and physicist Georges Lemaître when he suggested the universe emerged from a "primeval atom". Various cosmological models of the Big Bang explain the evolution of the observable universe from the earliest known periods through its subsequent large-scale form.[6][7][8] These models offer a comprehensive explanation for a broad range of observed phenomena, including the abundance of light elements, the cosmic microwave background (CMB) radiation, and large-scale structure. The uniformity of the universe, known as the flatness problem, is explained through cosmic inflation: a sudden and very rapid expansion of space during the earliest moments.

  Crucially, these models are compatible with the Hubble–Lemaître law—the observation that the farther away a galaxy is, the faster it is moving away from Earth. Extrapolating this cosmic expansion backward in time using the known laws of physics, the models describe an increasingly concentrated cosmos preceded by a singularity in which space and time lose meaning (typically named "the Big Bang singularity").[9] Physics lacks a widely accepted theory of quantum gravity that can model the earliest conditions of the Big Bang. In 1964 the CMB was discovered, which convinced many cosmologists that the competing steady-state model of cosmic evolution was falsified, since the Big Bang models predict a uniform background radiation caused by high temperatures and densities in the distant past.[10] A wide range of empirical evidence strongly favors the Big Bang event, which is now essentially universally accepted.[11] Detailed measurements of the expansion rate of the universe place the Big Bang singularity at an estimated 13.787±0.020 billion years ago, which is considered the age of the universe.[12]

  There remain aspects of the observed universe that are not yet adequately explained by the Big Bang models. After its initial expansion, the universe cooled sufficiently to allow the formation of subatomic particles, and later atoms. The unequal abundances of matter and antimatter that allowed this to occur is an unexplained effect known as baryon asymmetry. These primordial elements—mostly hydrogen, with some helium and lithium—later coalesced through gravity, forming early stars and galaxies. Astronomers observe the gravitational effects of an unknown dark matter surrounding galaxies. Most of the gravitational potential in the universe seems to be in this form, and the Big Bang models and various observations indicate that this excess gravitational potential is not created by baryonic matter, such as normal atoms. Measurements of the redshifts of supernovae indicate that the expansion of the universe is accelerating, an observation attributed to an unexplained phenomenon known as dark energy.[13]
  """

  result = model.run(input_text, model_params={"max_length": 40})


  output = result["output"]

  summary_text = output[0]["summary_text"]

  print(summary_text)
  ```



  Text classification involves categorizing text into predefined classes. Use cases include sentiment analysis, spam detection, and topic classification.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("AdamCodd/distilbert-base-uncased-finetuned-sentiment-amazon")

  model.load()

  input_text = "We are furious with the results of the experiment!"

  result = model.run(input_text)


  label_objects = result["output"]

  for label_object in label_objects:
      
      print(label_object)

      
      label = label_object.get("label")
      score = label_object.get("score")

      print({"label": label, "score": score})
  ```



  Feature extraction involves extracting features from data for further processing. Use cases include data preprocessing, embedding generation, and similarity search.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Salesforce/SFR-Embedding-2_R")

  model.load()

  input_text = "Your text for feature extraction goes here..."

  result = model.run(input)

  output = result.get("output")


  print(output)

  embedding = output[0]

  print(embedding)
  ```



  Translation involves translating text from one language to another. Use cases include multilingual communication, content localization, and language learning.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Helsinki-NLP/opus-mt-en-zh")

  model.load()

  input_text = "Hello, how are you? Beautiful day today, isn't it?"

  result = model.run(input_text)


  output = result["output"]

  translation_text = output[0]["translation_text"]

  print(translation_text)
  ```



  Question answering involves answering questions based on a given context. Use cases include customer support, information retrieval, and educational tools.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("deepset/roberta-base-squad2")

  model.load()

  qa_input = {
      "question": "Where does Holly live?",
      "context": "My name is Holly and I live in NYC",
  }

  result = model.run(qa_input)

  output = result.get("output")


  print(output)

  answer = output.get("answer")
  score = output.get("score")
  start = output.get("start")
  end = output.get("end")

  print({"answer": answer, "score": score, "start": start, "end": end})
  ```



  Text to video involves generating videos from textual descriptions. Use cases include content creation, entertainment, and education.

  ```py
  import os
  import base64
  from bytez import Bytez


  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("ali-vilab/text-to-video-ms-1.7b")

  model.load()

  input_text = "A cat playing with a rose"

  result = model.run(input_text)

  output_mp4 = result.get("output_mp4")


  video_bytes = base64.b64decode(output_mp4)


  output_path = os.path.join(WORKING_DIR, "output.mp4")
  with open(output_path, "wb") as video_file:
      video_file.write(video_bytes)

  print(f"Video successfully saved to {output_path}")
  ```



  Fill mask involves predicting missing words in a sentence. Use cases include text completion, language modeling, and text generation.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("almanach/camembert-base")

  model.load()

  input_text = "The capital of France is <mask>."

  result = model.run(input_text)

  sequence_objects = result.get("output")


  for sequence_object in sequence_objects:
      
      print(sequence_object)

      sequence = sequence_object["sequence"]
      score = sequence_object["score"]
      token = sequence_object["token"]
      token_str = sequence_object["token_str"]

      print(
          {"sequence": sequence, "score": score, "token": token, "token_str": token_str}
      )
  ```



  Audio classification involves categorizing audio clips into predefined classes. Use cases include speech emotion recognition, sound detection, and music genre classification.

  ```py
  import requests
  import base64
  from bytez import Bytez


  def get_base64_audio(url):
      response = requests.get(url)
      return base64.b64encode(response.content).decode("utf-8")


  input_audio_base64 = get_base64_audio(
      "https://huggingface.co/datasets/huggingfacejs/tasks/resolve/main/audio-classification/audio.wav"
  )

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("aaraki/wav2vec2-base-finetuned-ks")

  model.load()

  result = model.run({"b64AudioBufferWav": input_audio_base64})

  label_objects = result["output"]

  for label_object in label_objects:
      
      print(label_object)

      score = label_object["score"]
      label = label_object["label"]

      print({"score": score, "label": label})
  ```



  Image segmentation involves dividing an image into multiple segments. Use cases include medical imaging, object detection, and computer vision tasks.

  ```py
  import os
  import base64
  from bytez import Bytez

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("sayeed99/segformer-b3-fashion")

  model.load()

  input_image_url = "https://ocean.si.edu/sites/default/files/styles/3_2_largest/public/2023-11/Screen_Shot_2018-04-16_at_1_42_56_PM.png.webp?itok=Icvi-ek9"

  result = model.run(input_image_url)

  mask_objects = result.get("output")

  for index, mask_object in enumerate(mask_objects):
      
      print(mask_object)

      label = mask_object.get("label")
      score = mask_object.get("score")
      mask_png: str = mask_object.get("mask_png")

      print({"label": label, "score": score})

      mask_png_buffer = base64.decodebytes(mask_png.encode("utf-8"))

      with open(f"{WORKING_DIR}/mask-{index}.png", "wb") as file:
          file.write(mask_png_buffer)
  ```



  Visual question answering involves answering questions based on an image. Use cases include interactive learning, accessibility features, and content analysis.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Salesforce/blip-vqa-base")

  model.load()

  input_data = {
      "image": "https://ocean.si.edu/sites/default/files/styles/3_2_largest/public/2023-11/Screen_Shot_2018-04-16_at_1_42_56_PM.png.webp?itok=Icvi-ek9",
      "question": "What kind of animal is this?",
  }

  result = model.run(input_data)

  outputs = result["output"]

  answer = outputs[0]["answer"]

  print(answer)
  ```



  Text to speech involves converting text into spoken words. Use cases include virtual assistants, accessibility features, and content creation.

  ```py
  from bytez import Bytez
  import base64
  import os

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("suno/bark-small")

  model.load()

  input_text = "Hello, how are you today?"

  result = model.run(input_text)

  output_wav = result.get("output_wav")


  audio_bytes = base64.b64decode(output_wav)


  output_path = os.path.join(WORKING_DIR, "output.wav")
  with open(output_path, "wb") as audio_file:
      audio_file.write(audio_bytes)

  print(f"Audio successfully saved to {output_path}")
  ```



  Video classification involves categorizing videos into predefined classes. Use cases include video content analysis, security surveillance, and media organization.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("ahmedabdo/video-classifier")

  model.load()

  input_video_url = "https://video-previews.elements.envatousercontent.com/6d07b79d-b17a-47b5-9d24-4fe984c7ca36/watermarked_preview/watermarked_preview.mp4"

  result = model.run(input_video_url)

  outputs = result["output"]

  label_objects = outputs[0]


  label_objects.sort(key=lambda x: x["score"], reverse=True)

  for label_object in label_objects:
      
      print(label_object)

      
      score = label_object.get("score")
      label = label_object.get("label")

      print({"score": score, "label": label})
  ```



  Object detection involves identifying and locating objects in an image or video. Use cases include security systems, autonomous driving, and retail analytics.

  ```py
  import os
  import requests
  from PIL import Image, ImageDraw, ImageFont
  from bytez import Bytez
  import io

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("facebook/detr-resnet-50")

  model.load()

  img_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/1200px-Cat_November_2010-1a.jpg"

  result = model.run(img_url)

  box_objects = result.get("output")

  print(box_objects)

  for box_object in box_objects:
      
      print(box_object)

      score = box_object["score"]
      label = box_object["label"]
      box = box_object["box"]

      print({"score": score, "label": label, "box": box})


  response = requests.get(img_url)
  image_bytes = response.content


  image = Image.open(io.BytesIO(image_bytes)).convert("RGBA")
  image_buffer_png = io.BytesIO()
  image.save(image_buffer_png, format="PNG")
  image_buffer_png.seek(0)



  def debug_image(src_img_buffer, box_objects):
      src_img = Image.open(src_img_buffer).convert("RGBA")
      draw = ImageDraw.Draw(src_img)
      colors = [
          (255, 0, 0, 128),  # Red
          (0, 255, 0, 128),  # Green
          (0, 0, 255, 128),  # Blue
          (255, 255, 0, 128),  # Yellow
      ]

      font = ImageFont.load_default()

      for index, box_object in enumerate(box_objects):
          box = box_object["box"]
          score = box_object["score"]
          label = box_object["label"]
          color = colors[index % len(colors)]

          
          draw.rectangle(
              [(box["xmin"], box["ymin"]), (box["xmax"], box["ymax"])],
              outline=color,
              width=2,
          )

          
          text = f"{label} ({score:.2f})"

          
          text_width, text_height = draw.textbbox((0, 0), text, font=font)[2:]

          
          draw.rectangle(
              [
                  (box["xmin"], box["ymin"] - text_height - 2),
                  (box["xmin"] + text_width, box["ymin"]),
              ],
              fill=color,
          )
          draw.text(
              (box["xmin"], box["ymin"] - text_height - 2),
              text,
              fill=(255, 255, 255, 255),
              font=font,
          )

      
      output_path = f"{WORKING_DIR}/testImage.png"
      src_img.save(output_path)


  debug_image(image_buffer_png, box_objects)
  ```



  Text to text generation involves generating text from input text. Use cases include text completion, content generation, and dialogue systems.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("google/flan-t5-base")

  model.load()

  input_text = "Once upon a time there was a small little man who"

  model_params = {"max_new_tokens": 20, "temperature": 2}

  result = model.run(input_text, model_params=model_params)

  output = result["output"]

  generated_text = output[0]["generated_text"]

  print(generated_text)
  ```



  Zero-shot image classification involves classifying images into classes not seen during training. Use cases include novel object recognition, transfer learning, and few-shot learning.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("BilelDJ/clip-hugging-face-finetuned")

  model.load()

  input_data = {
      "image": "https://as1.ftcdn.net/v2/jpg/03/03/55/82/1000_F_303558268_YNUQp9NNMTE0X4zrj314mbWcDHd1pZPD.jpg",
      "candidate_labels": ["squid", "octopus", "human", "cat"],
  }

  result = model.run(input_data)


  label_objects = result["output"]


  label_objects.sort(key=lambda x: x["score"], reverse=True)

  for label_object in label_objects:
      
      print(label_object)

      
      score = label_object["score"]
      label = label_object["label"]

      print({"score": score, "label": label})
  ```



  Zero-shot classification involves classifying text into classes not seen during training. Use cases include intent detection, content moderation, and dynamic classification.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("facebook/bart-large-mnli")

  model.load()

  input_data = {
      "text": "One day I will see the world",
      "candidate_labels": ["travel", "cooking", "dancing"],
  }

  result = model.run(input_data)

  output = result["output"]
  sequence = output["sequence"]
  labels = output["labels"]
  scores = output["scores"]


  label_objects = [
      {"sequence": sequence, "label": v, "score": scores[i]} for i, v in enumerate(labels)
  ]


  label_objects.sort(key=lambda x: x["score"], reverse=True)

  for label_object in label_objects:
      sequence = label_object["sequence"]
      label = label_object["label"]
      score = label_object["score"]

      print({"sequence": sequence, "label": label, "score": score})
  ```



  Document question answering involves answering questions based on the content of documents. Use cases include document understanding, contract analysis, and information retrieval.

  ```py
  import os
  from bytez import Bytez

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("cloudqi/CQI_Visual_Question_Awnser_PT_v0")

  model.load()

  input_data = {
      "image": "https://templates.invoicehome.com/invoice-template-us-neat-750px.png",
      "question": "What's the total cost?",
  }

  result = model.run(input_data)

  output = result.get("output")


  print(output)

  output_object: dict = output[0]

  answer = output_object.get("answer")
  score = output_object.get("score")
  start = output_object.get("start")
  end = output_object.get("end")

  print({"answer": answer, "score": score, "start": start, "end": end})
  ```



  Text generation involves generating coherent text from an initial prompt. Use cases include story generation, dialogue systems, and creative writing.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Qwen/Qwen2-7B-Instruct")

  model.load()

  input_text = "Once upon a time there was a beautiful home where"

  model_params = {"max_new_tokens": 20, "max_new_tokens": 5, "temperature": 0.5}

  stream = model.run(
      input_text,
      stream=True,
      model_params=model_params,
  )

  for chunk in stream:
      print(f"Output: {chunk}")
  ```



  Unconditional image generation involves generating images without any specific conditions or inputs. Use cases include art generation, creative design, and data augmentation.

  ```py
  import os
  import base64
  from bytez import Bytez


  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("afshr/cam_finetune")

  model.load()

  input_text = "A rose"

  result = model.run(input_text)

  output_png = result.get("output_png")


  image_bytes = base64.b64decode(output_png)


  output_path = f"{WORKING_DIR}/output.png"
  with open(output_path, "wb") as image_file:
      image_file.write(image_bytes)

  print(f"Image successfully saved to {output_path}")
  ```



  Automatic speech recognition involves converting spoken language into written text. Use cases include transcription services, voice assistants, and accessibility features.

  ```py
  import requests
  import base64
  from bytez import Bytez


  def get_base64_audio(url):
      response = requests.get(url)
      return base64.b64encode(response.content).decode("utf-8")


  input_audio_base64 = get_base64_audio(
      "https://huggingface.co/datasets/huggingfacejs/tasks/resolve/main/audio-classification/audio.wav"
  )

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("facebook/data2vec-audio-base-960h")

  model.load()

  result = model.run({"b64AudioBufferWav": input_audio_base64})

  output = result["output"]


  print(output)

  text = output["text"]

  print("Inference is: ", text)
  ```



  Zero-shot object detection involves detecting objects in images without prior training on those specific objects. Use cases include novel object detection, transfer learning, and few-shot learning.

  ```py
  import requests
  import base64
  from bytez import Bytez


  def get_base64_image(url):
      response = requests.get(url)
      response.raise_for_status()
      image_bytes = response.content
      return base64.b64encode(image_bytes).decode("utf-8")


  input_image_base64 = get_base64_image(
      "https://ocean.si.edu/sites/default/files/styles/3_2_largest/public/2023-11/Screen_Shot_2018-04-16_at_1_42_56_PM.png.webp?itok=Icvi-ek9"
  )

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("BilelDJ/clip-hugging-face-finetuned")

  model.load()

  input_data = {
      "b64ImageBufferPng": input_image_base64,
      "candidate_labels": ["squid", "octopus", "human", "cat"],
  }

  result = model.run(input_data)

  label_objects = result["output"]


  label_objects.sort(key=lambda x: x["score"], reverse=True)

  for label_object in label_objects:
      
      print(label_object)

      
      score = label_object["score"]
      label = label_object["label"]

      print({"score": score, "label": label})
  ```



  Text to image involves generating images from textual descriptions. Use cases include content creation, advertising, and creative design.

  ```py
  from bytez import Bytez
  import base64
  import os

  WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("dreamlike-art/dreamlike-photoreal-2.0")

  model.load()

  input_text = "A beautiful landscape with mountains and a river"

  result = model.run(input_text)

  output_png = result.get("output_png")


  image_bytes = base64.b64decode(output_png)


  output_path = os.path.join(WORKING_DIR, "output.png")
  with open(output_path, "wb") as image_file:
      image_file.write(image_bytes)

  print(f"Image successfully saved to {output_path}")
  ```



  Chat models are used to create interactive conversational agents. These models can engage in dialogue with users, respond to questions, and provide information or entertainment.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("microsoft/Phi-3-mini-4k-instruct")

  model.load()

  messages = [
      {"role": "system", "content": "You are a friendly chatbot"},
      {"role": "user", "content": "What is the capital of England?"},
  ]

  result = model.run(messages, model_params={"max_length": 100})

  output = result.get("output")

  generated_text = output[0]["generated_text"]


  for message in generated_text:
      
      print(message)

      content = message["content"]
      role = message["role"]

      print({"content": content, "role": role})
  ```



  Some models support function calling, allowing them to interact with user-defined functions.

  ```py
  from bytez import Bytez

  client = Bytez("YOUR_BYTEZ_KEY_HERE")

  model = client.model("Nexusflow/NexusRaven-V2-13B")

  model.load()

  input_text = "What's the weather like in Seattle right now?"

  model_params = {
      "max_new_tokens": 20,
      "max_new_tokens": 5,
      "temperature": 0.001,
      "do_sample": False,
  }

  prompt_template = '''
  Function:
  def get_weather_data(coordinates):
      """
      Fetches weather data from the Open-Meteo API for the given latitude and longitude.

      Args:
      coordinates (tuple): The latitude and longitude of the location.

      Returns:
      float: The current temperature in the coordinates you've asked for
      """

  Function:
  def get_coordinates_from_city(city_name):
      """
      Fetches the latitude and longitude of a given city name using the Maps.co Geocoding API.

      Args:
      city_name (str): The name of the city.

      Returns:
      tuple: The latitude and longitude of the city.
      """

  User Query: {query}<human_end>
  '''


  prompt = prompt_template.format(query=input_text)

  stream = model.run(prompt, model_params=model_params, stream=True)
  for chunk in stream:
      print(f"Output: {chunk}")
  ```



  We value your feedback to improve our documentation and services. If you have any suggestions, please join our [Discord](https://discord.gg/Zrd5UbMEBA) or contact us via email at [help@bytez.com](mailto:help@bytez.com)


pkgId: c421946912e3a0c14716569c798b0afec7078b6b1a7a0bb45a0a32142fc751eb
name: python-byu-awslogin
arch: src
version: 0.15.2
epoch: 0
release: 1
summary: An aws-adfs spinoff that fits BYU's needs
url: https://github.com/byu-oit/awslogin
rpm_license: Apache-2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byu-awslogin-0.15.2-1.src.rpm
checksum_type: sha256
description:
  Python script for CLI and SDK access to AWS via ADFS while requiring MFA
  access using <https://duo.com/>



  BYU used to use the great [aws-adfs](https://github.com/venth/aws-adfs)
  CLI tool to login to our AWS accounts. It worked great, especially the
  DUO 2FA support. Eventually, we decided to write our own similar tool
  but make it BYU-specific so that we could tailor it to our needs (which
  basically means hard-code certain BYU-specific things) and remove some
  of the required parameters. Since this tool will be used by BYU
  employees only we had that option. We then morphed it a little more for
  our use cases. This isn't something that you could use outside of BYU,
  sorry.


  In order for Duo 2FA to work properly Automatic Push needs to be enabled.



    - Python 3.6+ is recommended as python2 is EOL January 2020.
    - It is highly recommended to use an application like [Pipx](https://pipxproject.github.io/pipx/) to install and use python cli applications.
    - Follow the pipx [installation documentation](https://pipxproject.github.io/pipx/installation/) then simply run `pipx install byu_awslogin`
    - Experimental Binaries are available on the releases page. These are new and in testing [Releases](https://gihtub.com/byu-oit/awslogin/releases)
    - See the [installation options](https://github.com/byu-oit/awslogin/blob/master/INSTALLATION_OPTIONS.md) For additional options
      page for step by step instructions for installing in various environments



  If you already have byu\_awslogin install and are looking to upgrade
  simply run

  `pip3 install --upgrade byu_awslogin` or `pip install --upgrade
  byu_awslogin` as appropriate for your python installation



  awslogin defaults to the default profile in your \~/.aws/config and
  \~/.aws/credentials files. **\*If you already have a default profile you
  want to save in your \~/.aws files make sure to do that before running
  awslogin.**\*

  Once you're logged in, you can execute commands using the AWS CLI or AWS
  SDK. Try running `aws s3 ls`.

  Currently, AWS temporary credentials are only valid for 1 hour. We cache
  your ADFS session, however, so you can just re-run `awslogin` again to
  get a new set of AWS credentials without logging in again to ADFS. Your
  ADFS login session is valid for 8 hours, after which time you'll be
  required to login to ADFS again to obtain a new session.

  To switch accounts after you've already authenticated to an account,
  just run awslogin again and select a new account/role combination.

  To use it:

    - Run `awslogin` and it will prompt you for the AWS account and role
      to use.
    - Run `awslogin --account <account name> --role <role name>` to skip
      the prompting for account and name. You could specify just one of
      the arguments as well.
    - Run `awslogin --profile <profile name>` to specifiy an alternative
      profile
    - Run `awslogin --region <region name>` to specify a different region.
      The default region is *us-west-2*.
    - Run `awslogin --status` for the current status of the default
      profile
    - Run `awslogin --status --profile dev` for the current status of the
      dev profile
    - Run `awslogin --status --profile all` for the current status of the
      all profiles
    - Run `awslogin --logout` to logout of a cached ADFS session
    - Run `awslogin --version` to display the running version of awslogin
    - Run `awslogin --help` for full help message


  Bash:
  - Run the following: `_AWSLOGIN_COMPLETE=source awslogin > ~/_awslogin` Then add `source /path/to/_awslogin` to .bashrc

  ZSH:
  - Run the following: `_AWSLOGIN_COMPLETE=source_zsh awslogin > ~/_awslogin` Then add `source /path/to/_awslogin` to .zshrc

  Alternatively put the `_awslogin` in your `/etc/bash_completion.d` or similiar directory (`~/.zfunc`) to load at shells startup

  To test if it works run awslogin at least once for the account and role cache to populate. On next login `awslogin -a [TAB][TAB]` should output available accounts and `awslogin -a {some account} -r [TAB][TAB]` should output available roles for the selected account

  Limitation: Accounts and Role completion at the CLI is loaded from a cache file. This file will be updated anytime awslogin is ran.



    - Enter an issue on the github repo.
    - Or, even better if you can, fix the issue and make a pull request.



    - Update the version in `pyproject.toml` and `__version__.py`
    - Commit the changes and push.
    - Build binaries
    - Create a new release (add binaries and sha256sums.txt) with the version number and Github Actions will build, test and publish



    - Write tests
      - Write more tests to increase overall coverage


pkgId: eea9994d61e9552a1421614b618406f7117ed7aa8a3a47bcd514552f3338329c
name: python-byzfl
arch: src
version: 0.0.1
epoch: 0
release: 1
summary: ByzFL Library
url: https://github.com/LPD-EPFL/byzfl
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-byzfl-0.0.1-1.src.rpm
checksum_type: sha256
description:
  This tool facilitates the testing of aggregations and attacks by simulating distributed machine learning environments in a fully configurable manner via the `settings.json` file.


  1. Install the dependencies listed in `requirements.txt`.
  2. Set the environment variable `CUBLAS_WORKSPACE_CONFIG=:4096:8`


  0. If this is your first time running the framework, execute `python main.py`. This will create a default settings.json file, which you can customize according to your requirements.
  1. Configure the experiment you wish to run in the `settings.json` file. Refer to the above section on how to configure it.
  2. Run the framework with `python main.py --nb_jobs n` where `n = how many trainings will be done in paralel`.
  3. When the code have finished, run `python evaluate_results.py` to choose the best hyperparameters and create the heatmaps with the results.


  - **General**:
      - **training_seed**: Starting training seed for the different seeds will be used.
      - **nb_training_seeds**: How many seeds will be done starting from the training_seed.
      - **device**: Device used in PyTorch to do the computations with torch tensors.
      - **nb_workers**: Number of workers the training will have. If null, this number will be nb_honest + nb_byz.
      - **nb_honest**: Number of honest clients.
      - **nb_byz**: Number of real byzantine clients.
      - **declared_nb_byz**: Number of byzantine clients the server is protecting against.
      - **declared_equal_real**: Boolean to filter configurations that don't have the same declared byzantine clients as the real ones.
      - **size_train_set**: Proportion of the training set that will be used for training; otherwise, it will be used as the validation set.
      - **nb_steps**: Number of iterations that the learning algorithm will perform.
      - **evaluation_delta**: Number of steps between evaluations of the model over the validation set or test set.
      - **evaluate_on_test**: Boolean to configure if test accuracy should be computed.
      - **store_training_accuracy**: Boolean to set if training accuracies will be stored.
      - **store_training_loss**: Boolean to set if training losses will be stored.
      - **store_models**: Boolean to set if the state dict of the PyTorch models should be stored every evaluation delta.
      - **batch_size_validation**: Batch size for the validation and test datasets, but not for the training set.
      - **data_folder**: Where the datasets will be stored.
      - **results_directory**: Where the information we desire to save of the training will be stored.

  - **Model**:
      - **name**: Name of the model we want to use in the training. This name must be the name of a class in the **models.py** file.
      - **dataset_name**: Name of the dataset used in training. Must be one of **dataset.py**.
      - **nb_labels**: Number of different targets that the used dataset has.
      - **data_distribution_seed**: Starting data distribution seed for the different seeds will be used. This seed is used to have different data splits among nodes.
      - **nb_data_distribution_seeds**: How many seeds will be done starting from the data_distribution_seed.
      - **data_distribution**:
          - **name**: Name of the data distribution that will be used.
          - **distribution_parameter**: Float with the parameter that will be used by the distribution.
      - **loss**: Pytorch training loss.

  - **Aggregator**:
      - **name**: Name of the aggregation used by the server. Must be one of the **aggregators.py**.
      - **parameters**: Dictionary with the parameters required by the aggregation used.

  - **PreAggregators**: List of pre-aggregations in order as they should be applied.
      - **name**: Name of the pre-aggregation used by the server. Must be defined in **preaggregators.py**.
      - **parameters**: Dictionary with the parameters required by the pre-aggregations used.

  - **Server**:
      - **batch_norm_momentum**: momentum for the federated batch norm.
      - **learning_rate_decay**: Factor of learning rate decay that will be applied every milestone.
      - **milestones**: List of steps at which the learning rate decay should be applied.

  - **Honest Nodes**:
      - **momentum**: Momentum value used in the training algorithm.
      - **batch_size**: Batch size for the SGD algorithm.
      - **learning_rate**: Learning rate value.
      - **weight_decay**: Weight decay value to avoid overfitting.

  - **Attack**:
      - **name**: Name of the attack used by the server. Must be defined in **attacks.py**.
      - **parameters**: Dictionary with the parameters required by the attack used.
      - **attack_optimizer**:
          - **name**: Name of the optimizer that will be used to optimize the attack factor.
          - **parameters**: Dictionary with the parameters required by the optimizer.


  Please note that this library is designed to run multiple settings simultaneously, which is essential for thoroughly exploring aggregations and attacks. To achieve this, you can provide a list of elements in the settings.json file instead of a single element. Below is an example:
  `distribution_parameter: 1.0` -> `distribution_parameter: [1.0, 0.5, 0.0]`


pkgId: 40b0f15e60a8504a70bc1298472ac0365c0ae99cbe9a839c5374b7e5af3ff846
name: python-bz2file
arch: src
version: 0.98
epoch: 0
release: 1
summary: Read and write bzip2-compressed files.
url: https://github.com/nvawda/bz2file
rpm_license: Apache License, Version 2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bz2file-0.98-1.src.rpm
checksum_type: sha256
description:
  - Supports multi-stream files.
  - Can read from or write to any file-like object.
  - Can open files in either text or binary mode.
  - Added methods: ``peek()``, ``read1()``, ``readinto()``, ``fileno()``,
    ``readable()``, ``writable()``, ``seekable()``.


pkgId: e1a94b2e1d0c3aeb61401d114e0e78404d742afc02929349bafa67a9886adecc
name: python-bzfs
arch: src
version: 1.5.0
epoch: 0
release: 1
url: https://github.com/whoschek/bzfs
rpm_license: Apache License (2.0)
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bzfs-1.5.0-1.src.rpm
checksum_type: sha256
summary:
  bzfs is a backup command line tool that reliably replicates ZFS snapshots from a (local or remote) source ZFS dataset (ZFS filesystem or ZFS volume) and its descendant datasets to a (local or remote) destination ZFS dataset to make the destination dataset a recursively synchronized copy of the source dataset, using zfs send/receive/rollback/destroy and ssh tunnel as directed. bzfs can be used to incrementally replicate all ZFS snapshots since the most recent common snapshot from source to destination.

description:
  <!--
   Copyright 2024 Wolfgang Hoschek AT mac DOT com

   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.
  -->


  [![Build](https://github.com/whoschek/bzfs/actions/workflows/python-app.yml/badge.svg)](https://github.com/whoschek/bzfs/actions/workflows/python-app.yml)
  [![Coverage](https://whoschek.github.io/bzfs/badges/coverage-badge.svg)](https://whoschek.github.io/bzfs/coverage/)
  [![os](https://whoschek.github.io/bzfs/badges/os-badge.svg)](https://github.com/whoschek/bzfs/blob/main/.github/workflows/python-app.yml)
  [![zfs](https://whoschek.github.io/bzfs/badges/zfs-badge.svg)](https://github.com/whoschek/bzfs/blob/main/.github/workflows/python-app.yml)
  [![python](https://whoschek.github.io/bzfs/badges/python-badge.svg)](https://github.com/whoschek/bzfs/blob/main/.github/workflows/python-app.yml)
  [![](https://whoschek.github.io/bzfs/badges/pypi-badge.svg)](https://pypi.org/project/bzfs)
  [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

  <!-- DO NOT EDIT (auto-generated from ArgumentParser help text as the source of "truth", via update_readme.py) -->
  <!-- BEGIN DESCRIPTION SECTION -->
  *bzfs is a backup command line tool that reliably replicates ZFS
  snapshots from a (local or remote) source ZFS dataset (ZFS filesystem or
  ZFS volume) and its descendant datasets to a (local or remote)
  destination ZFS dataset to make the destination dataset a recursively
  synchronized copy of the source dataset, using zfs
  send/receive/rollback/destroy and ssh tunnel as directed. For example,
  bzfs can be used to incrementally replicate all ZFS snapshots since the
  most recent common snapshot from source to destination, in order to help
  protect against data loss or ransomware.*

  When run for the first time, bzfs replicates the dataset and all its
  snapshots from the source to the destination. On subsequent runs, bzfs
  transfers only the data that has changed since the previous run, i.e. it
  incrementally replicates to the destination all intermediate snapshots
  that have been created on the source since the last run. Source ZFS
  snapshots older than the most recent common snapshot found on the
  destination are auto-skipped.

  bzfs does not create or delete ZFS snapshots on the source - it assumes
  you have a ZFS snapshot management tool to do so, for example
  policy-driven Sanoid, zrepl, pyznap, zfs-auto-snapshot, zfs_autobackup,
  manual zfs snapshot/destroy, etc. bzfs treats the source as read-only,
  thus the source remains unmodified. With the --dryrun flag, bzfs also
  treats the destination as read-only. In normal operation, bzfs treats
  the destination as append-only. Optional CLI flags are available to
  delete destination snapshots and destination datasets as directed, for
  example to make the destination identical to the source if the two have
  somehow diverged in unforeseen ways. This easily enables
  (re)synchronizing the backup from the production state, as well as
  restoring the production state from backup.

  In the spirit of rsync, bzfs supports a variety of powerful
  include/exclude filters that can be combined to select which datasets,
  snapshots and properties to replicate or delete or compare.

  The source 'pushes to' the destination whereas the destination 'pulls
  from' the source. bzfs is installed and executed on the 'initiator'
  host which can be either the host that contains the source dataset (push
  mode), or the destination dataset (pull mode), or both datasets (local
  mode, no network required, no ssh required), or any third-party (even
  non-ZFS OSX) host as long as that host is able to SSH (via standard
  'ssh' CLI) into both the source and destination host (pull-push mode).
  In pull-push mode the source 'zfs send's the data stream to the
  initiator which immediately pipes the stream (without storing anything
  locally) to the destination host that 'zfs receive's it. Pull-push
  mode means that bzfs need not be installed or executed on either source
  or destination host. Only the underlying 'zfs' CLI must be installed
  on both source and destination host. bzfs can run as root or non-root
  user, in the latter case via a) sudo or b) when granted corresponding
  ZFS permissions by administrators via 'zfs allow' delegation
  mechanism.

  bzfs is written in Python and continously runs a wide set of unit tests
  and integration tests to ensure coverage and compatibility with old and
  new versions of ZFS on Linux, FreeBSD and Solaris, on all Python
  versions >= 3.7 (including latest stable which is currently
  python-3.13).

  bzfs is a stand-alone program with zero required dependencies,
  consisting of a single file, akin to a stand-alone shell script or
  binary executable. It is designed to be able to run in restricted
  barebones server environments. No external Python packages are required;
  indeed no Python package management at all is required. You can just
  copy the file wherever you like, for example into /usr/local/bin or
  similar, and simply run it like any stand-alone shell script or binary
  executable.

  Optionally, bzfs applies bandwidth rate-limiting and progress monitoring
  (via 'pv' CLI) during 'zfs send/receive' data transfers. When run
  across the network, bzfs also transparently inserts lightweight data
  compression (via 'zstd -1' CLI) and efficient data buffering (via
  'mbuffer' CLI) into the pipeline between network endpoints during
  'zfs send/receive' network transfers. If one of these utilities is not
  installed this is auto-detected, and the operation continues reliably
  without the corresponding auxiliary feature.



  * Example in local mode (no network, no ssh) to replicate ZFS dataset
  tank1/foo/bar to tank2/boo/bar:


  ```
  $ bzfs tank1/foo/bar tank2/boo/bar
  ```



  ```
  $ zfs list -t snapshot tank1/foo/bar
  tank1/foo/bar@test_2024-11-06_08:30:05_daily
  tank1/foo/bar@test_2024-11-07_08:30:06_daily

  $ zfs list -t snapshot tank2/boo/bar
  tank2/boo/bar@test_2024-11-06_08:30:05_daily
  tank2/boo/bar@test_2024-11-07_08:30:06_daily
  ```


  * Same example in pull mode:

  ` bzfs root@host1.example.com:tank1/foo/bar tank2/boo/bar`

  * Same example in push mode:

  ` bzfs tank1/foo/bar root@host2.example.com:tank2/boo/bar`

  * Same example in pull-push mode:

  ` bzfs root@host1:tank1/foo/bar root@host2:tank2/boo/bar`

  * Example in local mode (no network, no ssh) to recursively replicate
  ZFS dataset tank1/foo/bar and its descendant datasets to tank2/boo/bar:


  ```
  $ bzfs tank1/foo/bar tank2/boo/bar --recursive
  ```



  ```
  $ zfs list -t snapshot -r tank1/foo/bar
  tank1/foo/bar@test_2024-11-06_08:30:05_daily
  tank1/foo/bar@test_2024-11-07_08:30:06_daily
  tank1/foo/bar/baz@test_2024-11-06_08:40:00_daily
  tank1/foo/bar/baz@test_2024-11-07_08:40:00_daily

  $ zfs list -t snapshot -r tank2/boo/bar
  tank2/boo/bar@test_2024-11-06_08:30:05_daily
  tank2/boo/bar@test_2024-11-07_08:30:06_daily
  tank2/boo/bar/baz@test_2024-11-06_08:40:00_daily
  tank2/boo/bar/baz@test_2024-11-07_08:40:00_daily
  ```


  * Example that makes destination identical to source even if the two
  have drastically diverged:

  ` bzfs tank1/foo/bar tank2/boo/bar --recursive --force
  --delete-dst-datasets --delete-dst-snapshots`

  * Replicate all daily snapshots created during the last 7 days, and at
  the same time ensure that the latest 7 daily snapshots (per dataset) are
  replicated regardless of creation time:

  ` bzfs tank1/foo/bar tank2/boo/bar --recursive
  --include-snapshot-regex '.*_daily'
  --include-snapshot-times-and-ranks '7 days ago..*' 'latest 7'`

  * Delete all daily snapshots older than 7 days, but ensure that the
  latest 7 daily snapshots (per dataset) are retained regardless of
  creation time:

  ` bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication
  --delete-dst-snapshots --include-snapshot-regex '.*_daily'
  --include-snapshot-times-and-ranks 'latest 7..latest 100%'
  --include-snapshot-times-and-ranks '*..7 days ago'`

  * Delete all daily snapshots older than 7 days, but ensure that the
  latest 7 daily snapshots (per dataset) are retained regardless of
  creation time. Additionally, only delete a snapshot if no corresponding
  snapshot or bookmark exists in the source dataset (same as above except
  replace the 'dummy' source with 'tank1/foo/bar'):

  ` bzfs tank1/foo/bar tank2/boo/bar --dryrun --recursive
  --skip-replication --delete-dst-snapshots --include-snapshot-regex
  '.*_daily' --include-snapshot-times-and-ranks 'latest 7..latest
  100%' --include-snapshot-times-and-ranks '*..7 days ago'`

  * Delete all daily snapshots older than 7 days, but ensure that the
  latest 7 daily snapshots (per dataset) are retained regardless of
  creation time. Additionally, only delete a snapshot if no corresponding
  snapshot exists in the source dataset (same as above except append
  'no-crosscheck'):

  ` bzfs tank1/foo/bar tank2/boo/bar --dryrun --recursive
  --skip-replication --delete-dst-snapshots --include-snapshot-regex
  '.*_daily' --include-snapshot-times-and-ranks 'latest 7..latest
  100%' --include-snapshot-times-and-ranks '*..7 days ago'
  --delete-dst-snapshots-no-crosscheck`

  * Delete all daily bookmarks older than 90 days, but retain the latest
  200 daily bookmarks (per dataset) regardless of creation time:

  ` bzfs dummy tank1/foo/bar --dryrun --recursive --skip-replication
  --delete-dst-snapshots=bookmarks --include-snapshot-regex
  '.*_daily' --include-snapshot-times-and-ranks 'latest 200..latest
  100%' --include-snapshot-times-and-ranks '*..90 days ago'`

  * Delete all tmp datasets within tank2/boo/bar:

  ` bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication
  --delete-dst-datasets --include-dataset-regex 'tmp.*'`

  * Compare source and destination dataset trees recursively, for example
  to check if all recently taken snapshots have been successfully
  replicated by a periodic job. List snapshots only contained in src
  (tagged with 'src'), only contained in dst (tagged with 'dst'), and
  contained in both src and dst (tagged with 'all'), restricted to
  hourly and daily snapshots taken within the last 7 days, excluding the
  last 4 hours (to allow for some slack/stragglers), excluding temporary
  datasets:

  ` bzfs tank1/foo/bar tank2/boo/bar --skip-replication
  --compare-snapshot-lists=src+dst+all --recursive
  --include-snapshot-regex '.*_(hourly|daily)'
  --include-snapshot-times-and-ranks '7 days ago..4 hours ago'
  --exclude-dataset-regex 'tmp.*'`

  If the resulting TSV output file contains zero lines starting with the
  prefix 'src' and zero lines starting with the prefix 'dst' then no
  source snapshots are missing on the destination, and no destination
  snapshots are missing on the source, indicating that the periodic
  replication and pruning jobs perform as expected. The TSV output is
  sorted by dataset, and by ZFS creation time within each dataset - the
  first and last line prefixed with 'all' contains the metadata of the
  oldest and latest common snapshot, respectively. The
  --compare-snapshot-lists option also directly logs various summary
  stats, such as the metadata of the latest common snapshot, latest
  snapshots and oldest snapshots, as well as the time diff between the
  latest common snapshot and latest snapshot only in src (and only in
  dst), as well as how many src snapshots and how many GB of data are
  missing on dst, etc.

  * Example with further options:

  ` bzfs tank1/foo/bar root@host2.example.com:tank2/boo/bar --recursive
  --exclude-snapshot-regex '.*_(hourly|frequent)'
  --exclude-snapshot-regex 'test_.*'
  --include-snapshot-times-and-ranks '7 days ago..*' 'latest 7'
  --exclude-dataset /tank1/foo/bar/temporary --exclude-dataset
  /tank1/foo/bar/baz/trash --exclude-dataset-regex '(.*/)?private'
  --exclude-dataset-regex
  '(.*/)?[Tt][Ee]?[Mm][Pp][-_]?[0-9]*'
  --ssh-dst-private-key /root/.ssh/id_rsa`

  <!-- END DESCRIPTION SECTION -->


  <!-- FINE TO EDIT -->


  ```

  sudo apt-get -y install zfsutils-linux python3 # ensure zfs and python are installed
  sudo apt-get -y install zstd pv mbuffer # auxiliary helpers are optional

  git clone https://github.com/whoschek/bzfs.git
  cd bzfs/bzfs
  ./bzfs --help # Run the CLI
  sudo cp bzfs /usr/local/bin/ # Optional system installation


  pip install bzfs
  bzfs --help # Run the CLI
  ```


  <!-- FINE TO EDIT -->


  * Rsync'ish look and feel.
  * Supports a variety of powerful include/exclude filters that can be combined to select which datasets, snapshots and
  properties to replicate or delete or compare.
  * Supports pull, push, pull-push and local transfer mode.
  * Prioritizes safe, reliable and predictable operations. Clearly separates read-only mode, append-only mode and
  delete mode.
  * Continously tested on Linux, FreeBSD and Solaris.
  * Code is almost 100% covered by tests.
  * Simple and straightforward: Can be installed in less than a minute. Can be fully scripted without configuration
  files, or scheduled via cron or similar. Does not require a daemon other than ubiquitous sshd.
  * Stays true to the ZFS send/receive spirit. Retains the ability to use ZFS CLI options for fine tuning. Does not
  attempt to "abstract away" ZFS concepts and semantics. Keeps simple things simple, and makes complex things possible.
  * All ZFS and SSH commands (even in --dryrun mode) are logged such that they can be inspected, copy-and-pasted into
  a terminal/shell, and run manually to help anticipate or diagnose issues.
  * Supports replicating (or deleting) dataset subsets via powerful include/exclude regexes and other filters, which can
  be combined into a mini filter pipeline. For example, can replicate (or delete) all except temporary datasets and
  private datasets. Can be told to do such deletions only if a corresponding source dataset does not exist.
  * Supports replicating (or deleting) snapshot subsets via powerful include/exclude regexes, time based filters, and
  oldest N/latest N filters, which can also be combined into a mini filter pipeline.
      * For example, can replicate (or delete) daily and weekly snapshots while ignoring hourly and 5 minute snapshots.
  Or, can replicate daily and weekly snapshots to a remote destination while replicating hourly and 5 minute snapshots
  to a local destination.
      * For example, can replicate (or delete) all daily snapshots older (or newer) than 90 days, and all weekly snapshots
  older (or newer) than 12 weeks.
      * For example, can replicate (or delete) all daily snapshots except the latest (or oldest) 90 daily snapshots,
  and all weekly snapshots except the latest (or oldest) 12 weekly snapshots.
      * For example, can replicate all daily snapshots that were created during the last 7 days, and at the
  same time ensure that at least the latest 7 daily snapshots are replicated regardless of creation time.
  This helps to safely cope with irregular scenarios where no snapshots were created or received within the last 7 days,
  or where more than 7 daily snapshots were created or received within the last 7 days.
      * For example, can delete all daily snapshots older than 7 days, but retain the latest 7 daily
  snapshots regardless of creation time. It can help to avoid accidental pruning of the last snapshot that source and
  destination have in common.
      * Can be told to do such deletions only if a corresponding snapshot does not exist in the source dataset.
  * Compare source and destination dataset trees recursively, in combination with snapshot filters and dataset filters.
  * Also supports replicating arbitrary dataset tree subsets by feeding it a list of flat datasets.
  * Efficiently supports complex replication policies with multiple sources and multiple destinations for each source.
  * Can be told what ZFS dataset properties to copy, also via include/exclude regexes.
  * Full and precise ZFS bookmark support for additional safety, or to reclaim disk space earlier.
  * Can be strict or told to be tolerant of runtime errors.
  * Automatically resumes ZFS send/receive operations that have been interrupted by network hiccups or other
  intermittent failures, via efficient 'zfs receive -s' and 'zfs send -t'.
  * Similarly, can be told to automatically retry snapshot delete operations.
  * Parametrizable retry logic.
  * Multiple bzfs processes can run in parallel. If multiple processes attempt to write to the same destination dataset
  simultaneously this is detected and the operation can be auto-retried safely.
  * A job that runs periodically declines to start if the same previous periodic job is still running without
  completion yet.
  * Can log to local and remote destinations out of the box. Logging mechanism is customizable and plugable for smooth
  integration.
  * Codebase is easy to change, hack and maintain. No hidden magic. Python is very readable to contemporary engineers.
  Chances are that CI tests will catch changes that have unintended side effects.
  * It's fast!




  Results of automated test runs on a matrix of various old and new versions of ZFS/Python/Linux/FreeBSD/Solaris are
  [here](https://github.com/whoschek/bzfs/actions/workflows/python-app.yml?query=event%3Aschedule), as generated
  by [this script](https://github.com/whoschek/bzfs/blob/main/.github/workflows/python-app.yml).
  The script also demonstrates functioning installation steps on Ubuntu, FreeBSD, Solaris, etc.
  The script also generates code coverage reports which are published
  [here](https://whoschek.github.io/bzfs/coverage).

  The gist is that it should work on any flavor, with python (3.7 or higher, no additional python packages required)
  only needed on the initiator host.




  * First, on the GitHub page of this repo, click on "Fork/Create a new fork".
  * Click the 'Actions' menu on your repo, and then enable GitHub Actions on your fork.
  * Then select 'All workflows' -> 'Tests' on the left side.
  * Then click the 'Run workflow' dropdown menu on the right side.
  * Select the name of the job to run (e.g. 'test_ubuntu_24_04' or 'test_freebsd_14_1' or 'test_solaris_11_4', etc) or
  select 'Run all jobs' to test all supported platforms, plus select the git branch to run with (typically the branch
  containing your changes).
  * Then click the 'Run workflow' button which kicks off the job.
  * Click on the job to watch job progress.
  * Once the run completes, you can click on the wheel on the top right to select 'Download log archive', which is
  a zip file containing the output of all jobs of the run. This is useful for debugging.
  * Once the job completes, also a coverage report appears on the bottom of the run page, which you can download by
  clicking on it. Unzip and browse the HTML coverage report to see if the code you added or changed is actually executed
  by a test. Experience shows that code that isn't executed likely contains bugs, so all changes (code lines and
  branch conditions) should be covered by a test before a merge is considered. In practice, this means to watch out
  for coverage report lines that aren't colored green.
  * FYI, example test runs with coverage reports are
  [here](https://github.com/whoschek/bzfs/actions/workflows/python-app.yml?query=event%3Aschedule).
  Click on any run and browse to the bottom of the resulting run page to find the coverage reports, including a combined
  coverage report that merges all jobs of the run.



  ```

  zfs --version


  python3 --version


  sudo ls






  ssh -p $bzfs_test_ssh_port 127.0.0.1 echo hello


  ssh -p $bzfs_test_ssh_port 127.0.0.1 zfs --version


  ssh -p $bzfs_test_ssh_port 127.0.0.1 zpool --version


  ssh -p $bzfs_test_ssh_port 127.0.0.1 zstd --version


  ssh -p $bzfs_test_ssh_port 127.0.0.1 pv --version


  ssh -p $bzfs_test_ssh_port 127.0.0.1 mbuffer --version


  ./test.sh


  bzfs-test
  ```




  <!-- DO NOT EDIT (auto-generated from ArgumentParser help text as the source of "truth", via update_readme.py) -->
  <!-- BEGIN HELP OVERVIEW SECTION -->
  ```
  usage: bzfs [-h] [--recursive]
              [--include-dataset DATASET [DATASET ...]]
              [--exclude-dataset DATASET [DATASET ...]]
              [--include-dataset-regex REGEX [REGEX ...]]
              [--exclude-dataset-regex REGEX [REGEX ...]]
              [--exclude-dataset-property STRING]
              [--include-snapshot-regex REGEX [REGEX ...]]
              [--exclude-snapshot-regex REGEX [REGEX ...]]
              [--include-snapshot-times-and-ranks TIMERANGE [RANKRANGE ...]]
              [--zfs-send-program-opts STRING]
              [--zfs-recv-program-opts STRING]
              [--zfs-recv-program-opt STRING]
              [--force-rollback-to-latest-snapshot]
              [--force-rollback-to-latest-common-snapshot] [--force]
              [--force-unmount] [--force-once] [--skip-parent]
              [--skip-missing-snapshots [{fail,dataset,continue}]]
              [--retries INT] [--retry-min-sleep-secs FLOAT]
              [--retry-max-sleep-secs FLOAT]
              [--retry-max-elapsed-secs FLOAT]
              [--skip-on-error {fail,tree,dataset}] [--skip-replication]
              [--delete-dst-datasets]
              [--delete-dst-snapshots [{snapshots,bookmarks}]]
              [--delete-dst-snapshots-no-crosscheck]
              [--delete-empty-dst-datasets [{snapshots,snapshots+bookmarks}]]
              [--compare-snapshot-lists [{src,dst,all,src+dst,src+all,dst+all,src+dst+all}]]
              [--dryrun [{recv,send}]] [--verbose] [--quiet]
              [--no-privilege-elevation] [--no-stream]
              [--no-resume-recv] [--no-create-bookmark]
              [--no-use-bookmark] [--ssh-cipher STRING]
              [--ssh-src-private-key FILE] [--ssh-dst-private-key FILE]
              [--ssh-src-user STRING] [--ssh-dst-user STRING]
              [--ssh-src-host STRING] [--ssh-dst-host STRING]
              [--ssh-src-port INT] [--ssh-dst-port INT]
              [--ssh-src-extra-opts STRING] [--ssh-src-extra-opt STRING]
              [--ssh-dst-extra-opts STRING] [--ssh-dst-extra-opt STRING]
              [--ssh-src-config-file FILE] [--ssh-dst-config-file FILE]
              [--bwlimit STRING] [--compression-program STRING]
              [--compression-program-opts STRING]
              [--mbuffer-program STRING] [--mbuffer-program-opts STRING]
              [--ps-program STRING] [--pv-program STRING]
              [--pv-program-opts STRING] [--shell-program STRING]
              [--ssh-program STRING] [--sudo-program STRING]
              [--zfs-program STRING] [--zpool-program STRING]
              [--log-dir DIR] [--log-file-prefix STRING]
              [--log-file-suffix STRING] [--log-syslog-address STRING]
              [--log-syslog-socktype {UDP,TCP}]
              [--log-syslog-facility INT] [--log-syslog-prefix STRING]
              [--log-syslog-level {CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE}]
              [--log-config-file STRING]
              [--log-config-var NAME:VALUE [NAME:VALUE ...]]
              [--include-envvar-regex REGEX [REGEX ...]]
              [--exclude-envvar-regex REGEX [REGEX ...]]
              [--zfs-recv-o-targets {full,incremental,full+incremental}]
              [--zfs-recv-o-sources STRING]
              [--zfs-recv-o-include-regex REGEX [REGEX ...]]
              [--zfs-recv-o-exclude-regex REGEX [REGEX ...]]
              [--zfs-recv-x-targets {full,incremental,full+incremental}]
              [--zfs-recv-x-sources STRING]
              [--zfs-recv-x-include-regex REGEX [REGEX ...]]
              [--zfs-recv-x-exclude-regex REGEX [REGEX ...]] [--version]
              [--help, -h]
              SRC_DATASET DST_DATASET [SRC_DATASET DST_DATASET ...]
  ```
  <!-- END HELP OVERVIEW SECTION -->

  <!-- BEGIN HELP DETAIL SECTION -->
  <div id="SRC_DATASET_DST_DATASET"></div>

  **SRC_DATASET DST_DATASET**

  *  SRC_DATASET: Source ZFS dataset (and its descendants) that will be
      replicated. Can be a ZFS filesystem or ZFS volume. Format is
      [[user@]host:]dataset. The host name can also be an IPv4 address
      (or an IPv6 address where each ':' colon character must be
      replaced with a '|' pipe character for disambiguation). If the
      host name is '-', the dataset will be on the local host, and the
      corresponding SSH leg will be omitted. The same is true if the host
      is omitted and the dataset does not contain a ':' colon at the
      same time. Local dataset examples: `tank1/foo/bar`, `tank1`,
      `-:tank1/foo/bar:baz:boo` Remote dataset examples:
      `host:tank1/foo/bar`, `host.example.com:tank1/foo/bar`,
      `root@host:tank`, `root@host.example.com:tank1/foo/bar`,
      `user@127.0.0.1:tank1/foo/bar:baz:boo`,
      `user@||1:tank1/foo/bar:baz:boo`. The first component of the ZFS
      dataset name is the ZFS pool name, here `tank1`. If the option
      starts with a `+` prefix then dataset names are read from the
      UTF-8 text file given after the `+` prefix, with each line in the
      file containing a SRC_DATASET and a DST_DATASET, separated by a tab
      character. Example: `+root_dataset_names.txt`,
      `+/path/to/root_dataset_names.txt`

      DST_DATASET: Destination ZFS dataset for replication and deletion.
      Has same naming format as SRC_DATASET. During replication,
      destination datasets that do not yet exist are created as necessary,
      along with their parent and ancestors.



  <div id="--recursive"></div>

  **--recursive**, **-r**

  *  During replication and deletion, also consider descendant datasets,
      i.e. datasets within the dataset tree, including children, and
      children of children, etc.

  <!-- -->

  <div id="--include-dataset"></div>

  **--include-dataset** *DATASET [DATASET ...]*

  *  During replication and deletion, select any ZFS dataset (and its
      descendants) that is contained within SRC_DATASET (DST_DATASET in
      case of deletion) if its dataset name is one of the given include
      dataset names but none of the exclude dataset names. If a dataset is
      excluded its descendants are automatically excluded too, and this
      decision is never reconsidered even for the descendants because
      exclude takes precedence over include.

      A dataset name is absolute if the specified dataset is prefixed by
      `/`, e.g. `/tank/baz/tmp`. Otherwise the dataset name is
      relative wrt. source and destination, e.g. `baz/tmp` if the source
      is `tank`.

      This option is automatically translated to an
      --include-dataset-regex (see below) and can be specified multiple
      times.

      If the option starts with a `+` prefix then dataset names are read
      from the newline-separated UTF-8 text file given after the `+`
      prefix, one dataset per line inside of the text file. Examples:
      `/tank/baz/tmp` (absolute), `baz/tmp` (relative),
      `+dataset_names.txt`, `+/path/to/dataset_names.txt`

  <!-- -->

  <div id="--exclude-dataset"></div>

  **--exclude-dataset** *DATASET [DATASET ...]*

  *  Same syntax as --include-dataset (see above) except that the option
      is automatically translated to an --exclude-dataset-regex (see
      below).

  <!-- -->

  <div id="--include-dataset-regex"></div>

  **--include-dataset-regex** *REGEX [REGEX ...]*

  *  During replication (and deletion), select any ZFS dataset (and its
      descendants) that is contained within SRC_DATASET (DST_DATASET in
      case of deletion) if its relative dataset path (e.g. `baz/tmp`)
      wrt. SRC_DATASET (DST_DATASET in case of deletion) matches at least
      one of the given include regular expressions but none of the exclude
      regular expressions. If a dataset is excluded its descendants are
      automatically excluded too, and this decision is never reconsidered
      even for the descendants because exclude takes precedence over
      include.

      This option can be specified multiple times. A leading `!`
      character indicates logical negation, i.e. the regex matches if the
      regex with the leading `!` character removed does not match.

      Default: `.*` (include all datasets). Examples: `baz/tmp`,
      `(.*/)?doc[^/]*/(private|confidential).*`, `!public`

  <!-- -->

  <div id="--exclude-dataset-regex"></div>

  **--exclude-dataset-regex** *REGEX [REGEX ...]*

  *  Same syntax as --include-dataset-regex (see above) except that the
      default is `(.*/)?[Tt][Ee]?[Mm][Pp][-_]?[0-9]*`
      (exclude tmp datasets). Example: `!.*` (exclude no dataset)

  <!-- -->

  <div id="--exclude-dataset-property"></div>

  **--exclude-dataset-property** *STRING*

  *  The name of a ZFS dataset user property (optional). If this option
      is specified, the effective value (potentially inherited) of that
      user property is read via 'zfs list' for each selected source
      dataset to determine whether the dataset will be included or
      excluded, as follows:

      a) Value is 'true' or '-' or empty string or the property is
      missing: Include the dataset.

      b) Value is 'false': Exclude the dataset and its descendants.

      c) Value is a comma-separated list of fully qualified host names
      (no spaces, for example:
      'store001.example.com,store002.example.com'): Include the dataset
      if the fully qualified host name of the host executing bzfs is
      contained in the list, otherwise exclude the dataset and its
      descendants.

      If a dataset is excluded its descendants are automatically excluded
      too, and the property values of the descendants are ignored because
      exclude takes precedence over include.

      Examples: 'syncoid:sync', 'com.example.eng.project.x:backup'

      *Note:* The use of --exclude-dataset-property is discouraged for
      most use cases. It is more flexible, more powerful, *and* more
      efficient to instead use a combination of
      --include/exclude-dataset-regex and/or --include/exclude-dataset
      to achieve the same or better outcome.

  <!-- -->

  <div id="--include-snapshot-regex"></div>

  **--include-snapshot-regex** *REGEX [REGEX ...]*

  *  During replication and deletion, select any source ZFS snapshot that
      has a name (i.e. the part after the '@') that matches at least one
      of the given include regular expressions but none of the exclude
      regular expressions. If a snapshot is excluded this decision is
      never reconsidered because exclude takes precedence over include.

      This option can be specified multiple times. A leading `!`
      character indicates logical negation, i.e. the regex matches if the
      regex with the leading `!` character removed does not match.

      Default: `.*` (include all snapshots). Examples: `test_.*`,
      `!prod_.*`, `.*_(hourly|frequent)`,
      `!.*_(weekly|daily)`

      *Note:* All --include/exclude-snapshot-* CLI option groups are
      combined into a mini filter pipeline. A filter pipeline is executed
      in the order given on the command line, left to right. For example
      if --include-snapshot-ranks (see below) is specified on the command
      line before --include/exclude-snapshot-regex, then
      --include-snapshot-ranks will be applied before
      --include/exclude-snapshot-regex. The pipeline results would not
      always be the same if the order were reversed. Order matters.

      *Note:* During replication, bookmarks are always retained aka
      selected in order to help find common snapshots between source and
      destination.

  <!-- -->

  <div id="--exclude-snapshot-regex"></div>

  **--exclude-snapshot-regex** *REGEX [REGEX ...]*

  *  Same syntax as --include-snapshot-regex (see above) except that the
      default is to exclude no snapshots.

  <!-- -->

  <div id="--include-snapshot-times-and-ranks"></div>

  **--include-snapshot-times-and-ranks** *TIMERANGE [RANKRANGE ...]*

  *  This option takes as input parameters a time range filter and an
      optional rank range filter. It separately computes the results for
      each filter and selects the UNION of both results. To instead use a
      pure rank range filter (no UNION), or a pure time range filter (no
      UNION), simply use '0..0' to indicate an empty time range, or omit
      the rank range, respectively. This option can be specified multiple
      times.

      <b>*Replication Example (UNION):* </b>

      Specify to replicate all daily snapshots created during the last 7
      days, and at the same time ensure that the latest 7 daily snapshots
      (per dataset) are replicated regardless of creation time, like so:
      `--include-snapshot-regex '.*_daily'
      --include-snapshot-times-and-ranks '7 days ago..*' 'latest
      7'`

      <b>*Deletion Example (no UNION):* </b>

      Specify to delete all daily snapshots older than 7 days, but ensure
      that the latest 7 daily snapshots (per dataset) are retained
      regardless of creation time, like so: `--include-snapshot-regex
      '.*_daily' --include-snapshot-times-and-ranks 'latest
      7..latest 100%' --include-snapshot-times-and-ranks '*..7 days
      ago'`

      This helps to safely cope with irregular scenarios where no
      snapshots were created or received within the last 7 days, or where
      more than 7 daily snapshots were created within the last 7 days. It
      can also help to avoid accidental pruning of the last snapshot that
      source and destination have in common.

      <b>*TIMERANGE:* </b>

      The ZFS 'creation' time of a snapshot (and bookmark) must fall
      into this time range in order for the snapshot to be included. The
      time range consists of a 'start' time, followed by a '..'
      separator, followed by an 'end' time. For example
      '2024-01-01..2024-04-01' or `*..*` aka all times. Only
      snapshots (and bookmarks) in the half-open time range [start, end)
      are included; other snapshots (and bookmarks) are excluded. If a
      snapshot is excluded this decision is never reconsidered because
      exclude takes precedence over include. Each of the two specified
      times can take any of the following forms:

      * a) a `*` wildcard character representing negative or positive
      infinity.

      * b) a non-negative integer representing a UTC Unix time in
      seconds. Example: 1728109805

      * c) an ISO 8601 datetime string with or without timezone.
      Examples: '2024-10-05', '2024-10-05T14:48:55',
      '2024-10-05T14:48:55+02', '2024-10-05T14:48:55-04:30'. If the
      datetime string does not contain time zone info then it is assumed
      to be in the local time zone. Timezone string support requires
      Python >= 3.11.

      * d) a duration that indicates how long ago from the current time,
      using the following syntax: a non-negative integer, followed by an
      optional space, followed by a duration unit that is *one* of
      'seconds', 'secs', 'minutes', 'mins', 'hours', 'days',
      'weeks', followed by an optional space, followed by the word
      'ago'. Examples: '0secs ago', '90 mins ago', '48hours ago',
      '90days ago', '12weeksago'.

      * Note: This option compares the specified time against the
      standard ZFS 'creation' time property of the snapshot (which is a
      UTC Unix time in integer seconds), rather than against a timestamp
      that may be part of the snapshot name. You can list the ZFS creation
      time of snapshots and bookmarks as follows: `zfs list -t
      snapshot,bookmark -o name,creation -s creation -d 1 $SRC_DATASET`
      (optionally add the -p flag to display UTC Unix time in integer
      seconds).

      *Note:* During replication, bookmarks are always retained aka
      selected in order to help find common snapshots between source and
      destination.

      <b>*RANKRANGE:* </b>

      Specifies to include the N (or N%) oldest snapshots or latest
      snapshots, and exclude all other snapshots (default: include no
      snapshots). Snapshots are sorted by creation time (actually, by the
      'createtxg' ZFS property, which serves the same purpose but is
      more precise). The rank position of a snapshot is the zero-based
      integer position of the snapshot within that sorted list. A rank
      consist of the word 'oldest' or 'latest', followed by a
      non-negative integer, followed by an optional '%' percent sign. A
      rank range consists of a lower rank, followed by a '..' separator,
      followed by a higher rank. If the optional lower rank is missing it
      is assumed to be 0. Examples:

      * 'oldest 10%' aka 'oldest 0..oldest 10%' (include the oldest
      10% of all snapshots)

      * 'latest 10%' aka 'latest 0..latest 10%' (include the latest
      10% of all snapshots)

      * 'oldest 90' aka 'oldest 0..oldest 90' (include the oldest 90
      snapshots)

      * 'latest 90' aka 'latest 0..latest 90' (include the latest 90
      snapshots)

      * 'oldest 90..oldest 100%' (include all snapshots except the
      oldest 90 snapshots)

      * 'latest 90..latest 100%' (include all snapshots except the
      latest 90 snapshots)

      * 'latest 1' aka 'latest 0..latest 1' (include the latest
      snapshot)

      * 'latest 1..latest 100%' (include all snapshots except the
      latest snapshot)

      * 'oldest 2' aka 'oldest 0..oldest 2' (include the oldest 2
      snapshots)

      * 'oldest 2..oldest 100%' (include all snapshots except the
      oldest 2 snapshots)

      * 'oldest 100%' aka 'oldest 0..oldest 100%' (include all
      snapshots)

      * 'oldest 0%' aka 'oldest 0..oldest 0%' (include no snapshots)

      * 'oldest 0' aka 'oldest 0..oldest 0' (include no snapshots)

      *Note:* Percentage calculations are not based on the number of
      snapshots contained in the dataset on disk, but rather based on the
      number of snapshots arriving at the filter. For example, if only two
      daily snapshots arrive at the filter because a prior filter excludes
      hourly snapshots, then 'latest 10' will only include these two
      daily snapshots, and 'latest 50%' will only include one of these
      two daily snapshots.

      *Note:* During replication, bookmarks are always retained aka
      selected in order to help find common snapshots between source and
      destination. Bookmarks do not count towards N or N% wrt. rank.

      *Note:* If a snapshot is excluded this decision is never
      reconsidered because exclude takes precedence over include.

  <!-- -->

  <div id="--zfs-send-program-opts"></div>

  **--zfs-send-program-opts** *STRING*

  *  Parameters to fine-tune 'zfs send' behaviour (optional); will be
      passed into 'zfs send' CLI. The value is split on runs of one or
      more whitespace characters. Default is '--props --raw
      --compressed'. See
      https://openzfs.github.io/openzfs-docs/man/master/8/zfs-send.8.html
      and https://github.com/openzfs/zfs/issues/13024

  <!-- -->

  <div id="--zfs-recv-program-opts"></div>

  **--zfs-recv-program-opts** *STRING*

  *  Parameters to fine-tune 'zfs receive' behaviour (optional); will
      be passed into 'zfs receive' CLI. The value is split on runs of
      one or more whitespace characters. Default is '-u'. Example: '-u
      -o canmount=noauto -o readonly=on -x keylocation -x keyformat -x
      encryption'. See
      https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html
      and
      https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html

  <!-- -->

  <div id="--zfs-recv-program-opt"></div>

  **--zfs-recv-program-opt** *STRING*

  *  Parameter to fine-tune 'zfs receive' behaviour (optional); will be
      passed into 'zfs receive' CLI. The value can contain spaces and is
      not split. This option can be specified multiple times. Example:
      `--zfs-recv-program-opt=-o
      --zfs-recv-program-opt='org.zfsbootmenu:commandline=ro debug
      zswap.enabled=1'`

  <!-- -->

  <div id="--force-rollback-to-latest-snapshot"></div>

  **--force-rollback-to-latest-snapshot**

  *  Before replication, rollback the destination dataset to its most
      recent destination snapshot (if there is one), via 'zfs rollback',
      just in case the destination dataset was modified since its most
      recent snapshot. This is much less invasive than the other
      --force* options (see below).

  <!-- -->

  <div id="--force-rollback-to-latest-common-snapshot"></div>

  **--force-rollback-to-latest-common-snapshot**

  *  Before replication, delete destination ZFS snapshots that are more
      recent than the most recent common snapshot selected on the source
      ('conflicting snapshots'), via 'zfs rollback'. Abort with an
      error if no common snapshot is selected but the destination already
      contains a snapshot.

  <!-- -->

  <div id="--force"></div>

  **--force**

  *  Same as --force-rollback-to-latest-common-snapshot (see above),
      except that additionally, if no common snapshot is selected, then
      delete all destination snapshots before starting replication, and
      proceed without aborting. Without the --force* flags, the
      destination dataset is treated as append-only, hence no destination
      snapshot that already exists is deleted, and instead the operation
      is aborted with an error when encountering a conflicting snapshot.

      Analogy: --force-rollback-to-latest-snapshot is a tiny hammer,
      whereas --force-rollback-to-latest-common-snapshot is a medium
      sized hammer, and --force is a large hammer. Consider using the
      smallest hammer that can fix the problem. No hammer is ever used by
      default.

  <!-- -->

  <div id="--force-unmount"></div>

  **--force-unmount**

  *  On destination, --force and
      --force-rollback-to-latest-common-snapshot will add the '-f' flag
      to their use of 'zfs rollback' and 'zfs destroy'.

  <!-- -->

  <div id="--force-once"></div>

  **--force-once**, **--f1**

  *  Use the --force option or
      --force-rollback-to-latest-common-snapshot option at most once to
      resolve a conflict, then abort with an error on any subsequent
      conflict. This helps to interactively resolve conflicts, one
      conflict at a time.

  <!-- -->

  <div id="--skip-parent"></div>

  **--skip-parent**

  *  During replication and deletion, skip processing of the SRC_DATASET
      and DST_DATASET and only process their descendant datasets, i.e.
      children, and children of children, etc (with --recursive). No
      dataset is processed unless --recursive is also specified. Analogy:
      `bzfs --recursive --skip-parent src dst` is akin to Unix `cp -r
      src/* dst/` whereas `bzfs --recursive --skip-parent
      --skip-replication --delete-dst-datasets dummy dst` is akin to
      Unix `rm -r dst/*`

  <!-- -->

  <div id="--skip-missing-snapshots"></div>

  **--skip-missing-snapshots** *[{fail,dataset,continue}]*

  *  During replication, handle source datasets that select no snapshots
      (and no relevant bookmarks) as follows:

      a) 'fail': Abort with an error.

      b) 'dataset' (default): Skip the source dataset with a warning.
      Skip descendant datasets if --recursive and destination dataset
      does not exist. Otherwise skip to the next dataset.

      c) 'continue': Skip nothing. If destination snapshots exist,
      delete them (with --force) or abort with an error (without
      --force). If there is no such abort, continue processing with the
      next dataset. Eventually create empty destination dataset and
      ancestors if they do not yet exist and source dataset has at least
      one descendant that selects at least one snapshot.

  <!-- -->

  <div id="--retries"></div>

  **--retries** *INT*

  *  The maximum number of times a retryable replication or deletion step
      shall be retried if it fails, for example because of network hiccups
      (default: 2, min: 0). Also consider this option if a periodic
      pruning script may simultaneously delete a dataset or snapshot or
      bookmark while bzfs is running and attempting to access it.

  <!-- -->

  <div id="--retry-min-sleep-secs"></div>

  **--retry-min-sleep-secs** *FLOAT*

  *  The minimum duration to sleep between retries (default: 0.125).

  <!-- -->

  <div id="--retry-max-sleep-secs"></div>

  **--retry-max-sleep-secs** *FLOAT*

  *  The maximum duration to sleep between retries initially starts with
      --retry-min-sleep-secs (see above), and doubles on each retry, up
      to the final maximum of --retry-max-sleep-secs (default: 300). On
      each retry a random sleep time in the [--retry-min-sleep-secs,
      current max] range is picked. The timer resets after each
      operation.

  <!-- -->

  <div id="--retry-max-elapsed-secs"></div>

  **--retry-max-elapsed-secs** *FLOAT*

  *  A single operation (e.g. 'zfs send/receive' of the current
      dataset, or deletion of a list of snapshots within the current
      dataset) will not be retried (or not retried anymore) once this much
      time has elapsed since the initial start of the operation, including
      retries (default: 3600). The timer resets after each operation
      completes or retries exhaust, such that subsequently failing
      operations can again be retried.

  <!-- -->

  <div id="--skip-on-error"></div>

  **--skip-on-error** *{fail,tree,dataset}*

  *  During replication and deletion, if an error is not retryable, or
      --retries has been exhausted, or --skip-missing-snapshots raises
      an error, proceed as follows:

      a) 'fail': Abort the program with an error. This mode is ideal
      for testing, clear error reporting, and situations where consistency
      trumps availability.

      b) 'tree': Log the error, skip the dataset tree rooted at the
      dataset for which the error occurred, and continue processing the
      next (sibling) dataset tree. Example: Assume datasets tank/user1/foo
      and tank/user2/bar and an error occurs while processing tank/user1.
      In this case processing skips tank/user1/foo and proceeds with
      tank/user2.

      c) 'dataset' (default): Same as 'tree' except if the
      destination dataset already exists, skip to the next dataset
      instead. Example: Assume datasets tank/user1/foo and tank/user2/bar
      and an error occurs while processing tank/user1. In this case
      processing skips tank/user1 and proceeds with tank/user1/foo if the
      destination already contains tank/user1. Otherwise processing
      continues with tank/user2. This mode is for production use cases
      that require timely forward progress even in the presence of partial
      failures. For example, assume the job is to backup the home
      directories or virtual machines of thousands of users across an
      organization. Even if replication of some of the datasets for some
      users fails due too conflicts, busy datasets, etc, the replication
      job will continue for the remaining datasets and the remaining
      users.

  <!-- -->

  <div id="--skip-replication"></div>

  **--skip-replication**

  *  Skip replication step (see above) and proceed to the optional
      --delete-dst-datasets step immediately (see below).

  <!-- -->

  <div id="--delete-dst-datasets"></div>

  **--delete-dst-datasets**

  *  Do nothing if the --delete-dst-datasets option is missing.
      Otherwise, after successful replication step, if any, delete
      existing destination datasets that are selected via
      --{include|exclude}-dataset* policy yet do not exist within
      SRC_DATASET (which can be an empty dataset, such as the hardcoded
      virtual dataset named 'dummy'!). Does not recurse without
      --recursive.

      For example, if the destination contains datasets h1,h2,h3,d1
      whereas source only contains h3, and the include/exclude policy
      selects h1,h2,h3,d1, then delete datasets h1,h2,d1 on the
      destination to make it 'the same'. On the other hand, if the
      include/exclude policy only selects h1,h2,h3 then only delete
      datasets h1,h2 on the destination to make it 'the same'.

      Example to delete all tmp datasets within tank2/boo/bar: `bzfs
      dummy tank2/boo/bar --dryrun --skip-replication
      --delete-dst-datasets --include-dataset-regex 'tmp.*'
      --recursive`

  <!-- -->

  <div id="--delete-dst-snapshots"></div>

  **--delete-dst-snapshots** *[{snapshots,bookmarks}]*

  *  Do nothing if the --delete-dst-snapshots option is missing.
      Otherwise, after successful replication, and successful
      --delete-dst-datasets step, if any, delete existing destination
      snapshots whose GUID does not exist within the source dataset (which
      can be an empty dummy dataset!) if the destination snapshots are
      selected by the --include/exclude-snapshot-* policy, and the
      destination dataset is selected via --{include|exclude}-dataset*
      policy. Does not recurse without --recursive.

      For example, if the destination dataset contains snapshots
      h1,h2,h3,d1 (h=hourly, d=daily) whereas the source dataset only
      contains snapshot h3, and the include/exclude policy selects
      h1,h2,h3,d1, then delete snapshots h1,h2,d1 on the destination
      dataset to make it 'the same'. On the other hand, if the
      include/exclude policy only selects snapshots h1,h2,h3 then only
      delete snapshots h1,h2 on the destination dataset to make it 'the
      same'.

      *Note:* To delete snapshots regardless, consider using
      --delete-dst-snapshots in combination with a source that is an
      empty dataset, such as the hardcoded virtual dataset named
      'dummy', like so: `bzfs dummy tank2/boo/bar --dryrun
      --skip-replication --delete-dst-snapshots
      --include-snapshot-regex '.*_daily' --recursive`

      *Note:* Use --delete-dst-snapshots=bookmarks to delete bookmarks
      instead of snapshots, in which case no snapshots are selected and
      the --{include|exclude}-snapshot-* filter options treat bookmarks
      as snapshots wrt. selecting.

  <!-- -->

  <div id="--delete-dst-snapshots-no-crosscheck"></div>

  **--delete-dst-snapshots-no-crosscheck**

  *  This flag indicates that --delete-dst-snapshots=snapshots shall
      check the source dataset only for a snapshot with the same GUID, and
      ignore whether a bookmark with the same GUID is present in the
      source dataset. Similarly, it also indicates that
      --delete-dst-snapshots=bookmarks shall check the source dataset
      only for a bookmark with the same GUID, and ignore whether a
      snapshot with the same GUID is present in the source dataset.

  <!-- -->

  <div id="--delete-empty-dst-datasets"></div>

  **--delete-empty-dst-datasets** *[{snapshots,snapshots+bookmarks}]*

  *  Do nothing if the --delete-empty-dst-datasets option is missing or
      --recursive is missing. Otherwise, after successful replication
      step and successful --delete-dst-datasets and successful
      --delete-dst-snapshots steps, if any, delete any selected
      destination dataset that has no snapshot and no bookmark if all
      descendants of that destination dataset are also selected and do not
      have a snapshot or bookmark either (again, only if the existing
      destination dataset is selected via --{include|exclude}-dataset*
      policy). Never delete excluded dataset subtrees or their ancestors.

      For example, if the destination contains datasets h1,d1, and the
      include/exclude policy selects h1,d1, then check if h1,d1 can be
      deleted. On the other hand, if the include/exclude policy only
      selects h1 then only check if h1 can be deleted.

      *Note:* Use --delete-empty-dst-datasets=snapshots to delete
      snapshot-less datasets even if they still contain bookmarks.

  <!-- -->

  <div id="--compare-snapshot-lists"></div>

  **--compare-snapshot-lists** *[{src,dst,all,src+dst,src+all,dst+all,src+dst+all}]*

  *  Do nothing if the --compare-snapshot-lists option is missing.
      Otherwise, after successful replication step and successful
      --delete-dst-datasets, --delete-dst-snapshots steps and
      --delete-empty-dst-datasets steps, if any, proceed as follows:

      Compare source and destination dataset trees recursively wrt.
      snapshots, for example to check if all recently taken snapshots have
      been successfully replicated by a periodic job.

      Example: List snapshots only contained in source (tagged with
      'src'), only contained in destination (tagged with 'dst'), and
      contained in both source and destination (tagged with 'all'),
      restricted to hourly and daily snapshots taken within the last 7
      days, excluding the last 4 hours (to allow for some
      slack/stragglers), excluding temporary datasets: `bzfs
      tank1/foo/bar tank2/boo/bar --skip-replication
      --compare-snapshot-lists=src+dst+all --recursive
      --include-snapshot-regex '.*_(hourly|daily)'
      --include-snapshot-times-and-ranks '7 days ago..4 hours ago'
      --exclude-dataset-regex 'tmp.*'`

      This outputs a TSV file containing the following columns:

      `location creation_iso createtxg rel_name guid root_dataset
      rel_dataset name creation written`

      Example output row:

      `src 2024-11-06_08:30:05 17435050
      /foo@test_2024-11-06_08:30:05_daily 2406491805272097867 tank1/src
      /foo tank1/src/foo@test_2024-10-06_08:30:04_daily 1730878205 24576`

      If the TSV output file contains zero lines starting with the prefix
      'src' and zero lines starting with the prefix 'dst' then no
      source snapshots are missing on the destination, and no destination
      snapshots are missing on the source, indicating that the periodic
      replication and pruning jobs perform as expected. The TSV output is
      sorted by rel_dataset, and by ZFS creation time within each
      rel_dataset - the first and last line prefixed with 'all' contains
      the metadata of the oldest and latest common snapshot, respectively.
      Third party tools can use this info for post-processing, for example
      using custom scripts using 'csplit' or duckdb analytics queries.

      The --compare-snapshot-lists option also directly logs various
      summary stats, such as the metadata of the latest common snapshot,
      latest snapshots and oldest snapshots, as well as the time diff
      between the latest common snapshot and latest snapshot only in src
      (and only in dst), as well as how many src snapshots and how many GB
      of data are missing on dst, etc.

      *Note*: Consider omitting the 'all' flag to reduce noise and
      instead focus on missing snapshots only, like so:
      --compare-snapshot-lists=src+dst

      *Note*: The source can also be an empty dataset, such as the
      hardcoded virtual dataset named 'dummy'.

      *Note*: --compare-snapshot-lists is typically *much* faster
      than standard 'zfs list -t snapshot' CLI usage because the former
      issues I/O requests with a higher degree of parallelism than the
      latter.

  <!-- -->

  <div id="--dryrun"></div>

  **--dryrun** *[{recv,send}]*, **-n** *[{recv,send}]*

  *  Do a dry run (aka 'no-op') to print what operations would happen
      if the command were to be executed for real (optional). This option
      treats both the ZFS source and destination as read-only. Accepts an
      optional argument for fine tuning that is handled as follows:

      a) 'recv': Send snapshot data via 'zfs send' to the destination
      host and receive it there via 'zfs receive -n', which discards the
      received data there.

      b) 'send': Do not execute 'zfs send' and do not execute 'zfs
      receive'. This is a less 'realistic' form of dry run, but much
      faster, especially for large snapshots and slow networks/disks, as
      no snapshot is actually transferred between source and destination.
      This is the default when specifying --dryrun.

      Examples: --dryrun, --dryrun=send, --dryrun=recv

  <!-- -->

  <div id="--verbose"></div>

  **--verbose**, **-v**

  *  Print verbose information. This option can be specified multiple
      times to increase the level of verbosity. To print what ZFS/SSH
      operation exactly is happening (or would happen), add the `-v -v`
      flag, maybe along with --dryrun. All ZFS and SSH commands (even
      with --dryrun) are logged such that they can be inspected,
      copy-and-pasted into a terminal shell and run manually to help
      anticipate or diagnose issues. ERROR, WARN, INFO, DEBUG, TRACE
      output lines are identified by [E], [W], [I], [D], [T]
      prefixes, respectively.

  <!-- -->

  <div id="--quiet"></div>

  **--quiet**, **-q**

  *  Suppress non-error, info, debug, and trace output.

  <!-- -->

  <div id="--no-privilege-elevation"></div>

  **--no-privilege-elevation**, **-p**

  *  Do not attempt to run state changing ZFS operations 'zfs
      create/rollback/destroy/send/receive' as root (via 'sudo -u root'
      elevation granted by administrators appending the following to
      /etc/sudoers: `<NON_ROOT_USER_NAME> ALL=NOPASSWD:/path/to/zfs`

      Instead, the --no-privilege-elevation flag is for non-root users
      that have been granted corresponding ZFS permissions by
      administrators via 'zfs allow' delegation mechanism, like so: sudo
      zfs allow -u $SRC_NON_ROOT_USER_NAME send,bookmark $SRC_DATASET;
      sudo zfs allow -u $DST_NON_ROOT_USER_NAME
      mount,create,receive,rollback,destroy,canmount,mountpoint,readonly,compression,encryption,keylocation,recordsize
      $DST_DATASET_OR_POOL.

      For extra security $SRC_NON_ROOT_USER_NAME should be different than
      $DST_NON_ROOT_USER_NAME, i.e. the sending Unix user on the source
      and the receiving Unix user at the destination should be separate
      Unix user accounts with separate private keys even if both accounts
      reside on the same machine, per the principle of least privilege.
      Further, if you do not plan to use the --force* flags and
      --delete-* CLI options then ZFS permissions 'rollback,destroy'
      can be omitted. If you do not plan to customize the respective ZFS
      dataset property then ZFS permissions
      'canmount,mountpoint,readonly,compression,encryption,keylocation,recordsize'
      can be omitted, arriving at the absolutely minimal set of required
      destination permissions: `mount,create,receive`.

      Also see
      https://openzfs.github.io/openzfs-docs/man/master/8/zfs-allow.8.html#EXAMPLES
      and https://tinyurl.com/9h97kh8n and
      https://youtu.be/o_jr13Z9f1k?si=7shzmIQJpzNJV6cq

  <!-- -->

  <div id="--no-stream"></div>

  **--no-stream**

  *  During replication, only replicate the most recent selected source
      snapshot of a dataset (using -i incrementals instead of -I
      incrementals), hence skip all intermediate source snapshots that may
      exist between that and the most recent common snapshot. If there is
      no common snapshot also skip all other source snapshots for the
      dataset, except for the most recent selected source snapshot. This
      option helps the destination to 'catch up' with the source ASAP,
      consuming a minimum of disk space, at the expense of reducing
      reliable options for rolling back to intermediate snapshots in the
      future.

  <!-- -->

  <div id="--no-resume-recv"></div>

  **--no-resume-recv**

  *  Replication of snapshots via 'zfs send/receive' can be interrupted
      by intermittent network hiccups, hardware issues, etc. Interrupted
      'zfs send/receive' operations are retried if the --retries and
      --retry-* options enable it (see above). In normal operation bzfs
      automatically retries such that only the portion of the snapshot is
      transmitted that has not yet been fully received on the destination.
      For example, this helps to progressively transfer a large individual
      snapshot over a wireless network in a timely manner despite frequent
      intermittent network hiccups. This optimization is called 'resume
      receive' and uses the 'zfs receive -s' and 'zfs send -t'
      feature.

      The --no-resume-recv option disables this optimization such that a
      retry now retransmits the entire snapshot from scratch, which could
      slow down or even prohibit progress in case of frequent network
      hiccups. bzfs automatically falls back to using the
      --no-resume-recv option if it is auto-detected that the ZFS pool
      does not reliably support the 'resume receive' optimization.

      *Note:* Snapshots that have already been fully transferred as part
      of the current 'zfs send/receive' operation need not be
      retransmitted regardless of the --no-resume-recv flag. For example,
      assume a single 'zfs send/receive' operation is transferring
      incremental snapshots 1 through 10 via 'zfs send -I', but the
      operation fails while transferring snapshot 10, then snapshots 1
      through 9 need not be retransmitted regardless of the
      --no-resume-recv flag, as these snapshots have already been
      successfully received at the destination either way.

  <!-- -->

  <div id="--no-create-bookmark"></div>

  **--no-create-bookmark**

  *  For increased safety, in normal operation bzfs behaves as follows
      wrt. ZFS bookmark creation, if it is autodetected that the source
      ZFS pool support bookmarks: Whenever it has successfully completed
      replication of the most recent source snapshot, bzfs creates a ZFS
      bookmark of that snapshot and attaches it to the source dataset.
      Bookmarks exist so an incremental stream can continue to be sent
      from the source dataset without having to keep the already
      replicated snapshot around on the source dataset until the next
      upcoming snapshot has been successfully replicated. This way you can
      send the snapshot from the source dataset to another host, then
      bookmark the snapshot on the source dataset, then delete the
      snapshot from the source dataset to save disk space, and then still
      incrementally send the next upcoming snapshot from the source
      dataset to the other host by referring to the bookmark.

      The --no-create-bookmark option disables this safety feature but is
      discouraged, because bookmarks are tiny and relatively cheap and
      help to ensure that ZFS replication can continue even if source and
      destination dataset somehow have no common snapshot anymore. For
      example, if a pruning script has accidentally deleted too many (or
      even all) snapshots on the source dataset in an effort to reclaim
      disk space, replication can still proceed because it can use the
      info in the bookmark (the bookmark must still exist in the source
      dataset) instead of the info in the metadata of the (now missing)
      source snapshot.

      A ZFS bookmark is a tiny bit of metadata extracted from a ZFS
      snapshot by the 'zfs bookmark' CLI, and attached to a dataset,
      much like a ZFS snapshot. Note that a ZFS bookmark does not contain
      user data; instead a ZFS bookmark is essentially a tiny pointer in
      the form of the GUID of the snapshot and 64-bit transaction group
      number of the snapshot and creation time of the snapshot, which is
      sufficient to tell the destination ZFS pool how to find the
      destination snapshot corresponding to the source bookmark and
      (potentially already deleted) source snapshot. A bookmark can be fed
      into 'zfs send' as the source of an incremental send. Note that
      while a bookmark allows for its snapshot to be deleted on the source
      after successful replication, it still requires that its snapshot is
      not somehow deleted prematurely on the destination dataset, so be
      mindful of that. By convention, a bookmark created by bzfs has the
      same name as its corresponding snapshot, the only difference being
      the leading '#' separator instead of the leading '@' separator.

      You can list bookmarks, like so: `zfs list -t bookmark -o
      name,guid,createtxg,creation -d 1 $SRC_DATASET`, and you can (and
      should) periodically prune obsolete bookmarks just like snapshots,
      like so: `zfs destroy $SRC_DATASET#$BOOKMARK`. Typically,
      bookmarks should be pruned less aggressively than snapshots, and
      destination snapshots should be pruned less aggressively than source
      snapshots. As an example starting point, here is a command that
      deletes all bookmarks older than 90 days, but retains the latest 200
      bookmarks (per dataset) regardless of creation time: `bzfs dummy
      tank2/boo/bar --dryrun --recursive --skip-replication
      --delete-dst-snapshots=bookmarks
      --include-snapshot-times-and-ranks '0..0' 'latest 200..latest
      100%' --include-snapshot-times-and-ranks '*..90 days ago'`

  <!-- -->

  <div id="--no-use-bookmark"></div>

  **--no-use-bookmark**

  *  For increased safety, in normal replication operation bzfs also
      looks for bookmarks (in addition to snapshots) on the source dataset
      in order to find the most recent common snapshot wrt. the
      destination dataset, if it is auto-detected that the source ZFS pool
      support bookmarks. The --no-use-bookmark option disables this
      safety feature but is discouraged, because bookmarks help to ensure
      that ZFS replication can continue even if source and destination
      dataset somehow have no common snapshot anymore.

      Note that it does not matter whether a bookmark was created by bzfs
      or a third party script, as only the GUID of the bookmark and the
      GUID of the snapshot is considered for comparison, and ZFS
      guarantees that any bookmark of a given snapshot automatically has
      the same GUID, transaction group number and creation time as the
      snapshot. Also note that you can create, delete and prune bookmarks
      any way you like, as bzfs (without --no-use-bookmark) will happily
      work with whatever bookmarks currently exist, if any.

  <!-- -->

  <div id="--ssh-cipher"></div>

  **--ssh-cipher** *STRING*

  *  SSH cipher specification for encrypting the session (optional); will
      be passed into ssh -c CLI. --ssh-cipher is a comma-separated list
      of ciphers listed in order of preference. See the 'Ciphers'
      keyword in ssh_config(5) for more information:
      https://manpages.ubuntu.com/manpages/man5/sshd_config.5.html.
      Default: `^aes256-gcm@openssh.com`

  <!-- -->

  <div id="--ssh-src-private-key"></div>

  **--ssh-src-private-key** *FILE*

  *  Path to SSH private key file on local host to connect to src
      (optional); will be passed into ssh -i CLI. This option can be
      specified multiple times. default: $HOME/.ssh/id_rsa

  <!-- -->

  <div id="--ssh-dst-private-key"></div>

  **--ssh-dst-private-key** *FILE*

  *  Path to SSH private key file on local host to connect to dst
      (optional); will be passed into ssh -i CLI. This option can be
      specified multiple times. default: $HOME/.ssh/id_rsa

  <!-- -->

  <div id="--ssh-src-user"></div>

  **--ssh-src-user** *STRING*

  *  Remote SSH username on src host to connect to (optional). Overrides
      username given in SRC_DATASET.

  <!-- -->

  <div id="--ssh-dst-user"></div>

  **--ssh-dst-user** *STRING*

  *  Remote SSH username on dst host to connect to (optional). Overrides
      username given in DST_DATASET.

  <!-- -->

  <div id="--ssh-src-host"></div>

  **--ssh-src-host** *STRING*

  *  Remote SSH hostname of src host to connect to (optional). Can also
      be an IPv4 or IPv6 address. Overrides hostname given in SRC_DATASET.

  <!-- -->

  <div id="--ssh-dst-host"></div>

  **--ssh-dst-host** *STRING*

  *  Remote SSH hostname of dst host to connect to (optional). Can also
      be an IPv4 or IPv6 address. Overrides hostname given in DST_DATASET.

  <!-- -->

  <div id="--ssh-src-port"></div>

  **--ssh-src-port** *INT*

  *  Remote SSH port on src host to connect to (optional).

  <!-- -->

  <div id="--ssh-dst-port"></div>

  **--ssh-dst-port** *INT*

  *  Remote SSH port on dst host to connect to (optional).

  <!-- -->

  <div id="--ssh-src-extra-opts"></div>

  **--ssh-src-extra-opts** *STRING*

  *  Additional options to be passed to ssh CLI when connecting to src
      host (optional). The value is split on runs of one or more
      whitespace characters. Example: `--ssh-src-extra-opts='-v -v'`
      to debug ssh config issues.

  <!-- -->

  <div id="--ssh-src-extra-opt"></div>

  **--ssh-src-extra-opt** *STRING*

  *  Additional option to be passed to ssh CLI when connecting to src
      host (optional). The value can contain spaces and is not split. This
      option can be specified multiple times. Example:
      `--ssh-src-extra-opt='-oProxyCommand=nc %h %p'` to disable the
      TCP_NODELAY socket option for OpenSSH.

  <!-- -->

  <div id="--ssh-dst-extra-opts"></div>

  **--ssh-dst-extra-opts** *STRING*

  *  Additional options to be passed to ssh CLI when connecting to dst
      host (optional). The value is split on runs of one or more
      whitespace characters. Example: `--ssh-dst-extra-opts='-v -v'`
      to debug ssh config issues.

  <!-- -->

  <div id="--ssh-dst-extra-opt"></div>

  **--ssh-dst-extra-opt** *STRING*

  *  Additional option to be passed to ssh CLI when connecting to dst
      host (optional). The value can contain spaces and is not split. This
      option can be specified multiple times. Example:
      `--ssh-dst-extra-opt='-oProxyCommand=nc %h %p'` to disable the
      TCP_NODELAY socket option for OpenSSH.

  <!-- -->

  <div id="--ssh-src-config-file"></div>

  **--ssh-src-config-file** *FILE*

  *  Path to SSH ssh_config(5) file to connect to src (optional); will be
      passed into ssh -F CLI.

  <!-- -->

  <div id="--ssh-dst-config-file"></div>

  **--ssh-dst-config-file** *FILE*

  *  Path to SSH ssh_config(5) file to connect to dst (optional); will be
      passed into ssh -F CLI.

  <!-- -->

  <div id="--bwlimit"></div>

  **--bwlimit** *STRING*

  *  Sets 'pv' bandwidth rate limit for zfs send/receive data transfer
      (optional). Example: `100m` to cap throughput at 100 MB/sec.
      Default is unlimited. Also see https://linux.die.net/man/1/pv

  <!-- -->

  <div id="--compression-program"></div>

  **--compression-program** *STRING*

  *  The name or path to the 'zstd' executable (optional). Default is
      'zstd'. Examples: 'lz4', 'pigz', 'gzip', '/opt/bin/zstd'.
      Use '-' to disable the use of this program. The use is
      auto-disabled if data is transferred locally instead of via the
      network. This option is about transparent compression-on-the-wire,
      not about compression-at-rest.

  <!-- -->

  <div id="--compression-program-opts"></div>

  **--compression-program-opts** *STRING*

  *  The options to be passed to the compression program on the
      compression step (optional). Default is '-1' (fastest).

  <!-- -->

  <div id="--mbuffer-program"></div>

  **--mbuffer-program** *STRING*

  *  The name or path to the 'mbuffer' executable (optional). Default
      is 'mbuffer'. Use '-' to disable the use of this program. The
      use on dst is auto-disabled if data is transferred locally instead
      of via the network. This tool is used to smooth out the rate of data
      flow and prevent bottlenecks caused by network latency or speed
      fluctuation.

  <!-- -->

  <div id="--mbuffer-program-opts"></div>

  **--mbuffer-program-opts** *STRING*

  *  Options to be passed to 'mbuffer' program (optional). Default:
      '-q -m 128M'.

  <!-- -->

  <div id="--ps-program"></div>

  **--ps-program** *STRING*

  *  The name or path to the 'ps' executable (optional). Default is
      'ps'. Use '-' to disable the use of this program.

  <!-- -->

  <div id="--pv-program"></div>

  **--pv-program** *STRING*

  *  The name or path to the 'pv' executable (optional). Default is
      'pv'. Use '-' to disable the use of this program. This is used
      for bandwidth rate-limiting and progress monitoring.

  <!-- -->

  <div id="--pv-program-opts"></div>

  **--pv-program-opts** *STRING*

  *  The options to be passed to the 'pv' program (optional). Default:
      '--progress --timer --eta --fineta --rate --average-rate
      --bytes --interval=1 --width=120 --buffer-size=2M'.

  <!-- -->

  <div id="--shell-program"></div>

  **--shell-program** *STRING*

  *  The name or path to the 'sh' executable (optional). Default is
      'sh'. Use '-' to disable the use of this program.

  <!-- -->

  <div id="--ssh-program"></div>

  **--ssh-program** *STRING*

  *  The name or path to the 'ssh' executable (optional). Default is
      'ssh'. Examples: 'hpnssh' or 'ssh' or '/opt/bin/ssh' or
      wrapper scripts around 'ssh'. Use '-' to disable the use of this
      program.

  <!-- -->

  <div id="--sudo-program"></div>

  **--sudo-program** *STRING*

  *  The name or path to the 'sudo' executable (optional). Default is
      'sudo'. Use '-' to disable the use of this program.

  <!-- -->

  <div id="--zfs-program"></div>

  **--zfs-program** *STRING*

  *  The name or path to the 'zfs' executable (optional). Default is
      'zfs'.

  <!-- -->

  <div id="--zpool-program"></div>

  **--zpool-program** *STRING*

  *  The name or path to the 'zpool' executable (optional). Default is
      'zpool'. Use '-' to disable the use of this program.

  <!-- -->

  <div id="--log-dir"></div>

  **--log-dir** *DIR*

  *  Path to the log output directory on local host (optional). Default:
      $HOME/bzfs-logs. The logger that is used by default writes log
      files there, in addition to the console. The current.log symlink
      always points to the most recent log file. The current.pv symlink
      always points to the most recent data transfer monitoring log. Run
      'tail --follow=name --max-unchanged-stats=1' on both symlinks to
      follow what's currently going on. The current.dir symlink always
      points to the sub directory containing the most recent log file.

  <!-- -->

  <div id="--log-file-prefix"></div>

  **--log-file-prefix** *STRING*

  *  Default is zrun_. The path name of the log file on local host is
      `${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-suffix}-<random>.log`.
      Example: `--log-file-prefix=zrun_ --log-file-suffix=_daily`
      will generate log file names such as
      `zrun_2024-09-03_12:26:15_daily-bl4i1fth.log`

  <!-- -->

  <div id="--log-file-suffix"></div>

  **--log-file-suffix** *STRING*

  *  Default is the empty string. The path name of the log file on local
      host is
      `${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-suffix}-<random>.log`.
      Example: `--log-file-prefix=zrun_ --log-file-suffix=_daily`
      will generate log file names such as
      `zrun_2024-09-03_12:26:15_daily-bl4i1fth.log`

  <!-- -->

  <div id="--log-syslog-address"></div>

  **--log-syslog-address** *STRING*

  *  Host:port of the syslog machine to send messages to (e.g.
      'foo.example.com:514' or '127.0.0.1:514'), or the file system
      path to the syslog socket file on localhost (e.g. '/dev/log'). The
      default is no address, i.e. do not log anything to syslog by
      default. See
      https://docs.python.org/3/library/logging.handlers.html#sysloghandler

  <!-- -->

  <div id="--log-syslog-socktype"></div>

  **--log-syslog-socktype** *{UDP,TCP}*

  *  The socket type to use to connect if no local socket file system
      path is used. Default is 'UDP'.

  <!-- -->

  <div id="--log-syslog-facility"></div>

  **--log-syslog-facility** *INT*

  *  The local facility aka category that identifies msg sources in
      syslog (default: 1, min=0, max=7).

  <!-- -->

  <div id="--log-syslog-prefix"></div>

  **--log-syslog-prefix** *STRING*

  *  The name to prepend to each message that is sent to syslog;
      identifies bzfs messages as opposed to messages from other sources.
      Default is 'bzfs'.

  <!-- -->

  <div id="--log-syslog-level"></div>

  **--log-syslog-level** *{CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE}*

  *  Only send messages with equal or higher priority than this log level
      to syslog. Default is 'ERROR'.

  <!-- -->

  <div id="--log-config-file"></div>

  **--log-config-file** *STRING*

  *  The contents of a JSON file that defines a custom python logging
      configuration to be used (optional). If the option starts with a
      `+` prefix then the contents are read from the UTF-8 JSON file
      given after the `+` prefix. Examples: +log_config.json,
      +/path/to/log_config.json. Here is an example config file that
      demonstrates usage:
      https://github.com/whoschek/bzfs/blob/main/tests/log_config.json

      For more examples see
      https://stackoverflow.com/questions/7507825/where-is-a-complete-example-of-logging-config-dictconfig
      and for details see
      https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema

      *Note:* Lines starting with a # character are ignored as comments
      within the JSON. Also, if a line ends with a # character the
      portion between that # character and the preceding # character on
      the same line is ignored as a comment.

  <!-- -->

  <div id="--log-config-var"></div>

  **--log-config-var** *NAME:VALUE [NAME:VALUE ...]*

  *  User defined variables in the form of zero or more NAME:VALUE pairs
      (optional). These variables can be used within the JSON passed with
      --log-config-file (see above) via `${name[:default]}`
      references, which are substituted (aka interpolated) as follows:

      If the variable contains a non-empty CLI value then that value is
      used. Else if a default value for the variable exists in the JSON
      file that default value is used. Else the program aborts with an
      error. Example: In the JSON variable
      `${syslog_address:/dev/log}`, the variable name is
      'syslog_address' and the default value is '/dev/log'. The
      default value is the portion after the optional : colon within the
      variable declaration. The default value is used if the CLI user does
      not specify a non-empty value via --log-config-var, for example via
      --log-config-var syslog_address:/path/to/socket_file or via
      --log-config-var syslog_address:[host,port].

      bzfs automatically supplies the following convenience variables:
      `${bzfs.log_level}`, `${bzfs.log_dir}`, `${bzfs.log_file}`,
      `${bzfs.sub.logger}`, `${bzfs.get_default_log_formatter}`,
      `${bzfs.timestamp}`. For a complete list see the source code of
      get_dict_config_logger().

  <!-- -->

  <div id="--include-envvar-regex"></div>

  **--include-envvar-regex** *REGEX [REGEX ...]*

  *  On program startup, unset all Unix environment variables for which
      the full environment variable name matches at least one of the
      excludes but none of the includes. If an environment variable is
      included this decision is never reconsidered because include takes
      precedence over exclude. The purpose is to tighten security and help
      guard against accidental inheritance or malicious injection of
      environment variable values that may have unintended effects.

      This option can be specified multiple times. A leading `!`
      character indicates logical negation, i.e. the regex matches if the
      regex with the leading `!` character removed does not match. The
      default is to include no environment variables, i.e. to make no
      exceptions to --exclude-envvar-regex. Example that retains at least
      these two env vars: `--include-envvar-regex PATH
      --include-envvar-regex bzfs_min_pipe_transfer_size`. Example that
      retains all environment variables without tightened security:
      `'.*'`

  <!-- -->

  <div id="--exclude-envvar-regex"></div>

  **--exclude-envvar-regex** *REGEX [REGEX ...]*

  *  Same syntax as --include-envvar-regex (see above) except that the
      default is to exclude no environment variables. Example:
      `bzfs_.*`

  <!-- -->

  <div id="--version"></div>

  **--version**

  *  Display version information and exit.

  <!-- -->

  <div id="--help,_-h"></div>

  **--help, -h**

  *  Show this help message and exit.



  The following group of parameters specifies additional zfs receive
  '-o' options that can be used to configure the copying of ZFS dataset
  properties from the source dataset to its corresponding destination
  dataset. The 'zfs-recv-o' group of parameters is applied before the
  'zfs-recv-x' group.

  <div id="--zfs-recv-o-targets"></div>

  **--zfs-recv-o-targets** *{full,incremental,full+incremental}*

  *  The zfs send phase or phases during which the extra '-o' options
      are passed to 'zfs receive'. This can be one of the following
      choices: 'full', 'incremental', 'full+incremental'. Default is
      'full+incremental'. A 'full' send is sometimes also known as an
      'initial' send.

  <!-- -->

  <div id="--zfs-recv-o-sources"></div>

  **--zfs-recv-o-sources** *STRING*

  *  The ZFS sources to provide to the 'zfs get -s' CLI in order to
      fetch the ZFS dataset properties that will be fed into the
      --zfs-recv-o-include/exclude-regex filter (see below). The sources
      are in the form of a comma-separated list (no spaces) containing one
      or more of the following choices: 'local', 'default',
      'inherited', 'temporary', 'received', 'none', with the
      default being 'local'. Uses 'zfs get -p -s $zfs-recv-o-sources
      all $SRC_DATASET' to fetch the properties to copy -
      https://openzfs.github.io/openzfs-docs/man/master/8/zfs-get.8.html.
      P.S: Note that the existing 'zfs send --props' option does not
      filter and that --props only reads properties from the 'local'
      ZFS property source (https://github.com/openzfs/zfs/issues/13024).

  <!-- -->

  <div id="--zfs-recv-o-include-regex"></div>

  **--zfs-recv-o-include-regex** *REGEX [REGEX ...]*

  *  Take the output properties of --zfs-recv-o-sources (see above) and
      filter them such that we only retain the properties whose name
      matches at least one of the --include regexes but none of the
      --exclude regexes. If a property is excluded this decision is never
      reconsidered because exclude takes precedence over include. Append
      each retained property to the list of '-o' options in
      --zfs-recv-program-opt(s), unless another '-o' or '-x' option
      with the same name already exists therein. In other words,
      --zfs-recv-program-opt(s) takes precedence.

      The --zfs-recv-o-include-regex option can be specified multiple
      times. A leading `!` character indicates logical negation, i.e.
      the regex matches if the regex with the leading `!` character
      removed does not match. If the option starts with a `+` prefix
      then regexes are read from the newline-separated UTF-8 text file
      given after the `+` prefix, one regex per line inside of the text
      file.

      The default is to include no properties, thus by default no extra
      '-o' option is appended. Example: `--zfs-recv-o-include-regex
      recordsize volblocksize`. More examples: `.*` (include all
      properties), `foo bar myapp:.*` (include three regexes)
      `+zfs-recv-o_regexes.txt`, `+/path/to/zfs-recv-o_regexes.txt`

  <!-- -->

  <div id="--zfs-recv-o-exclude-regex"></div>

  **--zfs-recv-o-exclude-regex** *REGEX [REGEX ...]*

  *  Same syntax as --zfs-recv-o-include-regex (see above), and the
      default is to exclude no properties. Example:
      --zfs-recv-o-exclude-regex encryptionroot keystatus origin
      volblocksize volsize



  The following group of parameters specifies additional zfs receive
  '-x' options that can be used to configure the copying of ZFS dataset
  properties from the source dataset to its corresponding destination
  dataset. The 'zfs-recv-o' group of parameters is applied before the
  'zfs-recv-x' group.

  <div id="--zfs-recv-x-targets"></div>

  **--zfs-recv-x-targets** *{full,incremental,full+incremental}*

  *  The zfs send phase or phases during which the extra '-x' options
      are passed to 'zfs receive'. This can be one of the following
      choices: 'full', 'incremental', 'full+incremental'. Default is
      'full+incremental'. A 'full' send is sometimes also known as an
      'initial' send.

  <!-- -->

  <div id="--zfs-recv-x-sources"></div>

  **--zfs-recv-x-sources** *STRING*

  *  The ZFS sources to provide to the 'zfs get -s' CLI in order to
      fetch the ZFS dataset properties that will be fed into the
      --zfs-recv-x-include/exclude-regex filter (see below). The sources
      are in the form of a comma-separated list (no spaces) containing one
      or more of the following choices: 'local', 'default',
      'inherited', 'temporary', 'received', 'none', with the
      default being 'local'. Uses 'zfs get -p -s $zfs-recv-x-sources
      all $SRC_DATASET' to fetch the properties to copy -
      https://openzfs.github.io/openzfs-docs/man/master/8/zfs-get.8.html.
      P.S: Note that the existing 'zfs send --props' option does not
      filter and that --props only reads properties from the 'local'
      ZFS property source (https://github.com/openzfs/zfs/issues/13024).
      Thus, -x opts do not benefit from source != 'local' (which is the
      default already).

  <!-- -->

  <div id="--zfs-recv-x-include-regex"></div>

  **--zfs-recv-x-include-regex** *REGEX [REGEX ...]*

  *  Take the output properties of --zfs-recv-x-sources (see above) and
      filter them such that we only retain the properties whose name
      matches at least one of the --include regexes but none of the
      --exclude regexes. If a property is excluded this decision is never
      reconsidered because exclude takes precedence over include. Append
      each retained property to the list of '-x' options in
      --zfs-recv-program-opt(s), unless another '-o' or '-x' option
      with the same name already exists therein. In other words,
      --zfs-recv-program-opt(s) takes precedence.

      The --zfs-recv-x-include-regex option can be specified multiple
      times. A leading `!` character indicates logical negation, i.e.
      the regex matches if the regex with the leading `!` character
      removed does not match. If the option starts with a `+` prefix
      then regexes are read from the newline-separated UTF-8 text file
      given after the `+` prefix, one regex per line inside of the text
      file.

      The default is to include no properties, thus by default no extra
      '-x' option is appended. Example: `--zfs-recv-x-include-regex
      recordsize volblocksize`. More examples: `.*` (include all
      properties), `foo bar myapp:.*` (include three regexes)
      `+zfs-recv-x_regexes.txt`, `+/path/to/zfs-recv-x_regexes.txt`

  <!-- -->

  <div id="--zfs-recv-x-exclude-regex"></div>

  **--zfs-recv-x-exclude-regex** *REGEX [REGEX ...]*

  *  Same syntax as --zfs-recv-x-include-regex (see above), and the
      default is to exclude no properties. Example:
      --zfs-recv-x-exclude-regex encryptionroot keystatus origin
      volblocksize volsize


pkgId: 477eb72e93853e09f9a988e4b3847995d129ece44539a57c7d7d317f603df24e
name: python-bzt
arch: src
version: 1.16.34
epoch: 0
release: 1
summary: Taurus Tool for Continuous Testing
url: http://gettaurus.org/
rpm_license: Apache 2.0
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-bzt-1.16.34-1.src.rpm
checksum_type: sha256
description:
  execution:
  - concurrency: 10
    ramp-up: 1m
    hold-for: 1m30s
    scenario: simple
  scenarios:
    simple:
      think-time: 0.75
      requests:
      - http://blazedemo.com/
      - http://blazedemo.com/vacation.html
  ```
  Then run `bzt test.yml`. After the tool finishes, observe resulting summary stats in console log (more reporting options [here](http://gettaurus.org/docs/Reporting.md)). All artifact files from the run will be placed in the directory mentioned in console log. Read more on command-line tool usage [here](http://gettaurus.org/docs/CommandLine.md).
  ![Analytics](https://ga-beacon.appspot.com/UA-63369152-1/taurus/readme)


pkgId: 25e5f650e1f76a5d46be0bb46fc5999496341e3f44f64a720cad71f6400a91e7
name: python-c-lasso
arch: src
version: 1.0.11
epoch: 0
release: 1
summary: Algorithms for constrained Lasso problems
url: https://github.com/Leo-Simpson/CLasso
rpm_license: MIT
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c-lasso-1.0.11-1.src.rpm
checksum_type: sha256
description:
  [![arXiv](https://img.shields.io/badge/arXiv-2011.00898-b31b1b.svg)](https://arxiv.org/abs/2011.00898)
  [![DOI](https://joss.theoj.org/papers/10.21105/joss.02844/status.svg)](https://doi.org/10.21105/joss.02844)

  <img src="https://i.imgur.com/2nGwlux.png" alt="c-lasso" height="145" align="right"/>




  c-lasso is a Python package that enables sparse and robust linear regression and classification with linear equality
  constraints on the model parameters. For detailed info, one can check the [documentation](https://c-lasso.readthedocs.io/en/latest/).

  The forward model is assumed to be:

  <img src="https://latex.codecogs.com/gif.latex?y=X\beta&plus;\sigma\epsilon\qquad\text{s.t.}\qquad&space;C\beta=0" title="y=X\beta+\sigma\epsilon\qquad\text{s.t.}\qquad C\beta=0" />

  Here, y and X are given outcome and predictor data. The vector y can be continuous (for regression) or binary (for classification). C is a general constraint matrix. The vector &beta; comprises the unknown coefficients and &sigma; an
  unknown scale.

  The package handles several different estimators for inferring &beta; (and &sigma;), including
  the constrained Lasso, the constrained scaled Lasso, sparse Huber M-estimation with linear equality constraints, and regularized Support Vector Machines.
  Several different algorithmic strategies, including path and proximal splitting algorithms, are implemented to solve
  the underlying convex optimization problems.

  We also include two model selection strategies for determining the sparsity of the model parameters: k-fold cross-validation and stability selection.

  This package is intended to fill the gap between popular python tools such as [scikit-learn](https://scikit-learn.org/stable/) which CANNOT solve sparse constrained problems and general-purpose optimization solvers that do not scale well or are inaccurate (see [benchmarks](./benchmark/README.md)) for the considered problems. In its current stage, however, c-lasso is not yet compatible with the scikit-learn API but rather a stand-alone tool.

  Below we show several use cases of the package, including an application of sparse *log-contrast*
  regression tasks for *compositional* microbiome data.

  The code builds on results from several papers which can be found in the [References](#references). We also refer to the accompanying [JOSS paper submission](https://github.com/Leo-Simpson/c-lasso/blob/master/paper/paper.md), also available on [arXiv](https://arxiv.org/pdf/2011.00898.pdf).



  * [Installation](#installation)
  * [Regression and classification problems](#regression-and-classification-problems)
  * [Getting started](#getting-started)
  * [Log-contrast regression for microbiome data](#log-contrast-regression-for-microbiome-data)
  * [Optimization schemes](#optimization-schemes)


  * [References](#references)




  c-lasso is available on pip. You can install the package
  in the shell using

  ```shell
  pip install c-lasso
  ```
  To use the c-lasso package in Python, type

  ```python

  from classo import classo_problem

  ```

  The `c-lasso` package depends on the following Python packages:

  - `numpy`;
  - `matplotlib`;
  - `scipy`;
  - `pandas`;
  - `pytest` (for tests)



  The c-lasso package can solve six different types of estimation problems:
  four regression-type and two classification-type formulations.



  <img src="https://latex.codecogs.com/gif.latex?\arg\min_{\beta\in&space;R^d}&space;||&space;X\beta-y&space;||^2&space;&plus;&space;\lambda&space;||\beta||_1&space;\qquad\mbox{s.t.}\qquad&space;C\beta=0" />

  This is the standard Lasso problem with linear equality constraints on the &beta; vector.
  The objective function combines Least-Squares for model fitting with l1 penalty for sparsity.



  <img src="https://latex.codecogs.com/gif.latex?\arg\min_{\beta\in&space;R^d}&space;h_{\rho}(X\beta-y&space;)&space;&plus;&space;\lambda&space;||\beta||_1&space;\qquad\mbox{s.t.}\qquad&space;C\beta=0" />

  This regression problem uses the [Huber loss](https://en.wikipedia.org/wiki/Huber_loss) as objective function
  for robust model fitting with l1 and linear equality constraints on the &beta; vector. The parameter &rho;=1.345.



  <img src="https://latex.codecogs.com/gif.latex?\arg&space;\min_{\beta&space;\in&space;\mathbb{R}^d,&space;\sigma&space;>&space;0}&space;\frac{||&space;X\beta&space;-&space;y||^2}{\sigma}&space;&plus;&space;\frac{n}{2}&space;\sigma&plus;&space;\lambda&space;||\beta||_1&space;\qquad&space;\mbox{s.t.}&space;\qquad&space;C\beta&space;=&space;0" title="\arg \min_{\beta \in \mathbb{R}^d, \sigma > 0} \frac{|| X\beta - y||^2}{\sigma} + \frac{n}{2} \sigma+ \lambda ||\beta||_1 \qquad \mbox{s.t.} \qquad C\beta = 0" />

  This formulation is similar to [R1] but allows for joint estimation of the (constrained) &beta; vector and
  the standard deviation &sigma; in a concomitant fashion (see [References](#references) [4,5] for further info).
  This is the default problem formulation in c-lasso.



  <img src="https://latex.codecogs.com/gif.latex?\arg&space;\min_{\beta&space;\in&space;\mathbb{R}^d,&space;\sigma&space;>&space;0}&space;\left(&space;h_{\rho}&space;\left(&space;\frac{&space;X\beta&space;-&space;y}{\sigma}&space;\right)&plus;&space;n&space;\right)&space;\sigma&plus;&space;\lambda&space;||\beta||_1&space;\qquad&space;\mbox{s.t.}&space;\qquad&space;C\beta&space;=&space;0" title="\arg \min_{\beta \in \mathbb{R}^d, \sigma > 0} \left( h_{\rho} \left( \frac{ X\beta - y}{\sigma} \right)+ n \right) \sigma+ \lambda ||\beta||_1 \qquad \mbox{s.t.} \qquad C\beta = 0" />

  This formulation combines [R2] and [R3] to allow robust joint estimation of the (constrained) &beta; vector and
  the scale &sigma; in a concomitant fashion (see [References](#references) [4,5] for further info).



  <img src="https://latex.codecogs.com/gif.latex?\arg&space;\min_{\beta&space;\in&space;\mathbb{R}^d}&space;\sum_{i=1}^n&space;l(y_i&space;x_i^\top&space;\beta)&space;&plus;&space;\lambda&space;\left\lVert&space;\beta\right\rVert_1&space;\qquad&space;s.t.&space;\qquad&space;C\beta&space;=&space;0" title="\arg \min_{\beta \in \mathbb{R}^d} \sum_{i=1}^n l(y_i x_i \beta) + \lambda \left\lVert \beta\right\rVert_1 \qquad s.t. \qquad C\beta = 0" />

  where the x<sub>i</sub> are the rows of X and l is defined as:

  <img src="https://latex.codecogs.com/gif.latex?l(r)&space;=&space;\begin{cases}&space;(1-r)^2&space;&&space;if&space;\quad&space;r&space;\leq&space;1&space;\\&space;0&space;&if&space;\quad&space;r&space;\geq&space;1&space;\end{cases}" title="l(r) = \begin{cases} (1-r)^2 & if \quad r \leq 1 \\ 0 &if \quad r \geq 1 \end{cases}" />

  This formulation is similar to [R1] but adapted for classification tasks using the Square Hinge loss
  with (constrained) sparse &beta; vector estimation.



  <img src="https://latex.codecogs.com/gif.latex?\arg&space;\min_{\beta&space;\in&space;\mathbb{R}^d}&space;\sum_{i=1}^n&space;l_{\rho}(y_i&space;x_i^\top\beta)&space;&plus;&space;\lambda&space;\left\lVert&space;\beta\right\rVert_1&space;\qquad&space;s.t.&space;\qquad&space;C\beta&space;=&space;0" title="\arg \min_{\beta \in \mathbb{R}^d} \sum_{i=1}^n l_{\rho}(y_i x_i\beta) + \lambda \left\lVert \beta\right\rVert_1 \qquad s.t. \qquad C\beta = 0" />

  where the x<sub>i</sub> are the rows of X and l<sub>ρ</sub> is defined as:

  <img src="https://latex.codecogs.com/gif.latex?l_{\rho}(r)&space;=&space;\begin{cases}&space;(1-r)^2&space;&if&space;\quad&space;\rho&space;\leq&space;r&space;\leq&space;1&space;\\&space;(1-\rho)(1&plus;\rho-2r)&space;&&space;if&space;\quad&space;r&space;\leq&space;\rho&space;\\&space;0&space;&if&space;\quad&space;r&space;\geq&space;1&space;\end{cases}" title="l_{\rho}(r) = \begin{cases} (1-r)^2 &if \quad \rho \leq r \leq 1 \\ (1-\rho)(1+\rho-2r) & if \quad r \leq \rho \\ 0 &if \quad r \geq 1 \end{cases}" />


  This formulation is similar to [C1] but uses the Huberized Square Hinge loss for robust classification
  with (constrained) sparse &beta; vector estimation.






  We begin with a basic example that shows how to run c-lasso on synthetic data. This example and the next one can be found on the notebook 'Synthetic data Notebook.ipynb'

  The c-lasso package includes
  the routine ```random_data``` that allows you to generate problem instances using normally distributed data.

  ```python
  m, d, d_nonzero, k, sigma = 100, 200, 5, 1, 0.5
  (X, C, y), sol = random_data(m, d, d_nonzero, k, sigma, zerosum=True, seed=1)
  ```
  This code snippet generates a problem instance with sparse &beta; in dimension
  d=100 (sparsity d_nonzero=5). The design matrix X comprises n=100 samples generated from an i.i.d standard normal
  distribution. The dimension of the constraint matrix C is d x k matrix. The noise level is &sigma;=0.5.
  The input ```zerosum=True``` implies that C is the all-ones vector and C&beta;=0. The n-dimensional outcome vector y
  and the regression vector &beta; is then generated to satisfy the given constraints.

  Next we can define a default c-lasso problem instance with the generated data:
  ```python
  problem = classo_problem(X, y, C)
  ```
  You can look at the generated problem instance by typing:

  ```python
  print(problem)
  ```

  This gives you a summary of the form:

  ```
  FORMULATION: R3

  MODEL SELECTION COMPUTED:
       Stability selection

  STABILITY SELECTION PARAMETERS:
       numerical_method : not specified
       method : first
       B = 50
       q = 10
       percent_nS = 0.5
       threshold = 0.7
       lamin = 0.01
       Nlam = 50
  ```
  As we have not specified any problem, algorithm, or model selection settings, this problem instance
  represents the *default* settings for a c-lasso instance:
  - The problem is of regression type and uses formulation [R3], i.e. with concomitant scale estimation.
  - The *default* optimization scheme is the path algorithm (see [Optimization schemes](#optimization-schemes) for further info).
  - For model selection, stability selection at a theoretically derived &lambda; value is used (see [Reference](#references) [4] for details). Stability selection comprises a relatively large number of parameters. For a description of the settings, we refer to the more advanced examples below and the API.

  You can solve the corresponding c-lasso problem instance using

  ```python
  problem.solve()
  ```

  After completion, the results of the optimization and model selection routines
  can be visualized using

  ```python
  print(problem.solution)
  ```

  The command shows the running time(s) for the c-lasso problem instance, and the selected variables for sability selection

  ```
  STABILITY SELECTION :
     Selected variables :  7    63    148    164    168
     Running time :  1.546s

  ```

  Here, we only used stability selection as *default* model selection strategy.
  The command also allows you to inspect the computed stability profile for all variables
  at the theoretical &lambda;

  ![1.StabSel](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/basic/StabSel.png)


  The refitted &beta; values on the selected support are also displayed in the next plot

  ![beta](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/basic/beta.png)




  In the next example, we show how one can specify different aspects of the problem
  formulation and model selection strategy.

  ```python
  m,  d,  d_nonzero,  k, sigma = 100, 200, 5, 0, 0.5
  (X, C, y), sol = random_data(m, d, d_nonzero, k, sigma, zerosum = True, seed = 4)
  problem                                     = classo_problem(X, y, C)
  problem.formulation.huber                   = True
  problem.formulation.concomitant             = False
  problem.model_selection.CV                  = True
  problem.model_selection.LAMfixed            = True
  problem.model_selection.PATH                = True
  problem.model_selection.StabSelparameters.method = 'max'
  problem.model_selection.CVparameters.seed = 1
  problem.model_selection.LAMfixedparameters.rescaled_lam = True
  problem.model_selection.LAMfixedparameters.lam = .1

  problem.solve()
  print(problem)

  print(problem.solution)

  ```

  Results :
  ```
       FORMULATION: R2

       MODEL SELECTION COMPUTED:
            Lambda fixed
            Path
            Cross Validation
            Stability selection

       LAMBDA FIXED PARAMETERS:
            numerical_method = Path-Alg
            rescaled lam : True
            threshold = 0.09
            lam = 0.1
            theoretical_lam = 0.224

       PATH PARAMETERS:
            numerical_method : Path-Alg
            lamin = 0.001
            Nlam = 80


       CROSS VALIDATION PARAMETERS:
            numerical_method : Path-Alg
            one-SE method : True
            Nsubset = 5
            lamin = 0.001
            Nlam = 80


       STABILITY SELECTION PARAMETERS:
            numerical_method : Path-Alg
            method : max
            B = 50
            q = 10
            percent_nS = 0.5
            threshold = 0.7
            lamin = 0.01
            Nlam = 50

       LAMBDA FIXED :
       Selected variables :  17    59    123
       Running time :  0.104s

       PATH COMPUTATION :
       Running time :  0.638s

       CROSS VALIDATION :
       Selected variables :  16    17    57    59    64    73    74    76    93    115    123    134    137    181
       Running time :  2.1s

       STABILITY SELECTION :
       Selected variables :  17    59    76    123    137
       Running time :  6.062s

  ```


  ![2.StabSel](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/StabSel.png)

  ![2.StabSel-beta](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/StabSel-beta.png)

  ![2.CV-beta](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/CVbeta.png)

  ![2.CV-graph](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/CV.png)

  ![2.LAM-beta](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/beta.png)

  ![2.Path](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/advanced/Beta-path.png)




  In the [the accompanying notebook](./examples/example-notebook.ipynb) we study several microbiome data sets. We showcase two examples below.



  We first consider the [COMBO data set](./examples/COMBO_data) and show how to predict Body Mass Index (BMI) from microbial genus abundances and two non-compositional covariates  using "filtered_data".

  ```python
  from classo import csv_to_np, classo_problem, clr


  X0  = csv_to_np('COMBO_data/complete_data/GeneraCounts.csv', begin = 0).astype(float)
  X_C = csv_to_np('COMBO_data/CaloriData.csv', begin = 0).astype(float)
  X_F = csv_to_np('COMBO_data/FatData.csv', begin = 0).astype(float)


  y   = csv_to_np('COMBO_data/BMI.csv', begin = 0).astype(float)[:, 0]
  labels = csv_to_np('COMBO_data/complete_data/GeneraPhylo.csv').astype(str)[:, -1]



  y   = y - np.mean(y) #BMI data (n = 96)
  X_C = X_C - np.mean(X_C, axis = 0)  #Covariate data (Calorie)
  X_F = X_F - np.mean(X_F, axis = 0)  #Covariate data (Fat)
  X0 = clr(X0, 1 / 2).T


  X     = np.concatenate((X0, X_C, X_F, np.ones((len(X0), 1))), axis = 1) # Joint microbiome and covariate data and offset
  label = np.concatenate([labels, np.array(['Calorie', 'Fat', 'Bias'])])
  C = np.ones((1, len(X[0])))
  C[0, -1], C[0, -2], C[0, -3] = 0., 0., 0.



  problem = classo_problem(X, y, C, label = label)



  problem.model_selection.StabSelparameters.method      = 'lam'
  problem.model_selection.StabSelparameters.threshold_label = 0.5


  problem.formulation.concomitant = True

  problem.solve()
  print(problem)
  print(problem.solution)


  problem.formulation.huber = True
  problem.formulation.concomitant = True

  problem.solve()
  print(problem)
  print(problem.solution)

  ```

  ![3.Stability profile R3](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/exampleFilteredCOMBO/R3-StabSel.png)

  ![3.Beta solution R3](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/exampleFilteredCOMBO/R3-StabSel-beta.png)

  ![3.Stability profile R4](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/exampleFilteredCOMBO/R4-StabSel.png)

  ![3.Beta solution R4](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/exampleFilteredCOMBO/R4-StabSel-beta.png)


  <!---
  <img src="https://i.imgur.com/8tFmM8T.png" alt="Central Park Soil Microbiome" height="250" align="right"/>

  The next microbiome example considers the [Central Park Soil dataset](./examples/pH_data) from [Ramirez et al.](https://royalsocietypublishing.org/doi/full/10.1098/rspb.2014.1988). The sample locations are shown in the Figure on the right.)
  -->



  The next microbiome example considers the [88 soils dataset](./examples/pH_data) from [Lauber et al., 2009](https://pubmed.ncbi.nlm.nih.gov/19502440/).

  The task is to predict pH concentration in the soil from microbial abundance data. A similar analysis is available
  in [Tree-Aggregated Predictive Modeling of Microbiome Data](https://www.biorxiv.org/content/10.1101/2020.09.01.277632v1)
  with Central Park soil data from [Ramirez et al.](https://royalsocietypublishing.org/doi/full/10.1098/rspb.2014.1988).

  Code to run this application is available in [the accompanying notebook](./examples/example-notebook.ipynb) under `pH data`. Below is a summary of a c-lasso problem instance (using the R3 formulation).

  ```
  FORMULATION: R3

  MODEL SELECTION COMPUTED:
       Lambda fixed
       Path
       Stability selection

  LAMBDA FIXED PARAMETERS:
       numerical_method = Path-Alg
       rescaled lam : True
       threshold = 0.004
       lam : theoretical
       theoretical_lam = 0.2182

  PATH PARAMETERS:
       numerical_method : Path-Alg
       lamin = 0.001
       Nlam = 80


  STABILITY SELECTION PARAMETERS:
       numerical_method : Path-Alg
       method : lam
       B = 50
       q = 10
       percent_nS = 0.5
       threshold = 0.7
       lam = theoretical
       theoretical_lam = 0.3085
  ```

  The c-lasso estimation results are summarized below:

  ```
  LAMBDA FIXED :
     Sigma  =  0.198
     Selected variables :  14    18    19    39    43    57    62    85    93    94    104    107
     Running time :  0.008s

   PATH COMPUTATION :
     Running time :  0.12s

   STABILITY SELECTION :
     Selected variables :  2    12    15
     Running time :  0.287s
  ```

  ![Ex4.1](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/examplePH/R3-Beta-path.png)

  ![Ex4.2](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/examplePH/R3-Sigma-path.png)

  ![Ex4.3](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/examplePH/R3-StabSel.png)

  ![Ex4.4](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/examplePH/R3-StabSel-beta.png)

  ![Ex4.5](https://github.com/Leo-Simpson/c-lasso/blob/master/figures/examplePH/R3-beta.png)




  The available problem formulations [R1-C2] require different algorithmic strategies for
  efficiently solving the underlying optimization problem. We have implemented four
  algorithms (with provable convergence guarantees) that vary in generality and are not
  necessarily applicable to all problems. For each problem type, c-lasso has a default algorithm
  setting that proved to be the fastest in our numerical experiments.


  This is the default algorithm for non-concomitant problems [R1,R3,C1,C2].
  The algorithm uses the fact that the solution path along &lambda; is piecewise-
  affine (as shown, e.g., in [1]). When Least-Squares is used as objective function,
  we derive a novel efficient procedure that allows us to also derive the
  solution for the concomitant problem [R2] along the path with little extra computational overhead.


  This algorithm is derived from [2] and belongs to the class of
  proximal splitting algorithms. It extends the classical Forward-Backward (FB)
  (aka proximal gradient descent) algorithm to handle an additional linear equality constraint
  via projection. In the absence of a linear constraint, the method reduces to FB.
  This method can solve problem [R1]. For the Huber problem [R3],
  P-PDS can solve the mean-shift formulation of the problem (see [6]).


  This algorithm is a special case of an algorithm proposed in [3] (Eq.4.5) and also belongs to the class of
  proximal splitting algorithms. The algorithm does not require projection operators
  which may be beneficial when C has a more complex structure. In the absence of a linear constraint,
  the method reduces to the Forward-Backward-Forward scheme. This method can solve problem [R1].
  For the Huber problem [R3], PF-PDS can solve the mean-shift formulation of the problem (see [6]).


  This algorithm is the most general algorithm and can solve all regression problems
  [R1-R4]. It is based on Doulgas Rachford splitting in a higher-dimensional product space.
  It makes use of the proximity operators of the perspective of the LS objective (see [4,5])
  The Huber problem with concomitant scale [R4] is reformulated as scaled Lasso problem
  with the mean shift (see [6]) and thus solved in (n + d) dimensions.





  * [1] B. R. Gaines, J. Kim, and H. Zhou, [Algorithms for Fitting the Constrained Lasso](https://www.tandfonline.com/doi/abs/10.1080/10618600.2018.1473777?journalCode=ucgs20), J. Comput. Graph. Stat., vol. 27, no. 4, pp. 861–871, 2018.

  * [2] L. Briceno-Arias and S.L. Rivera, [A Projected Primal–Dual Method for Solving Constrained Monotone Inclusions](https://link.springer.com/article/10.1007/s10957-018-1430-2?shared-article-renderer), J. Optim. Theory Appl., vol. 180, Issue 3, March 2019.

  * [3] P. L. Combettes and J.C. Pesquet, [Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators](https://arxiv.org/pdf/1107.0081.pdf), Set-Valued and Variational Analysis, vol. 20, pp. 307-330, 2012.

  * [4] P. L. Combettes and C. L. Müller, [Perspective M-estimation via proximal decomposition](https://arxiv.org/abs/1805.06098), Electronic Journal of Statistics, 2020, [Journal version](https://projecteuclid.org/euclid.ejs/1578452535)

  * [5] P. L. Combettes and C. L. Müller, [Regression models for compositional data: General log-contrast formulations, proximal optimization, and microbiome data applications](https://arxiv.org/abs/1903.01050), Statistics in Bioscience, 2020.

  * [6] A. Mishra and C. L. Müller, [Robust regression with compositional covariates](https://arxiv.org/abs/1909.04990), arXiv, 2019.

  * [7] S. Rosset and J. Zhu, [Piecewise linear regularized solution paths](https://projecteuclid.org/euclid.aos/1185303996), Ann. Stat., vol. 35, no. 3, pp. 1012–1030, 2007.

  * [8] J. Bien, X. Yan, L. Simpson, and C. L. Müller,   [Tree-Aggregated Predictive Modeling of Microbiome Data](https://www.biorxiv.org/content/10.1101/2020.09.01.277632v1), biorxiv, 2020.


pkgId: 3cfbddc401d9ae2770560a62676c89a1fb8bb4f53db7333174dfea6db56e0deb
name: python-c-pmat
arch: src
version: 0.9.0
epoch: 0
release: 1
url: https://github.com/IntegratedPathologyUnit-ICR/c-pmat
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c-pmat-0.9.0-1.src.rpm
checksum_type: sha256
summary:
  P-MAT - a napari-based workflow for scale-independent quantification of the extracellular matrix and its topological descriptors

description:
  c-pmat ***C***omputational ***P***reprocessing of extracellular ***mat***rix

  c-pmat is a interactive user interface, allowing user to easily preprocess, quality control (QC)
  and generate quantitative extracellular matrix features.

  Preprocess constitutes generating tiles and the corresponding metadata from the whole slide images.
  Quality control is dependant on annotations and this retains the tiles free of artifact and
  tiles belonging to the annotated regions and ensures enough tissue on the tile is present to
  perform downstream analysis.

  Through the usage of the step by step process, user will be able to preprocess,
  extract and quantify extracellular matrix features.




  In this section we will summarize the organization of directory structure to enable
  the end user extract the information needed to directly interact with the annotations which comes as a string
  or different names provided by the pathologists (generic).

  In the below example we have a first whole slide image with 6 regions of interest and they are named as
  ROI1, ROI2, .... ROI6 respectively and the second whole slide image with 3 regions of interest
  ROI1, ROI2, ROI3 respectively. These annotations are free hand polygon annotations drawn on the tissue by the
  pathologists to infer the changes in the extracellular matrix components with respect to the individual ROI and
  cater for inter-tumour and intra-tumour heterogeneity and its implication of features at slide and ROI levels.

  <p align="center">
    <img src="screenshot_images/dp1.PNG" width="850" title="WSI-1" />
    <img src="screenshot_images/dp2.PNG" width="850" title="WSI-2" />
  </p>


  Currently, we have the support for the annotations performed by the pathologists using Imagescope on the
  PSR stained whole slide images.

  Once you have annotations, it will retain the ROIs with respect to the individual slide automatically and extract the tiles
  corresponding to each ROI.

  Note: This code can be generically used for other brightfield images and extraction of the annotations performed on Imagescope.



  It also helps to restitch the ROI's at a lower resolution for sanity check so it can be further processed by TWOMBLI






  1. https://doi.org/10.25418/crick.26565343


pkgId: 01b303493528edac4b10a8c451d4aa9dd86d8ed33e1c8c557edeecb317281900
name: python-c-pmat
arch: src
version: 1.0.1
epoch: 0
release: 1
url: https://github.com/IntegratedPathologyUnit-ICR/c-pmat
rpm_license: None
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c-pmat-1.0.1-1.src.rpm
checksum_type: sha256
summary:
  c-pmat - a napari-based workflow for scale-independent quantification of the extracellular matrix and its topological descriptors

description:
  c-pmat ***C***omputational ***P***reprocessing of extracellular ***mat***rix

  c-pmat is a interactive user interface, allowing user to easily preprocess, quality control (QC)
  and generate quantitative extracellular matrix features.

  Preprocess constitutes generating tiles and the corresponding metadata from the whole slide images.
  Quality control is dependant on annotations and this retains the tiles free of artifact and
  tiles belonging to the annotated regions and ensures enough tissue on the tile is present to
  perform downstream analysis.

  Through the usage of the step by step process, user will be able to preprocess,
  extract and quantify extracellular matrix features.




  In this section we will summarize the organization of directory structure to enable
  the end user extract the information needed to directly interact with the annotations which comes as a string
  or different names provided by the pathologists (generic).

  In the below example we have a first whole slide image with 6 regions of interest and they are named as
  ROI1, ROI2, .... ROI6 respectively and the second whole slide image with 3 regions of interest
  ROI1, ROI2, ROI3 respectively. These annotations are free hand polygon annotations drawn on the tissue by the
  pathologists to infer the changes in the extracellular matrix components with respect to the individual ROI and
  cater for inter-tumour and intra-tumour heterogeneity and its implication of features at slide and ROI levels.

  <p align="center">
    <img src="screenshot_images/dp1.PNG" width="850" title="WSI-1" />
    <img src="screenshot_images/dp2.PNG" width="850" title="WSI-2" />
  </p>


  Currently, we have the support for the annotations performed by the pathologists using Imagescope on the
  PSR stained whole slide images.

  Once you have annotations, it will retain the ROIs with respect to the individual slide automatically and extract the tiles
  corresponding to each ROI.

  Note: This code can be generically used for other brightfield images and extraction of the annotations performed on Imagescope.



  It also helps to restitch the ROI's at a lower resolution for sanity check so it can be further processed by TWOMBLI






  1. https://doi.org/10.25418/crick.26565343


pkgId: 9e1fdd9a68f9f6e94674184e72b0d7940a18803a266b86a02c46638191d96948
name: python-c-print
arch: src
version: 1.0.1
epoch: 0
release: 1
summary: Small python package for printing text in different colours and typography.
url: https://github.com/SimonPfeifer/cprint
rpm_license: UNKNOWN
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c-print-1.0.1-1.src.rpm
checksum_type: sha256
description:
  Small python package for printing text in different colours and typography.



  pip install c-print



  You need to have a Python2.7+ or Python3+ installation with the following libraries:



  The `cprint()` function simply adds one keyword argument, `c`, to the python `print()` function. This extra keyword argument can change the output of `print()` to one colour and any number of given styles. Colours are given in lower case and styles in upper case. Any other keyword arguments are passed on to `print()`.

  | Character  |  Colour         |
  | ---------- | --------------- |
  | 'k'        | black           |
  | 'r'        | red             |
  | 'g'        | green           |
  | 'y'        | yellow          |
  | 'b'        | blue            |
  | 'm'        | magenta         |
  | 'c'        | cyan            |
  | 'w'        | white           |

  | Character  | Style           |
  | ---------- | --------------- |
  | 'B'        | bold            |
  | 'D'        | dark            |
  | 'I'        | italic          |
  | 'U'        | underline       |
  | 'F'        | flash           |
  | 'R'        | reverse colour  |
  | 'C'        | crossed through |

  Example:
  ```python
  from cprint import cprint
  cprint('Hello World!', c='rB')
  cprint('Hello World!', 'Monty Python', c='bUI', end='!')
  ```


pkgId: 71d85c163ca8685171a037f72fed571c1d26dbb6226563906f11724e849e8fcf
name: python-c2cciutils
arch: src
version: 1.6.23
epoch: 0
release: 1
summary: Common utilities for Camptocamp CI
url: https://github.com/camptocamp/c2cciutils
rpm_license: FreeBSD
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2cciutils-1.6.23-1.src.rpm
checksum_type: sha256
description:
  The main goals of C2C CI utils is to offer the commands and the workflows to have the following project structure:

  Have stabilization branches named by default `<major>.<minor>`.
  Have the release named by default `<major>.<minor>.<patch>`.

  With C2C CI utils you can publish a python package and a Docker image from the same repository.

  The default publishing are:

  - Push on the `<major>.<minor>` branch will publish Docker images.
  - Create the tag `<major>.<minor>.<patch>` will publish the Docker images, and the Python package.
  - Push on a feature branch (whatever other name) will publish the Docker images.
  - Delete a feature branch will delete the Docker images.
  - Push on the `master` branch will publish the Docker images with the master tag (Publishing a python package is also possible).
  - The version at the last line of the `SECURITY.md` of the `master` branch will be also published using the `latest` tag,
    this will respect the `tags` present in the configuration
  - In the `SECURITY.md` file of the `master` branch we can also add a column `Alternate Tag` to publish the Docker images with another tag,
    this will respect the `tags` present in the configuration (only for Docker).

  The Docker images are published on Docker Hub and GitHub Container Registry.

  You can run the publishing locally in dry-run mode:

  ```bash
  GITHUB_REF=... c2cciutils-publish --dry-run ...
  ```



  When we create a tag by default with the `changelog` workflow a release is created on GitHub, a changelog is
  generated and added to the release.



  The security is managed by the `c2cciutils-audit` command with Snyk, it will audit the dependencies of the project on every
  stabilization branches, if possible a pull request is created automatically to update the dependencies.

  When we publish a Docker image the generated image is monitored by Snyk, this means that Snyk will search
  for all the dependencies and send the list to the Snyk web site to be monitored.
  We also do a test of the image and log the result (This will never cause the build to fail).



  C2C CI utils will no more provide a tool to do a check of the project, this is replaced by `pre-commit`,
  a base configuration is provided in the example project.



  A workflow is provided to run the checks on the pull requests, it will run the `c2cciutils-pr-checks` command.

  - Check that the commit message and the pull request title start with a capital letter.
  - Check that there aren't any spelling issue in the commit message and in the pull request title.
  - Add a message to the pull request with a link to the JIRA issue if the pull request branch name starts with
    `[a-zA-Z]+-[0-9]+-` or end with `-[a-zA-Z]+-[0-9]+`.



  In the example project there is a basic Renovate configuration, it will update the dependencies of the project.
  There is also a workflow to add a review on the Renovate pull requests to make the auto merge working on
  repository that required a review.



  A workflow is provided to backport the pull requests on the stabilization branches, it will be triggered by
  adding a label named `backport <destination_branch>` on the pull request.



  GitHub will retain all the old workflows, so we need to delete them, the `delete-old-workflows-run`
  workflow will delete the workflows older than 500 days.



  C2cciutils make easier to have those workflows in a project:

  - `audit.yaml`: Audit the stabilization branches of the application against vulnerabilities in the python and node dependency
  - `auto-review.yaml`: Auto review the Renovate pull requests
  - `backport.yaml`: Trigger the backports (work with labels)
  - `clean.yaml`: Clean the Docker images related on a deleted feature branch
  - `main.yaml`: Main workflow especially with the c2cciutils-checks command
  - `changelog.yaml`: Generate the changelog and create the release on GitHub
  - `delete-old-workflows-run.yaml`: Delete the old workflows
  - `pr-checks.yaml`: Run the checks on the pull requests

  All the provided commands used in the workflow:

  - `c2cciutils`: some generic tools.
  - `c2cciutils-version`: Create a new version of the project.
  - `c2cciutils-checks`: Run the checks on the code (those checks don't need any project dependencies).
  - `c2cciutils-audit`: Do the audit, the main difference with checks is that it can change between runs on the same code.
  - `c2cciutils-publish`: Publish the project.
  - `c2cciutils-clean`: Delete Docker images on Docker Hub after corresponding branch have been deleted.



  The following utilities are provided:

  - `c2cciutils`: some generic tools.
  - `c2cciutils-download-applications`: Download the applications with version managed by Renovate, see below.
  - `c2cciutils-docker-logs`: Display the logs of the application in Docker (compose).
  - `c2cciutils-k8s-install`: Install a k3d / k3s cluster, see below.
  - `c2cciutils-k8s-logs`: Display the logs of the application in the k8s cluster, see below.
  - `c2cciutils-k8s-db`: Create a database in the k8s cluster, see below.
  - `c2cciutils-k8s-wait`: Wait that the application started correctly in the cluster, see below.
  - `c2cciutils-docker-versions-gen`: Generate the Docker package versions file (`ci/dpkg-versions.yaml`), see below.
  - `c2cciutils-pin-pipenv`: Display all the dependencies that's in the `Pipenv.lock` but not in the `Pipenv` to be able to pin them.
  - `c2cciutils-trigger-image-update`: Trigger the ArgoCD repository about image update on the CI (automatically done in the publishing).
  - `c2cciutils-google-calendar`: Tool to test the Google credentials for calendar API and refresh them if needed. See `c2cciutils-google-calendar -h` for more information.



  The content of `example-project` can be a good base for a new project.



  Requirements: the right version (>= 1.6) of `c2cciutils` should be installed with the `version` extra.

  To create a new minor version you just should run `c2cciutils-version --version=<version>`.

  You are welcome to run `c2cciutils-version --help` to see what's it's done.

  Note that it didn't create a tag, you should do it manually.

  To create a patch version you should just create tag.



  In the CI we need to have the following secrets::

  - `HAS_SECRETS` to be set to 'HAS_SECRETS', to avoid error errors from external
    pull requests, already set globally on Camptocamp organization.
  - `GOPASS_CI_GITHUB_TOKEN` and `CI_GPG_PRIVATE_KEY` required to initialize the gopass password store,
    the secrets exists in the Camptocamp organization but not shared on all project, then you should add
    your project to the shared list.



  Install it: `python3 -m pip install --user --requirement ci/requirements.txt`
  Dry run publish: `GITHUB_REF=... c2cciutils-publish --dry-run ...`



  You can get the current configuration with `c2cciutils --get-config`, the default configuration depends on your project.
  Note that it didn't contain the default defined the schema and visible in the [generated documentation](./config.md).

  You can override the configuration with the file `ci/config.yaml`.

  At the base of the configuration you have:

  - `version`: Contains some regular expressions to find the versions branches and tags, and to convert them into application versions.
  - `audit`: The audit configuration, see `c2cciutils/audit.py` for more information.
  - `publish`: The publishing configuration, see `c2cciutils/publish.py` for more information.

  Many actions can be disabled by setting the corresponding configuration part to `False`.



  The `SECURITY.md` file should contain the security policy of the repository, especially the end of
  support dates.

  For compatibility with `c2cciutils` it should contain an array with at least the columns
  `Version` and `Supported Until`. The `Version` column will contain the concerned version.
  The `Supported Until` will contain the date of end of support `dd/mm/yyyy`.
  It can also contain the following sentences:

  - `Unsupported`: no longer supported => no audit, no rebuild.
  - `Best effort`: the support is ended, it is still rebuilt and audited, but this can be stopped without any notice.
  - `To be defined`: not yet released or the date will be set related of another project release date (like for GeoMapFish).

  See also [GitHub Documentation](https://docs.github.com/en/github/managing-security-vulnerabilities/adding-a-security-policy-to-your-repository)



  The IDE should be configured as:

  - using `black` and `isort` without any arguments,
  - using the `editorconfig` configuration.



  - Recommend extensions to work well with c2cciutils:
    - [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) And use EditorConfig
    - [shell-format](https://marketplace.visualstudio.com/items?itemName=foxundermoon.shell-format) With the configuration
      `"shellformat.flag": "-bn"`.
    - [Better TOML](https://marketplace.visualstudio.com/items?itemName=bodil.prettier-toml)
  - Other recommend extensions:
    - [hadolint](https://marketplace.visualstudio.com/items?itemName=exiasr.hadolint)
    - [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker)

  Select a formatter:

  - `CTRL+MAJ+P`
  - Format document With...
  - Configure Default Formatter...
  - Select the formatter





  The config is like this:

  ```yaml
  versions:
    
    
    
  ```

  It we have a `setup.py` file, we will be in legacy mode:
  When publishing, the version computed from arguments or `GITHUB_REF` is put in environment variable `VERSION`, thus you should use it in `setup.py`, example:

  ```python
  VERSION = os.environ.get("VERSION", "1.0.0")
  ```

  Also we consider that we use `poetry` with [poetry-dynamic-versioning](https://pypi.org/project/poetry-dynamic-versioning/) to manage the version, and [poetry-plugin-tweak-dependencies-version](https://pypi.org/project/poetry-plugin-tweak-dependencies-version/) to manage the dependencies versions.

  Example of configuration:

  ```toml
  [tool.poetry-dynamic-versioning]
  enable = true
  vcs = "git"
  pattern = "^(?P<base>\\d+(\\.\\d+)*)"
  format-jinja = """
  { if env.get("VERSION_TYPE") == "version_branch" -%}
  {{serialize_pep440(bump_version(base, 1 if env.get("IS_MASTER") == "TRUE" else 2), dev=distance)}}
  { elif distance == 0 -%}
  {{serialize_pep440(base)}}
  { else -%}
  {{serialize_pep440(bump_version(base), dev=distance)}}
  { endif -%}
  """

  ```

  Note that we can access to the environment variables `VERSION`,`VERSION_TYPE` and `IS_MASTER`.

  Then by default:

  - Tag with `1.2.3` => release `1.2.3`
  - Commit on feature branch just do a validation
  - Commit on `master` branch after the tag 1.3.0 => release `1.4.0.dev1`
  - Commit on `1.3` branch after the tag 1.3.0 => release `1.3.1.dev1`

  To make it working in the `Dockerfile` you should have in the `poetry` stage:

  ```Dockerfile
  ENV POETRY_DYNAMIC_VERSIONING_BYPASS=dev
  RUN poetry export --extras=checks --extras=publish --extras=audit --output=requirements.txt \
      && poetry export --with=dev --output=requirements-dev.txt
  ```

  And in the `run` stage

  ```Dockerfile
  ARG VERSION=dev
  RUN --mount=type=cache,target=/root/.cache \
      POETRY_DYNAMIC_VERSIONING_BYPASS=${VERSION} python3 -m pip install --disable-pip-version-check --no-deps --editable=.
  ```

  And in the `Makefile`:

  ```Makefile
  VERSION = $(strip $(shell poetry version --short))

  .PHONY: build
  build: ## Build the Docker images
      docker build --build-arg=VERSION=$(VERSION) --tag=$(GITHUB_REPOSITORY) .
  ```



  The config is like this:

  ```yaml
  latest: True
  images:
    - # The base name of the image we want to publish
      name:
  repository:
    <internal_name>:
      
      server:
      
      
      version:
      
      
      
      
      tags:
      
      
      group:
  ```

  By default, the last line of the `SECURITY.md` file will be published (`docker`) with the tag
  `latest`. Set `latest` to `False` to disable it.

  With the `c2cciutils-clean` the images on Docker hub for `feature_branch` will be removed on branch removing.



  In case some executables or applications from GitHub releases or any other URLs are required on the CI host
  and are not handled by any dependency manager, we provide a set of tools to install them and manage upgrades
  through Renovate.

  Create an application file (e.-g. `applications.yaml`) with:

  ```yaml



  <organization>/<project>:
    get-file-name: <file name present in the release>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: # The command you want to run after the file is downloaded
      - - chmod # To be executable (usually required)
        - +x
        - <to-file-name>
      - - <to-file-name> # Print the version of the application
        - --version

  <organization>/<project>:
    get-file-name: <file name present in the release>
    type: tar
    tar-file-name: <The file name available in the tar file>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: [...] # The command you want to run after the file is downloaded

  <application reference name>:
    url-pattern: <The URL used to download the application>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: [...] # The command you want to run after the file is downloaded
  ```

  In the attributes `url-pattern`, `get-file-name` you can use the following variables:

  - `{version}`: The version of the application present in the version file.
  - `{version_quote}`: The URL encoded version.
  - `{short_version}`: The version without the `v` prefix.

  The `applications-versions.yaml` file is a map of applications and their versions.

  Add in your Renovate configuration:

  ```json5
    regexManagers: [
      {
        fileMatch: ['^applications-versions.yaml$'],
        matchStrings: [
          '(?<depName>[^\\s]+): (?<currentValue>[^\\s]+) # (?<datasource>[^\\s]+)',
        ],
      },
    ],
  ```

  Now you need to call `c2cciutils-download-applications --applications-file=applications.yaml --versions-file=applications-version.yaml`
  to install required applications on CI host before using them (an already installed application is installed only if needed).



  Run the command `c2cciutils-docker-versions-gen camptocamp/image[:tag]` to generate a file that is a kind of package lock of the Debian packages in the file `ci/dpkg-versions.yaml`.

  Add in your renovate configuration:

  ```javascript
    regexManagers: [
      {
        fileMatch: ['^ci/dpkg-versions.yaml$'],
        matchStrings: [" *(?<depName>[^'\\s]+): '?(?<currentValue>[^'\\s/]*[0-9][^'\\s/]*)'?"],
        datasourceTemplate: 'repology',
        versioningTemplate: 'loose',
      },
    ],
  ```

  When a new version of a Debian package will be available:

  - Renovate will automatically open a pull request to update the file `ci/dpkg-versions.yaml`.
  - And the continuous integration will build a new fresh Docker image with latest versions of all Debian packages.



  C2cciutils provide some commands for Kubernetes.

  You can define a workflow like that:

  ```yaml
  - name: Install k3s/k3d (Kubernetes cluster)
    run: c2cciutils-k8s-install

  - name: Create a database to do the tests
    run: c2cciutils-k8s-db --script=<my_script>.sql

  - name: Install the application in the Kubernetes cluster
    run: kubectl apply -f <my_application>.yaml

  - name: Wait that the application is ready
    run: c2cciutils-k8s-wait
  - name: Print the application status and logs
    run: c2cciutils-k8s-logs
    if: always()

  - name: Uninstall the application
    run: kubectl delete -f <my_application>.yaml || true

  - name: Cleanup the database
    run: c2cciutils-k8s-db --cleanup
  ```

  `c2cciutils-k8s-install` can be configured in the `ci/config.yaml` file, in section `k8s/k3d/install-commands`, default is:

  ```yaml
  - - k3d
      cluster
      create
      test-cluster
      --no-lb
      --no-rollback
  ```

  See also: [K3d cluster create documentation](https://k3d.io/v4.4.8/usage/commands/k3d_cluster_create/).

  `c2cciutils-k8s-db` can be configured in the `ci/config.yaml` file, in section `k8s/db/chart-options`, default is:

  ```yaml
  persistence.enabled: 'false'
  tls.enabled: 'true'
  tls.autoGenerated: 'true'
  postgresqlPassword: mySuperTestingPassword
  volumePermissions.enabled: 'true'
  ```

  See also: [Parameters documentations](https://github.com/bitnami/charts/tree/master/bitnami/postgresql#parameters).



  Install the pre-commit hooks:

  ```bash
  pip install pre-commit
  pre-commit install --allow-missing-config
  ```


pkgId: 0c93d6a68bdcb1f36c4182512770d55a982a447fa0ea129c634215f297ae95e2
name: python-c2cciutils
arch: src
version: 1.7.1
epoch: 0
release: 1
summary: Common utilities for Camptocamp CI
url: https://github.com/camptocamp/c2cciutils
rpm_license: FreeBSD
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2cciutils-1.7.1-1.src.rpm
checksum_type: sha256
description:
  The main goals of C2C CI utils is to offer the commands to publish the project,
  see the [documentation](https://github.com/camptocamp/c2cciutils/wiki/Publishing).



  When we create a tag by default with the `changelog` workflow a release is created on GitHub, a changelog is
  generated and added to the release.



  C2C CI utils will no more provide a tool to do a check of the project, this is replaced by `pre-commit`,
  a base configuration is provided in the example project.



  A workflow is provided to run the checks on the pull requests, it will run the `c2cciutils-pr-checks` command.

  - Check that the commit message and the pull request title start with a capital letter.
  - Check that there aren't any spelling issue in the commit message and in the pull request title.
  - Add a message to the pull request with a link to the JIRA issue if the pull request branch name starts with
    `[a-zA-Z]+-[0-9]+-` or end with `-[a-zA-Z]+-[0-9]+`.



  In the example project there is a basic Renovate configuration, it will update the dependencies of the project.
  There is also a workflow to add a review on the Renovate pull requests to make the auto merge working on
  repository that required a review.



  A workflow is provided to backport the pull requests on the stabilization branches, it will be triggered by
  adding a label named `backport <destination_branch>` on the pull request.



  GitHub will retain all the old workflows, so we need to delete them, the `delete-old-workflows-run`
  workflow will delete the workflows older than 500 days.



  C2cciutils make easier to have those workflows in a project:

  - `auto-review.yaml`: Auto review the Renovate pull requests
  - `backport.yaml`: Trigger the backports (work with labels)
  - `clean.yaml`: Clean the Docker images related on a deleted feature branch
  - `main.yaml`: Main workflow especially with the c2cciutils-checks command

  All the provided commands used in the workflow:

  - `c2cciutils`: some generic tools.
  - `c2cciutils-version`: Create a new version of the project.
  - `c2cciutils-env`: Print some environment information.
  - `c2cciutils-publish`: Publish the project.
  - `c2cciutils-clean`: Delete Docker images on Docker Hub after corresponding branch have been deleted.



  The following utilities are provided:

  - `c2cciutils`: some generic tools.
  - `c2cciutils-download-applications`: Download the applications with version managed by Renovate, see below.
  - `c2cciutils-docker-logs`: Display the logs of the application in Docker (compose).
  - `c2cciutils-k8s-install`: Install a k3d / k3s cluster, see below.
  - `c2cciutils-k8s-logs`: Display the logs of the application in the k8s cluster, see below.
  - `c2cciutils-k8s-db`: Create a database in the k8s cluster, see below.
  - `c2cciutils-k8s-wait`: Wait that the application started correctly in the cluster, see below.
  - `c2cciutils-docker-versions-gen`: Generate the Docker package versions file (`ci/dpkg-versions.yaml`), see below.
  - `c2cciutils-pin-pipenv`: Display all the dependencies that's in the `Pipenv.lock` but not in the `Pipenv` to be able to pin them.
  - `c2cciutils-trigger-image-update`: Trigger the ArgoCD repository about image update on the CI (automatically done in the publishing).
  - `c2cciutils-google-calendar`: Tool to test the Google credentials for calendar API and refresh them if needed. See `c2cciutils-google-calendar -h` for more information.



  The content of `example-project` can be a good base for a new project.



  Requirements: the right version (>= 1.6) of `c2cciutils` should be installed with the `version` extra.

  To create a new minor version you just should run `c2cciutils-version --version=<version>`.

  You are welcome to run `c2cciutils-version --help` to see what's it's done.

  Note that it didn't create a tag, you should do it manually.

  To create a patch version you should just create tag.



  In the CI we need to have the following secrets::

  - `HAS_SECRETS` to be set to 'HAS_SECRETS', to avoid error errors from external
    pull requests, already set globally on Camptocamp organization.
  - `GOPASS_CI_GITHUB_TOKEN` and `CI_GPG_PRIVATE_KEY` required to initialize the gopass password store,
    the secrets exists in the Camptocamp organization but not shared on all project, then you should add
    your project to the shared list.



  Install it: `python3 -m pip install --user --requirement ci/requirements.txt`
  Dry run publish: `GITHUB_REF=... c2cciutils-publish --dry-run ...`



  You can get the current configuration with `c2cciutils --get-config`, the default configuration depends on your project.
  Note that it didn't contain the default defined the schema and visible in the [generated documentation](./config.md).

  You can override the configuration with the file `ci/config.yaml`.

  At the base of the configuration you have:

  - `version`: Contains some regular expressions to find the versions branches and tags, and to convert them into application versions.
  - `publish`: The publishing configuration, see `c2cciutils/publish.py` for more information.

  Many actions can be disabled by setting the corresponding configuration part to `False`.



  The `SECURITY.md` file should contain the security policy of the repository, especially the end of
  support dates.

  For compatibility with `c2cciutils` it should contain an array with at least the columns
  `Version` and `Supported Until`. The `Version` column will contain the concerned version.
  The `Supported Until` will contain the date of end of support `dd/mm/yyyy`.
  It can also contain the following sentences:

  - `Unsupported`: no longer supported => no audit, no rebuild.
  - `Best effort`: the support is ended, it is still rebuilt and audited, but this can be stopped without any notice.
  - `To be defined`: not yet released or the date will be set related of another project release date (like for GeoMapFish).

  See also [GitHub Documentation](https://docs.github.com/en/github/managing-security-vulnerabilities/adding-a-security-policy-to-your-repository)



  The IDE should be configured as:

  - using `black` and `isort` without any arguments,
  - using the `editorconfig` configuration.



  - Recommend extensions to work well with c2cciutils:
    - [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) And use EditorConfig
    - [shell-format](https://marketplace.visualstudio.com/items?itemName=foxundermoon.shell-format) With the configuration
      `"shellformat.flag": "-bn"`.
    - [Better TOML](https://marketplace.visualstudio.com/items?itemName=bodil.prettier-toml)
  - Other recommend extensions:
    - [hadolint](https://marketplace.visualstudio.com/items?itemName=exiasr.hadolint)
    - [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker)

  Select a formatter:

  - `CTRL+MAJ+P`
  - Format document With...
  - Configure Default Formatter...
  - Select the formatter





  The config is like this:

  ```yaml
  versions:
    
    
    
  ```

  It we have a `setup.py` file, we will be in legacy mode:
  When publishing, the version computed from arguments or `GITHUB_REF` is put in environment variable `VERSION`, thus you should use it in `setup.py`, example:

  ```python
  VERSION = os.environ.get("VERSION", "1.0.0")
  ```

  Also we consider that we use `poetry` with [poetry-dynamic-versioning](https://pypi.org/project/poetry-dynamic-versioning/) to manage the version, and [poetry-plugin-tweak-dependencies-version](https://pypi.org/project/poetry-plugin-tweak-dependencies-version/) to manage the dependencies versions.

  Example of configuration:

  ```toml
  [tool.poetry-dynamic-versioning]
  enable = true
  vcs = "git"
  pattern = "^(?P<base>\\d+(\\.\\d+)*)"
  format-jinja = """
  { if env.get("VERSION_TYPE") == "version_branch" -%}
  {{serialize_pep440(bump_version(base, 1 if env.get("IS_MASTER") == "TRUE" else 2), dev=distance)}}
  { elif distance == 0 -%}
  {{serialize_pep440(base)}}
  { else -%}
  {{serialize_pep440(bump_version(base), dev=distance)}}
  { endif -%}
  """

  ```

  Note that we can access to the environment variables `VERSION`,`VERSION_TYPE` and `IS_MASTER`.

  Then by default:

  - Tag with `1.2.3` => release `1.2.3`
  - Commit on feature branch just do a validation
  - Commit on `master` branch after the tag 1.3.0 => release `1.4.0.dev1`
  - Commit on `1.3` branch after the tag 1.3.0 => release `1.3.1.dev1`



  If the file `~/.pypirc` exists we consider that we ar already logged in also
  we will do the login with the `pypi` server with OpenID Connect (OIDC).

  The OIDC login is recommended because it didn't needs any additional secrets,
  but it need some configuration on pypi in the package,
  see the [GitHub Documentation](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-pypi#adding-the-identity-provider-to-pypi).



  To make it working in the `Dockerfile` you should have in the `poetry` stage:

  ```Dockerfile
  ENV POETRY_DYNAMIC_VERSIONING_BYPASS=dev
  RUN poetry export --extras=checks --extras=publish --output=requirements.txt \
      && poetry export --with=dev --output=requirements-dev.txt
  ```

  And in the `run` stage

  ```Dockerfile
  ARG VERSION=dev
  RUN --mount=type=cache,target=/root/.cache \
      POETRY_DYNAMIC_VERSIONING_BYPASS=${VERSION} python3 -m pip install --disable-pip-version-check --no-deps --editable=.
  ```

  And in the `Makefile`:

  ```Makefile
  VERSION = $(strip $(shell poetry version --short))

  .PHONY: build
  build: ## Build the Docker images
      docker build --build-arg=VERSION=$(VERSION) --tag=$(GITHUB_REPOSITORY) .
  ```



  The config is like this:

  ```yaml
  latest: True
  images:
    - # The base name of the image we want to publish
      name:
  repository:
    <internal_name>:
      
      server:
      
      
      version:
      
      
      
      
      tags:
      
      
      group:
  ```

  By default, the last line of the `SECURITY.md` file will be published (`docker`) with the tag
  `latest`. Set `latest` to `False` to disable it.

  With the `c2cciutils-clean` the images on Docker hub for `feature_branch` will be removed on branch removing.



  In case some executables or applications from GitHub releases or any other URLs are required on the CI host
  and are not handled by any dependency manager, we provide a set of tools to install them and manage upgrades
  through Renovate.

  Create an application file (e.-g. `applications.yaml`) with:

  ```yaml



  <organization>/<project>:
    get-file-name: <file name present in the release>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: # The command you want to run after the file is downloaded
      - - chmod # To be executable (usually required)
        - +x
        - <to-file-name>
      - - <to-file-name> # Print the version of the application
        - --version

  <organization>/<project>:
    get-file-name: <file name present in the release>
    type: tar
    tar-file-name: <The file name available in the tar file>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: [...] # The command you want to run after the file is downloaded

  <application reference name>:
    url-pattern: <The URL used to download the application>
    to-file-name: <The file name you want to create in ~/.local/bin>
    finish-command: [...] # The command you want to run after the file is downloaded
  ```

  In the attributes `url-pattern`, `get-file-name` you can use the following variables:

  - `{version}`: The version of the application present in the version file.
  - `{version_quote}`: The URL encoded version.
  - `{short_version}`: The version without the `v` prefix.

  The `applications-versions.yaml` file is a map of applications and their versions.

  Add in your Renovate configuration:

  ```json5
    regexManagers: [
      {
        fileMatch: ['^applications-versions.yaml$'],
        matchStrings: [
          '(?<depName>[^\\s]+): (?<currentValue>[^\\s]+) # (?<datasource>[^\\s]+)',
        ],
      },
    ],
  ```

  Now you need to call `c2cciutils-download-applications --applications-file=applications.yaml --versions-file=applications-version.yaml`
  to install required applications on CI host before using them (an already installed application is installed only if needed).



  Run the command `c2cciutils-docker-versions-gen camptocamp/image[:tag]` to generate a file that is a kind of package lock of the Debian packages in the file `ci/dpkg-versions.yaml`.

  Add in your renovate configuration:

  ```javascript
    regexManagers: [
      {
        fileMatch: ['^ci/dpkg-versions.yaml$'],
        matchStrings: [" *(?<depName>[^'\\s]+): '?(?<currentValue>[^'\\s/]*[0-9][^'\\s/]*)'?"],
        datasourceTemplate: 'repology',
        versioningTemplate: 'loose',
      },
    ],
  ```

  When a new version of a Debian package will be available:

  - Renovate will automatically open a pull request to update the file `ci/dpkg-versions.yaml`.
  - And the continuous integration will build a new fresh Docker image with latest versions of all Debian packages.



  C2cciutils provide some commands for Kubernetes.

  You can define a workflow like that:

  ```yaml
  - name: Install k3s/k3d (Kubernetes cluster)
    run: c2cciutils-k8s-install

  - name: Create a database to do the tests
    run: c2cciutils-k8s-db --script=<my_script>.sql

  - name: Install the application in the Kubernetes cluster
    run: kubectl apply -f <my_application>.yaml

  - name: Wait that the application is ready
    run: c2cciutils-k8s-wait
  - name: Print the application status and logs
    run: c2cciutils-k8s-logs
    if: always()

  - name: Uninstall the application
    run: kubectl delete -f <my_application>.yaml || true

  - name: Cleanup the database
    run: c2cciutils-k8s-db --cleanup
  ```

  `c2cciutils-k8s-install` can be configured in the `ci/config.yaml` file, in section `k8s/k3d/install-commands`, default is:

  ```yaml
  - - k3d
      cluster
      create
      test-cluster
      --no-lb
      --no-rollback
  ```

  See also: [K3d cluster create documentation](https://k3d.io/v4.4.8/usage/commands/k3d_cluster_create/).

  `c2cciutils-k8s-db` can be configured in the `ci/config.yaml` file, in section `k8s/db/chart-options`, default is:

  ```yaml
  persistence.enabled: 'false'
  tls.enabled: 'true'
  tls.autoGenerated: 'true'
  postgresqlPassword: mySuperTestingPassword
  volumePermissions.enabled: 'true'
  ```

  See also: [Parameters documentations](https://github.com/bitnami/charts/tree/master/bitnami/postgresql#parameters).



  Install the pre-commit hooks:

  ```bash
  pip install pre-commit
  pre-commit install --allow-missing-config
  ```


pkgId: a87c563a0784e6f0b0beb5c4c2dc1d9f2f735baf6fc29c1baa3dd107d4c08850
name: python-c2client
arch: src
version: 0.23
epoch: 0
release: 1
summary: CROC Cloud Platform - API Client
url: https://github.com/c2devel/c2-client
rpm_license: GPL3
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2client-0.23-1.src.rpm
checksum_type: sha256
description:
  Simple command-line utility for sending custom requests to CROC Cloud platform.
  **Warning: this utility is not intended for automation cases.
  Use https://github.com/c2devel/boto3.git and python scripts instead.**


pkgId: 721c8a8e0fb3f14096524b3d71fcd6849799d5c8c2ba83c14afb2386ccefaac8
name: python-c2cwsgiutils
arch: src
version: 6.0.8
epoch: 0
release: 1
summary: Common utilities for Camptocamp WSGI applications
url: https://github.com/camptocamp/c2cwsgiutils
rpm_license: BSD-2-Clause
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2cwsgiutils-6.0.8-1.src.rpm
checksum_type: sha256
description:
  This is a Python 3 library (>=3.5) providing common tools for Camptocamp WSGI
  applications:

  - Provide prometheus metrics
  - Allow to use a master/slave PostgresQL configuration
  - Logging handler for CEE/UDP logs
    - An optional view to change runtime the log levels
  - SQL profiler to debug DB performance problems, disabled by default. Warning, it will slow down everything.
  - A view to get the version information about the application and the installed packages
  - A framework for implementing a health_check service
  - Error handlers to send JSON messages to the client in case of error
  - A cornice service drop in replacement for setting up CORS

  Also provide tools for writing acceptance tests:

  - A class that can be used from a py.test fixture to control a composition
  - A class that can be used from a py.text fixture to test a REST API

  As an example on how to use it in an application provided by a Docker image, you can look at the
  test application in [acceptance_tests/app](acceptance_tests/app).
  To see how to test such an application, look at [acceptance_tests/tests](acceptance_tests/tests).





  Here we didn't do a minimal install of c2cwsgiutils, but be put in place everything needed to
  monitor the application in integration and production environment.

  The library is available in PYPI:
  [https://pypi.python.org/pypi/c2cwsgiutils](https://pypi.python.org/pypi/c2cwsgiutils)

  Copy and adapt these template configuration file into your project:

  - [production.ini](acceptance_tests/app/production.ini);
  - [gunicorn.conf.py](acceptance_tests/app/gunicorn.conf.py).
    Then replace `c2cwsgiutils_app` by your package name.

  You should install `c2cwsgiutils` with the tool you use to manage your pip dependencies.

  In the `Dockerfile` you should add the following lines:

  ```dockerfile

  RUN c2cwsgiutils-genversion $(git rev-parse HEAD)

  CMD ["gunicorn", "--paste=/app/production.ini"]


  ENV \
    DEVELOPMENT=0 \
    SQLALCHEMY_POOL_RECYCLE=30 \
    SQLALCHEMY_POOL_SIZE=5 \
    SQLALCHEMY_MAX_OVERFLOW=25 \
    SQLALCHEMY_SLAVE_POOL_RECYCLE=30 \
    SQLALCHEMY_SLAVE_POOL_SIZE=5 \
    SQLALCHEMY_SLAVE_MAX_OVERFLOW=25\
    LOG_TYPE=console \
    OTHER_LOG_LEVEL=WARNING \
    GUNICORN_LOG_LEVEL=WARNING \
    SQL_LOG_LEVEL=WARNING \
    C2CWSGIUTILS_LOG_LEVEL=WARNING \
    LOG_LEVEL=INFO
  ```

  Add in your `main` function.

  ```python
  config.include("c2cwsgiutils.pyramid")
  dbsession = c2cwsgiutils.db.init(config, "sqlalchemy", "sqlalchemy_slave")

  config.scan(...)


  health_check = c2cwsgiutils.health_check.HealthCheck(config)
  health_check.add_db_session_check(dbsession)
  health_check.add_alembic_check(dbsession, "/app/alembic.ini", 1)
  ```

  The related environment variables:

  - `DEVELOPMENT`: set to `1` to enable the development mode, default is `0`.
  - `SQLALCHEMY_URL`: SQL alchemy URL, like `postgresql://user:password@host:port/dbname`.
  - `SQLALCHEMY_POOL_RECYCLE`: The SQL alchemy pool recycle, default is `30`.
  - `SQLALCHEMY_POOL_SIZE`: The SQL alchemy pool size, default is `5`.
  - `SQLALCHEMY_MAX_OVERFLOW`: SQL alchemy max overflow, default is `25`.
  - `SQLALCHEMY_SLAVE_URL`: The SQL alchemy slave (read only) URL, like `postgresql://user:password@host:port/dbname`.
  - `SQLALCHEMY_SLAVE_POOL_RECYCLE`: The SQL alchemy slave pool recycle, default is `30`.
  - `SQLALCHEMY_SLAVE_POOL_SIZE`: The SQL alchemy slave pool size, default is `5`.
  - `SQLALCHEMY_SLAVE_MAX_OVERFLOW`: The SQL alchemy slave max overflow, default is `25`.
  - `GUNICORN_WORKERS`: The number of workers, default is `2`.
  - `GUNICORN_THREADS`: The number of threads per worker, default is `10`.
  - `LOG_TYPE`: The types of logs, default is `console`, should be `json` on kubernetes to work well with
    [elk](https://www.elastic.co/fr/what-is/elk-stack).
  - `LOG_LEVEL`: The application log level, default is `INFO`.
  - `SQL_LOG_LEVEL`: The SQL query log level, `WARNING`: no logs, `INFO`: logs the queries,
    `DEBUG` also logs the results, default is `WARNING`.
  - `GUNICORN_ERROR_LOG_LEVEL`: The Gunicorn error log level, default is `WARNING`.
  - `C2CWSGIUTILS_CONFIG`: The fallback ini file to use by gunicorn, default is `production.ini`.
  - `C2CWSGIUTILS_LOG_LEVEL`: The c2c WSGI utils log level, default is `WARNING`.
  - `OTHER_LOG_LEVEL`: The log level for all the other logger, default is `WARNING`.

  Those environment variables can be useful for investigation on production environments.



  Or (deprecated) as a base Docker image:
  [camptocamp/c2cwsgiutils:release_5](https://hub.docker.com/r/camptocamp/c2cwsgiutils/) or
  [ghcr.io/camptocamp/c2cwsgiutils:release_5](https://github.com/orgs/camptocamp/packages/container/package/c2cwsgiutils)

  If you need an image with a smaller foot print, use the tags prefixed with `-light`. Those are without
  GDAL and without the build tools.

  We deprecate the Docker image because:

  - The project wants to choose the base image.
  - The project pin different versions of the dependencies.



  In general, configuration can be done both with environment variables (taken first) or with entries in the
  `production.ini` file.

  You can configure the base URL for accessing the views provided by c2cwsgiutils with an environment variable
  named `C2C_BASE_PATH` or in the `production.ini` file with a property named `c2c.base_path`.

  A few REST APIs are added and can be seen with this URL:
  `{C2C_BASE_PATH}`.

  Some APIs are protected by a secret. This secret is specified in the `C2C_SECRET` variable or `c2c.secret`
  property. It is either passed as the `secret` query parameter or the `X-API-Key` header. Once
  accessed with a good secret, a cookie is stored and the secret can be omitted.

  An alternative of using `C2C_SECRET` is to use an authentication on GitHub,
  [create the GitHub application](https://github.com/settings/applications/new).

  Configure the json renderers with the `C2C_JSON_PRETTY_PRINT` and `C2C_JSON_SORT_KEYS` environment
  variables or `c2c.json.pretty_print`and `c2c.json.sort_keys` properties. Default is `false`.

  Then it will redirect the user to the github authentication form if not already authenticated
  (using `C2C_AUTH_GITHUB_CLIENT_ID`, `C2C_AUTH_GITHUB_CLIENT_SECRET` and `C2C_AUTH_GITHUB_SCOPE`).

  Then we will check if the user is allowed to access to the application, for that we check
  if the user has enough right on a GitHub repository (using `C2C_AUTH_GITHUB_REPOSITORY`
  and `C2C_AUTH_GITHUB_REPOSITORY_ACCESS_TYPE`).

  Finally we store the session information in an encrypted cookie (using `C2C_AUTH_SECRET`
  and `C2C_AUTH_COOKIE`).

  Configuration details:

  Using the environment variable `C2C_AUTH_GITHUB_REPOSITORY` or the config key `c2c.auth.github.repository`
  to define the related GitHub repository (required).

  Using the environment variable `C2C_AUTH_GITHUB_ACCESS_TYPE` or the config key
  `c2c.auth.github.access_type` to define the type of required access can be `pull`, `push` or
  `admin` (default is `push`)

  Using the environment variable `C2C_AUTH_GITHUB_CLIENT_ID` or the config key `c2c.auth.github.client_id` to
  define the GitHub application ID (required)

  Using the environment variable `C2C_AUTH_GITHUB_CLIENT_SECRET` or the config key
  `c2c.auth.github.client_secret` to define the GitHub application secret (required)

  Using the environment variable `C2C_AUTH_GITHUB_SCOPE` or the config key `c2c.auth.github.scope` to define
  the GitHub scope (default is `repo`), see [GitHub documentation](https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/)

  Using the environment variable `C2C_AUTH_GITHUB_SECRET` or the config key `c2c.auth.github.auth.secret` to
  define the used secret for JWD encryption (required, with a length at least of 16)

  Using the environment variable `C2C_AUTH_GITHUB_COOKIE` or the config key `c2c.auth.github.auth.cookie` to
  define the used cookie name (default is `c2c-auth-jwt`)

  Using the environment variable `C2C_AUTH_GITHUB_AUTH_URL` or the config key `c2c.auth.github.auth_url` to
  define the GitHub auth URL (default is `https://github.com/login/oauth/authorize`)

  Using the environment variable `C2C_AUTH_GITHUB_TOKEN_URL` or the config key `c2c.auth.github.token_url` to
  define the GitHub auth URL (default is `https://github.com/login/oauth/access_token`)

  Using the environment variable `C2C_AUTH_GITHUB_USER_URL` or the config key `c2c.auth.github.user_url` to
  define the GitHub auth URL (default is `https://api.github.com/user`)

  Using the environment variable `C2C_AUTH_GITHUB_REPO_URL` or the config key `c2c.auth.github.repo_url` to
  define the GitHub auth URL (default is `https://api.github.com/repo`)

  Using the environment variable `C2C_AUTH_GITHUB_PROXY_URL` or the config key `c2c.auth.github.auth.proxy_url` to
  define a redirect proxy between GitHub and our application to be able to share an OAuth2 application on GitHub (default is no proxy).
  Made to work with [this proxy](https://github.com/camptocamp/redirect/).

  Using the environment variable `C2C_USE_SESSION` or the config key `c2c.use_session` to
  define if we use a session. Currently, we can use the session to store a state, used to prevent CSRF, during OAuth2 login (default is `false`)



  All the environment variables are usable in the configuration file using stuff like `s`.

  To enable most of the features of c2cwsgiutils, you need to add this line to your WSGI main:

  ```python
  import c2cwsgiutils.pyramid
  config.include(c2cwsgiutils.pyramid.includeme)
  ```

  Error catching views will be put in place to return errors as JSON.

  A custom loader is provided to run pyramid scripts against configuration files containing environment variables:

  ```shell
  proutes c2c://production.ini      # relative path
  proutes c2c:///app/production.ini # absolute path
  ```

  A filter is automatically installed to handle the HTTP headers set by common proxies and have correct values
  in the request object (`request.client_addr`, for example). This filter is equivalent to what the
  `PasteDeploy#prefix` (minus the prefix part) does, but supports newer headers as well (`Forwarded`).
  If you need to prefix your routes, you can use the `route_prefix` parameter of the `Configurator` constructor.



  Two new logging backends are provided:

  - `c2cwsgiutils.pyramid_logging.PyramidCeeSysLogHandler`: to send @cee formatted logs to syslog through UDP.
  - `c2cwsgiutils.pyramid_logging.JsonLogHandler`: to output (on stdout or stderr) JSON formatted logs.

  Look at the logging configuration part of
  [acceptance_tests/app/production.ini](acceptance_tests/app/production.ini) for paste and commands line.

  The logging configuration is imported automatically by gunicorn, it is possible to visualize the dict config by setting the environment variable `DEBUG_LOGCONFIG=1`.

  You can enable a view to configure the logging level on a live system using the `C2C_LOG_VIEW_ENABLED` environment
  variable. Then, the current status of a logger can be queried with a GET on
  `{C2C_BASE_PATH}/logging/level?secret={C2C_SECRET}&name={logger_name}` and can be changed with
  `{C2C_BASE_PATH}/logging/level?secret={C2C_SECRET}&name={logger_name}&level={level}`. Overrides are stored in
  Redis, if `C2C_REDIS_URL` (`c2c.redis_url`) or `C2C_REDIS_SENTINELS` is configured.



  You can enable a view to force usage of the slave engine using the `C2C_DB_MAINTENANCE_VIEW_ENABLED` environment
  variable. Then, the database can be made "readonly" with
  `{C2C_BASE_PATH}/db/maintenance?secret={C2C_SECRET}&readonly=true`.
  The current state is stored in Redis, if `C2C_REDIS_URL` (`c2c.redis_url`) or `C2C_REDIS_SENTINELS` is configured.



  In order to follow the logs generated by a request across all the services (think separate processes),
  c2cwsgiutils tries to flag averything with a request ID. This field can come from the input as request headers
  (`X-Request-ID`, `X-Correlation-ID`, `Request-ID` or `X-Varnish`) or will default to a UUID. You can add an
  additional request header as source for that by defining the `C2C_REQUEST_ID_HEADER` environment variable
  (`c2c.request_id_header`).

  In JSON logging formats, a `request_id` field is automatically added.

  You can enable (disabled by default since it can have a cost) the flagging of the SQL requests as well by
  setting the C2C_SQL_REQUEST_ID environment variable (or c2c.sql_request_id in the .ini file). This will use
  the application name to pass along the request id. If you do that, you must include the application name in
  the PostgreSQL logs by setting `log_line_prefix` to something like `"%a "` (don't forget the space).

  Then, in your application, it is recommended to transmit the request ID to the external REST APIs. Use
  the `X-Request-ID` HTTP header, for example. The value of the request ID is accessible through an added
  `c2c_request_id` attribute on the Pyramid Request objects. The `requests` module is patched to automatically
  add this header.

  The requests module is also patched to monitor requests done without timeout. In that case, you can
  configure a default timeout with the `C2C_REQUESTS_DEFAULT_TIMEOUT` environment variable
  (`c2c.requests_default_timeout`). If no timeout and no default is specified, a warning is issued.



  The SQL profiler must be configured with the `C2C_SQL_PROFILER_ENABLED` environment variable. That enables a view
  to query the status of the profiler (`{C2C_BASE_PATH}/sql_profiler?secret={C2C_SECRET}`) or to
  enable/disable it (`{C2C_BASE_PATH}/sql_profiler?secret={C2C_SECRET}&enable={1|0}`).

  If enabled, for each `SELECT` query sent by SQLAlchemy, another query it done with `EXPLAIN ANALYZE`
  prepended to it. The results are sent to the `c2cwsgiutils.sql_profiler` logger.

  Don't enable that on a busy production system. It will kill your performances.



  C2cwsgiutils provide an easy way to profile an application:

  With a decorator:

  ```python
  from c2cwsgiutils.profile import Profiler

  @Profile('/my_file.prof')
  my_function():
      ...
  ```

  Or with the `with` statement:

  ```python
  from c2cwsgiutils.profile import Profiler

  with Profile('/my_file.prof'):
      ...
  ```

  Then open your file with SnakeViz:

  ```bash
  docker cp container_name:/my_file.prof .
  pip install --user snakeviz
  snakeviz my_file.prof
  ```



  The `c2cwsgiutils.db.init` allows you to setup a DB session that has two engines for accessing a
  master/slave PostgresQL setup. The slave engine (read only) will be used automatically for `GET` and `OPTIONS`
  requests and the master engine (read write) will be used for the other queries.

  To use that, your `production.ini` must look like that:

  ```ini
  sqlalchemy.url = s
  sqlalchemy.pool_recycle = s
  sqlalchemy.pool_size = s
  sqlalchemy.max_overflow = s

  sqlalchemy_slave.url = s
  sqlalchemy_slave.pool_recycle = s
  sqlalchemy_slave.pool_size = s
  sqlalchemy_slave.max_overflow = s
  ```

  And your code that initializes the DB connection must look like that:

  ```python
  import c2cwsgiutils.db

  def main(config):
      c2cwsgiutils.db.init(config, 'sqlalchemy', 'sqlalchemy_slave', force_slave=[
          "POST /api/hello"
      ])[0]
  ```

  You can use the `force_slave` and `force_master` parameters to override the defaults and force a route to use
  the master or the slave engine.



  To enable health checks, you must add some setup in your WSGI main (usually after the DB connections are
  setup). For example:

  ```python
  from c2cwsgiutils.health_check import HealthCheck

  def custom_check(request):
      global not_happy
      if not_happy:
          raise Exception("I'm not happy")
      return "happy"

  health_check = HealthCheck(config)
  health_check.add_db_session_check(models.DBSession, at_least_one_model=models.Hello)
  health_check.add_url_check('http://localhost:8080/api/hello')
  health_check.add_custom_check('custom', custom_check, 2)
  health_check.add_alembic_check(models.DBSession, '/app/alembic.ini', 3)
  ```

  Then, the URL `{C2C_BASE_PATH}/health_check?max_level=3` can be used to run the health checks and get a report
  looking like that (in case of error):

  ```json
  {
    "status": 500,
    "successes": {
      "db_engine_sqlalchemy": { "timing": 0.002 },
      "db_engine_sqlalchemy_slave": { "timing": 0.003 },
      "http://localhost/api/hello": { "timing": 0.01 },
      "alembic_app_alembic.ini_alembic": { "timing": 0.005, "result": "4a8c1bb4e775" }
    },
    "failures": {
      "custom": {
        "message": "I'm not happy",
        "timing": 0.001
      }
    }
  }
  ```

  The levels are:

  - 0: Don't add checks at this level. This max_level is used for doing a simple ping.
  - 1: Checks for anything vital for the usefulness of the service (DB, redis, ...). This is the max_level set
    by default and used by load balancers to determine if the service is alive.
  - \>=2: Use those at your convenience. Pingdom and CO are usually setup at max_level=100. So stay below.

  The URL `{C2C_BASE_PATH}/health_check?checks=<check_name>` can be used to run the health checks on some
  checks, coma separated list.

  When you instantiate the `HealthCheck` class, two checks may be automatically enabled:

  - If redis is configured, check that redis is reachable.
  - If redis is configured and the version information is available, check that the version matches
    across all instances.

  Look at the documentation of the `c2cwsgiutils.health_check.HealthCheck` class for more information.



  A command is provided that can generate Doxygen graphs of an SQLAlchemy ORM model.
  See [acceptance_tests/app/models_graph.py](acceptance_tests/app/models_graph.py) how it's used.



  If the `/app/versions.json` exists, a view is added (`{C2C_BASE_PATH}/versions.json`) to query the current
  version of a app. This file is generated by calling the `c2cwsgiutils-genversion [$GIT_TAG] $GIT_HASH`
  command line. Usually done in the [Dockerfile](acceptance_tests/app/Dockerfile) of the WSGI application.



  [Prometheus client](https://github.com/prometheus/client_python) is integrated in c2cwsgiutils.

  It will work in multi process mode with the limitation listed in the
  [`prometheus_client` documentation](https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn).

  To enable it you should provide the `C2C_PROMETHEUS_PORT` environment variable.
  For security reason, this port should not be exposed.

  We can customize it with the following environment variables:

  - `C2C_PROMETHEUS_PREFIX`: to customize the prefix, default is `c2cwsggiutils-`.
  - `C2C_PROMETHEUS_PACKAGES` the packages that will be present in the version information, default is `c2cwsgiutils,pyramid,gunicorn,sqlalchemy`.
  - `C2C_PROMETHEUS_APPLICATION_PACKAGE` the packages that will be present in the version information as application.

  And you should add in your `gunicorn.conf.py`:

  ```python
  from prometheus_client import multiprocess


  def on_starting(server):
      from c2cwsgiutils import prometheus

      del server

      prometheus.start()


  def post_fork(server, worker):
      from c2cwsgiutils import prometheus

      del server, worker

      prometheus.cleanup()


  def child_exit(server, worker):
      del server

      multiprocess.mark_process_dead(worker.pid)
  ```

  In your `Dockerfile` you should add:

  ```dockerfile
  RUN mkdir -p /prometheus-metrics \
      && chmod a+rwx /prometheus-metrics
  ENV PROMETHEUS_MULTIPROC_DIR=/prometheus-metrics
  ```



  See [official documentation](https://github.com/prometheus/client_python#custom-collectors).

  Related to the Unix process.

  ```python
  from c2cwsgiutils import broadcast, prometheus

  prometheus.MULTI_PROCESS_COLLECTOR_BROADCAST_CHANNELS.append("prometheus_collector_custom")
  broadcast.subscribe("c2cwsgiutils_prometheus_collect_gc", _broadcast_collector_custom)
  my_custom_collector_instance = MyCustomCollector()


  def _broadcast_collector_custom() -> List[prometheus.SerializedGauge]:
      """Get the collected GC gauges."""

      return prometheus.serialize_collected_data(my_custom_collector_instance)
  ```

  Related to the host, use that in the `gunicorn.conf.py`:

  ```python
  def on_starting(server):
      from c2cwsgiutils import prometheus

      del server

      registry = CollectorRegistry()
      registry.register(MyCollector())
      prometheus.start(registry)
  ```



  Look at the `c2cwsgiutils-stats-db` utility if you want to generate statistics (gauges) about the
  row counts.



  With c2cwsgiutils each instance (Pod) has its own metrics, so we need to aggregate them to have the metrics for the service you probably need to use `sum by (<fields>) (<metric>)` to get the metric (especially for counters).



  To have the application initialized in a script you should use the
  `c2cwsgiutils.setup_process.bootstrap_application_from_options` function.

  Example of `main` function:

  ```python
  def main() -> None:
      parser = argparse.ArgumentParser(description="My scrypt.")
      
      c2cwsgiutils.setup_process.fill_arguments(parser)
      args = parser.parse_args()
      env = c2cwsgiutils.setup_process.bootstrap_application_from_options(args)
      settings = env["registry"].settings

      
  ```

  If you need an access to the database you should add:

  ```python
      engine = c2cwsgiutils.db.get_engine(settings)
      session_factory = c2cwsgiutils.db.get_session_factory(engine)
      with transaction.manager:
          
  ```

  If you need the database connection without the application context, you can replace:

  ```python
      env = c2cwsgiutils.setup_process.bootstrap_application_from_options(args)
      settings = env["registry"].settings
  ```

  by:

  ```python
      loader = pyramid.scripts.common.get_config_loader(args.config_uri)
      loader.setup_logging(parse_vars(args.config_vars) if args.config_vars else None)
      settings = loader.get_settings()
  ```



  To enable the debugging interface, you must set the `C2C_DEBUG_VIEW_ENABLED` environment variable. Then you can
  have dumps of a few things:

  - every threads' stacktrace: `{C2C_BASE_PATH}/debug/stacks?secret={C2C_SECRET}`
  - memory usage: `{C2C_BASE_PATH}/debug/memory?secret={C2C_SECRET}&limit=30&analyze_type=builtins.dict&python_internals_map=false`
  - object ref: `{C2C_BASE_PATH}/debug/show_refs.dot?secret={C2C_SECRET}&analyze_type=gunicorn.app.wsgiapp.WSGIApplication&analyze_id=12345&max_depth=3&too_many=10&filter=1024&no_extra_info&backrefs`
    `analyze_type` and `analyze_id` should not ve used toogether, you can use it like:
    ```bash
    curl "<URL>" > /tmp/show_refs.dot
    dot -Lg -Tpng /tmp/show_refs.dot > /tmp/show_refs.png
    ```
  - memory increase when calling another API: `{C2C_BASE_PATH}/debug/memory_diff?path={path_info}&secret={C2C_SECRET}&limit=30&no_warmup`
  - sleep the given number of seconds (to test load balancer timeouts): `{C2C_BASE_PATH}/debug/sleep?secret={C2C_SECRET}&time=60.2`
  - see the HTTP headers received by WSGI: `{C2C_BASE_PATH}/debug/headers?secret={C2C_SECRET}&status=500`
  - return an HTTP error: `{C2C_BASE_PATH}/debug/error?secret={C2C_SECRET}&status=500`

  To ease local development, the views are automatically reloaded when files change. In addition, the filesystem is mounted by the `docker-compose.override.yaml` file. Make sure not to use such file / mechanism in production.



  Some c2cwsgiutils APIs effect or query the state of the WSGI server. Since only one process out of the 5
  (by default) time the number of servers gets a query, only this one will be affected. To avoid that, you
  can configure c2cwsgiutils to use Redis pub/sub to broadcast those requests and collect the answers.

  The impacted APIs are:

  - `{C2C_BASE_PATH}/debug/stacks`
  - `{C2C_BASE_PATH}/debug/memory`
  - `{C2C_BASE_PATH}/logging/level`
  - `{C2C_BASE_PATH}/sql_profiler`

  The configuration parameters are:

  - `C2C_REDIS_URL` (`c2c.redis_url`): The URL to the Redis single instance to use
  - `C2C_REDIS_OPTIONS`: The Redis options, comma separated list of <key>=<value>, the value is parsed as YAML
  - `C2C_REDIS_SENTINELS`: The coma separated list of Redis host:port sentinel instances to use
  - `C2C_REDIS_SERVICENAME`: The redis service name in case of using sentinels
  - `C2C_REDIS_DB`: The redis database number in case of using sentinels
  - `C2C_BROADCAST_PREFIX` (`c2c.broadcast_prefix`): The prefix to add to the channels being used (must be
    different for 2 different services)

  If not configured, only the process receiving the request is impacted.



  To have CORS compliant views, define your views like that:

  ```python
  from c2cwsgiutils import services
  hello_service = services.create("hello", "/hello", cors_credentials=True)

  @hello_service.get()
  def hello_get(request):
      return {'hello': True}
  ```



  c2cwsgiutils can install exception handling views that will catch any exception raised by the
  application views and will transform it into a JSON response with a HTTP status corresponding to the error.

  You can enable this by setting `C2C_ENABLE_EXCEPTION_HANDLING` (`c2c.enable_exception_handling`) to "1".

  In development mode (`DEVELOPMENT=1`), all the details (SQL statement, stacktrace, ...) are sent to the
  client. In production mode, you can still get them by sending the secret defined in `C2C_SECRET` in the query.

  If you want to use pyramid_debugtoolbar, you need to disable exception handling and configure it like that:

  ```ini
  pyramid.includes =
      pyramid_debugtoolbar
  debugtoolbar.enabled = true
  debugtoolbar.hosts = 0.0.0.0/0
  debugtoolbar.intercept_exc = debug
  debugtoolbar.show_on_exc_only = true
  c2c.enable_exception_handling = 0
  ```



  Some JSON renderers are available:

  - `json`: the normal JSON renderer (default).
  - `fast_json`: a faster JSON renderer using ujson.
  - `cornice_json`: the normal JSON renderer wrapped around cornice CorniceRenderer.
  - `cornice_fast_json`: a faster JSON renderer wrapped around cornice CorniceRenderer.

  Both pretty prints the rendered JSON. While this adds significant amount of whitespace, the difference in
  bytes transmitted on the network is negligible thanks to gzip compression.

  The `fast_json` renderer is using ujson which is faster, but doesn't offer the ability to change the rendering
  of some types (the `default` parameter of json.dumps). This will interact badly with `papyrus` and such.

  The cornice versions should be used to avoid the "'JSON' object has no attribute 'render_errors'" error.



  The stacktraces can be sent to a sentry.io service for collection. To enable it, you must set the `SENTRY_URL`
  (`c2c.sentry_url`) to point the the project's public DSN.

  A few other environment variables can be used to tune the info sent with each report:

  - `SENTRY_EXCLUDES` (`c2c.sentry.excludes`): list of loggers (colon separated, without spaces) to exclude for sentry
  - `GIT_HASH` (`c2c.git_hash`): will be used for the release
  - `SENTRY_CLIENT_RELEASE`: If not equal to "latest", will be taken for the release instead of the GIT_HASH
  - `SENTRY_CLIENT_ENVIRONMENT`: the environment (dev, int, prod, ...)
  - `SENTRY_CLIENT_IGNORE_EXCEPTIONS`: list (coma separated) of exceptions to ignore (defaults to SystemExit)
  - `SENTRY_TAG_...`: to add other custom tags
  - `SENTRY_LEVEL`: starting from what logging level to send events to Sentry (defaults to ERROR)
  - `SENTRY_TRACES_SAMPLE_RATE`: The percentage of events to send to sentry in order to compute the performance. Value between 0 and 1, default is 0.



  You will need `docker` (>=1.12.0), `docker-compose` (>=1.10.0) and
  `make` installed on the machine to play with this project.
  Check available versions of `docker-engine` with
  `apt-get policy docker-engine` and eventually force install the
  up-to-date version using a command similar to
  `apt-get install docker-engine=1.12.3-0~xenial`.

  To lint and test everything, run the following command:

  ```shell
  make
  ```

  Make sure you are strict with the version numbers:

  - bug fix version change: Nothing added, removed or changed in the API and only bug fix
    version number changes in the dependencies
  - minor version change: The API must remain backward compatible and only minor version
    number changes in the dependencies
  - major version change: The API and the dependencies are not backward compatible

  To make a release:

  - Change the the version in [setup.py](setup.py).
  - Commit and push to master.
  - Tag the GIT commit.
  - Add the new branch name in the `.github/workflows/rebuild.yaml` and
    `.github/workflows/audit.yaml` files.





  To test the screenshots, you need to install `node` with `npm`, to do that add the following lines in your `Dockerfile`:

  ```dockerfile
  RUN --mount=type=cache,target=/var/lib/apt/lists \
      --mount=type=cache,target=/var/cache,sharing=locked \
      apt-get install --yes --no-install-recommends gnupg \
      && . /etc/os-release \
      && echo "deb https://deb.nodesource.com/node_18.x ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/nodesource.list \
      && curl --silent https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - \
      && apt-get update \
      && apt-get install --assume-yes --no-install-recommends 'nodejs=18.*' \
          libx11-6 libx11-xcb1 libxcomposite1 libxcursor1 \
          libxdamage1 libxext6 libxi6 libxtst6 libnss3 libcups2 libxss1 libxrandr2 libasound2 libatk1.0-0 \
          libatk-bridge2.0-0 libpangocairo-1.0-0 libgtk-3.0 libxcb-dri3-0 libgbm1 libxshmfence1
  ```

  To do the image test call `check_screenshot` e.g.:

  ```python
  from c2cwsgiutils.acceptance import image

  def test_screenshot(app_connection):
      image.check_screenshot(
          app_connection.base_url + "my-path",
          width=800,
          height=600,
          result_folder="results",
          expected_filename=os.path.join(os.path.dirname(__file__), "my-check.expected.png"),
      )
  ```



  Install the pre-commit hooks:

  ```bash
  pip install pre-commit
  pre-commit install --allow-missing-config
  ```


pkgId: 3113905295f052e06b2d142cacb9de00183c00ef4726de93b04dcfeff8645194
name: python-c2cwsgiutils
arch: src
version: 6.1.3
epoch: 0
release: 1
summary: Common utilities for Camptocamp WSGI applications
url: https://github.com/camptocamp/c2cwsgiutils
rpm_license: BSD-2-Clause
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2cwsgiutils-6.1.3-1.src.rpm
checksum_type: sha256
description:
  This is a Python 3 library providing common tools for Camptocamp WSGI
  applications:

  - Provide prometheus metrics
  - Allow to use a master/slave PostgresQL configuration
  - Logging handler for CEE/UDP logs
    - An optional view to change runtime the log levels
  - SQL profiler to debug DB performance problems, disabled by default. Warning, it will slow down everything.
  - A view to get the version information about the application and the installed packages
  - A framework for implementing a health_check service
  - Error handlers to send JSON messages to the client in case of error
  - A cornice service drop in replacement for setting up CORS

  Also provide tools for writing acceptance tests:

  - A class that can be used from a py.test fixture to control a composition
  - A class that can be used from a py.text fixture to test a REST API

  As an example on how to use it in an application provided by a Docker image, you can look at the
  test application in [acceptance_tests/app](acceptance_tests/app).
  To see how to test such an application, look at [acceptance_tests/tests](acceptance_tests/tests).





  Here we didn't do a minimal install of c2cwsgiutils, but be put in place everything needed to
  monitor the application in integration and production environment.

  The library is available in PYPI:
  [https://pypi.python.org/pypi/c2cwsgiutils](https://pypi.python.org/pypi/c2cwsgiutils)

  Copy and adapt these template configuration file into your project:

  - [production.ini](acceptance_tests/app/production.ini);
  - [gunicorn.conf.py](acceptance_tests/app/gunicorn.conf.py).
    Then replace `c2cwsgiutils_app` by your package name.

  You should install `c2cwsgiutils` with the tool you use to manage your pip dependencies.

  In the `Dockerfile` you should add the following lines:

  ```dockerfile

  RUN c2cwsgiutils-genversion $(git rev-parse HEAD)

  CMD ["gunicorn", "--paste=/app/production.ini"]


  ENV \
    DEVELOPMENT=0 \
    SQLALCHEMY_POOL_RECYCLE=30 \
    SQLALCHEMY_POOL_SIZE=5 \
    SQLALCHEMY_MAX_OVERFLOW=25 \
    SQLALCHEMY_SLAVE_POOL_RECYCLE=30 \
    SQLALCHEMY_SLAVE_POOL_SIZE=5 \
    SQLALCHEMY_SLAVE_MAX_OVERFLOW=25\
    LOG_TYPE=console \
    OTHER_LOG_LEVEL=WARNING \
    GUNICORN_LOG_LEVEL=WARNING \
    SQL_LOG_LEVEL=WARNING \
    C2CWSGIUTILS_LOG_LEVEL=WARNING \
    LOG_LEVEL=INFO
  ```

  Add in your `main` function.

  ```python
  config.include("c2cwsgiutils.pyramid")
  dbsession = c2cwsgiutils.db.init(config, "sqlalchemy", "sqlalchemy_slave")

  config.scan(...)


  health_check = c2cwsgiutils.health_check.HealthCheck(config)
  health_check.add_db_session_check(dbsession)
  health_check.add_alembic_check(dbsession, "/app/alembic.ini", 1)
  ```

  The related environment variables:

  - `DEVELOPMENT`: set to `1` to enable the development mode, default is `0`.
  - `SQLALCHEMY_URL`: SQL alchemy URL, like `postgresql://user:password@host:port/dbname`.
  - `SQLALCHEMY_POOL_RECYCLE`: The SQL alchemy pool recycle, default is `30`.
  - `SQLALCHEMY_POOL_SIZE`: The SQL alchemy pool size, default is `5`.
  - `SQLALCHEMY_MAX_OVERFLOW`: SQL alchemy max overflow, default is `25`.
  - `SQLALCHEMY_SLAVE_URL`: The SQL alchemy slave (read only) URL, like `postgresql://user:password@host:port/dbname`.
  - `SQLALCHEMY_SLAVE_POOL_RECYCLE`: The SQL alchemy slave pool recycle, default is `30`.
  - `SQLALCHEMY_SLAVE_POOL_SIZE`: The SQL alchemy slave pool size, default is `5`.
  - `SQLALCHEMY_SLAVE_MAX_OVERFLOW`: The SQL alchemy slave max overflow, default is `25`.
  - `GUNICORN_WORKERS`: The number of workers, default is `2`.
  - `GUNICORN_THREADS`: The number of threads per worker, default is `10`.
  - `LOG_TYPE`: The types of logs, default is `console`, should be `json` on kubernetes to work well with
    [elk](https://www.elastic.co/fr/what-is/elk-stack).
  - `LOG_LEVEL`: The application log level, default is `INFO`.
  - `SQL_LOG_LEVEL`: The SQL query log level, `WARNING`: no logs, `INFO`: logs the queries,
    `DEBUG` also logs the results, default is `WARNING`.
  - `GUNICORN_ERROR_LOG_LEVEL`: The Gunicorn error log level, default is `WARNING`.
  - `C2CWSGIUTILS_CONFIG`: The fallback ini file to use by gunicorn, default is `production.ini`.
  - `C2CWSGIUTILS_LOG_LEVEL`: The c2c WSGI utils log level, default is `WARNING`.
  - `OTHER_LOG_LEVEL`: The log level for all the other logger, default is `WARNING`.

  Those environment variables can be useful for investigation on production environments.



  Or (deprecated) as a base Docker image:
  [camptocamp/c2cwsgiutils:release_5](https://hub.docker.com/r/camptocamp/c2cwsgiutils/) or
  [ghcr.io/camptocamp/c2cwsgiutils:release_5](https://github.com/orgs/camptocamp/packages/container/package/c2cwsgiutils)

  If you need an image with a smaller foot print, use the tags prefixed with `-light`. Those are without
  GDAL and without the build tools.

  We deprecate the Docker image because:

  - The project wants to choose the base image.
  - The project pin different versions of the dependencies.



  In general, configuration can be done both with environment variables (taken first) or with entries in the
  `production.ini` file.

  You can configure the base URL for accessing the views provided by c2cwsgiutils with an environment variable
  named `C2C_BASE_PATH` or in the `production.ini` file with a property named `c2c.base_path`.

  A few REST APIs are added and can be seen with this URL:
  `{C2C_BASE_PATH}`.

  Some APIs are protected by a secret. This secret is specified in the `C2C_SECRET` variable or `c2c.secret`
  property. It is either passed as the `secret` query parameter or the `X-API-Key` header. Once
  accessed with a good secret, a cookie is stored and the secret can be omitted.

  An alternative of using `C2C_SECRET` is to use an authentication on GitHub,
  [create the GitHub application](https://github.com/settings/applications/new).

  Configure the json renderers with the `C2C_JSON_PRETTY_PRINT` and `C2C_JSON_SORT_KEYS` environment
  variables or `c2c.json.pretty_print`and `c2c.json.sort_keys` properties. Default is `false`.

  Then it will redirect the user to the github authentication form if not already authenticated
  (using `C2C_AUTH_GITHUB_CLIENT_ID`, `C2C_AUTH_GITHUB_CLIENT_SECRET` and `C2C_AUTH_GITHUB_SCOPE`).

  Then we will check if the user is allowed to access to the application, for that we check
  if the user has enough right on a GitHub repository (using `C2C_AUTH_GITHUB_REPOSITORY`
  and `C2C_AUTH_GITHUB_REPOSITORY_ACCESS_TYPE`).

  Finally we store the session information in an encrypted cookie (using `C2C_AUTH_SECRET`
  and `C2C_AUTH_COOKIE`).

  Configuration details:

  Using the environment variable `C2C_AUTH_GITHUB_REPOSITORY` or the config key `c2c.auth.github.repository`
  to define the related GitHub repository (required).

  Using the environment variable `C2C_AUTH_GITHUB_ACCESS_TYPE` or the config key
  `c2c.auth.github.access_type` to define the type of required access can be `pull`, `push` or
  `admin` (default is `push`)

  Using the environment variable `C2C_AUTH_GITHUB_CLIENT_ID` or the config key `c2c.auth.github.client_id` to
  define the GitHub application ID (required)

  Using the environment variable `C2C_AUTH_GITHUB_CLIENT_SECRET` or the config key
  `c2c.auth.github.client_secret` to define the GitHub application secret (required)

  Using the environment variable `C2C_AUTH_GITHUB_SCOPE` or the config key `c2c.auth.github.scope` to define
  the GitHub scope (default is `repo`), see [GitHub documentation](https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/)

  Using the environment variable `C2C_AUTH_GITHUB_SECRET` or the config key `c2c.auth.github.auth.secret` to
  define the used secret for JWD encryption (required, with a length at least of 16)

  Using the environment variable `C2C_AUTH_GITHUB_COOKIE` or the config key `c2c.auth.github.auth.cookie` to
  define the used cookie name (default is `c2c-auth-jwt`)

  Using the environment variable `C2C_AUTH_GITHUB_AUTH_URL` or the config key `c2c.auth.github.auth_url` to
  define the GitHub auth URL (default is `https://github.com/login/oauth/authorize`)

  Using the environment variable `C2C_AUTH_GITHUB_TOKEN_URL` or the config key `c2c.auth.github.token_url` to
  define the GitHub auth URL (default is `https://github.com/login/oauth/access_token`)

  Using the environment variable `C2C_AUTH_GITHUB_USER_URL` or the config key `c2c.auth.github.user_url` to
  define the GitHub auth URL (default is `https://api.github.com/user`)

  Using the environment variable `C2C_AUTH_GITHUB_REPO_URL` or the config key `c2c.auth.github.repo_url` to
  define the GitHub auth URL (default is `https://api.github.com/repo`)

  Using the environment variable `C2C_AUTH_GITHUB_PROXY_URL` or the config key `c2c.auth.github.auth.proxy_url` to
  define a redirect proxy between GitHub and our application to be able to share an OAuth2 application on GitHub (default is no proxy).
  Made to work with [this proxy](https://github.com/camptocamp/redirect/).

  Using the environment variable `C2C_USE_SESSION` or the config key `c2c.use_session` to
  define if we use a session. Currently, we can use the session to store a state, used to prevent CSRF, during OAuth2 login (default is `false`)



  All the environment variables are usable in the configuration file using stuff like `s`.

  To enable most of the features of c2cwsgiutils, you need to add this line to your WSGI main:

  ```python
  import c2cwsgiutils.pyramid
  config.include(c2cwsgiutils.pyramid.includeme)
  ```

  Error catching views will be put in place to return errors as JSON.

  A custom loader is provided to run pyramid scripts against configuration files containing environment variables:

  ```shell
  proutes c2c://production.ini      # relative path
  proutes c2c:///app/production.ini # absolute path
  ```

  A filter is automatically installed to handle the HTTP headers set by common proxies and have correct values
  in the request object (`request.client_addr`, for example). This filter is equivalent to what the
  `PasteDeploy#prefix` (minus the prefix part) does, but supports newer headers as well (`Forwarded`).
  If you need to prefix your routes, you can use the `route_prefix` parameter of the `Configurator` constructor.



  Two new logging backends are provided:

  - `c2cwsgiutils.pyramid_logging.PyramidCeeSysLogHandler`: to send @cee formatted logs to syslog through UDP.
  - `c2cwsgiutils.pyramid_logging.JsonLogHandler`: to output (on stdout or stderr) JSON formatted logs.

  Look at the logging configuration part of
  [acceptance_tests/app/production.ini](acceptance_tests/app/production.ini) for paste and commands line.

  The logging configuration is imported automatically by gunicorn, it is possible to visualize the dict config by setting the environment variable `DEBUG_LOGCONFIG=1`.

  You can enable a view to configure the logging level on a live system using the `C2C_LOG_VIEW_ENABLED` environment
  variable. Then, the current status of a logger can be queried with a GET on
  `{C2C_BASE_PATH}/logging/level?secret={C2C_SECRET}&name={logger_name}` and can be changed with
  `{C2C_BASE_PATH}/logging/level?secret={C2C_SECRET}&name={logger_name}&level={level}`. Overrides are stored in
  Redis, if `C2C_REDIS_URL` (`c2c.redis_url`) or `C2C_REDIS_SENTINELS` is configured.



  You can enable a view to force usage of the slave engine using the `C2C_DB_MAINTENANCE_VIEW_ENABLED` environment
  variable. Then, the database can be made "readonly" with
  `{C2C_BASE_PATH}/db/maintenance?secret={C2C_SECRET}&readonly=true`.
  The current state is stored in Redis, if `C2C_REDIS_URL` (`c2c.redis_url`) or `C2C_REDIS_SENTINELS` is configured.



  In order to follow the logs generated by a request across all the services (think separate processes),
  c2cwsgiutils tries to flag averything with a request ID. This field can come from the input as request headers
  (`X-Request-ID`, `X-Correlation-ID`, `Request-ID` or `X-Varnish`) or will default to a UUID. You can add an
  additional request header as source for that by defining the `C2C_REQUEST_ID_HEADER` environment variable
  (`c2c.request_id_header`).

  In JSON logging formats, a `request_id` field is automatically added.

  You can enable (disabled by default since it can have a cost) the flagging of the SQL requests as well by
  setting the C2C_SQL_REQUEST_ID environment variable (or c2c.sql_request_id in the .ini file). This will use
  the application name to pass along the request id. If you do that, you must include the application name in
  the PostgreSQL logs by setting `log_line_prefix` to something like `"%a "` (don't forget the space).

  Then, in your application, it is recommended to transmit the request ID to the external REST APIs. Use
  the `X-Request-ID` HTTP header, for example. The value of the request ID is accessible through an added
  `c2c_request_id` attribute on the Pyramid Request objects. The `requests` module is patched to automatically
  add this header.

  The requests module is also patched to monitor requests done without timeout. In that case, you can
  configure a default timeout with the `C2C_REQUESTS_DEFAULT_TIMEOUT` environment variable
  (`c2c.requests_default_timeout`). If no timeout and no default is specified, a warning is issued.



  The SQL profiler must be configured with the `C2C_SQL_PROFILER_ENABLED` environment variable. That enables a view
  to query the status of the profiler (`{C2C_BASE_PATH}/sql_profiler?secret={C2C_SECRET}`) or to
  enable/disable it (`{C2C_BASE_PATH}/sql_profiler?secret={C2C_SECRET}&enable={1|0}`).

  If enabled, for each `SELECT` query sent by SQLAlchemy, another query it done with `EXPLAIN ANALYZE`
  prepended to it. The results are sent to the `c2cwsgiutils.sql_profiler` logger.

  Don't enable that on a busy production system. It will kill your performances.



  C2cwsgiutils provide an easy way to profile an application:

  With a decorator:

  ```python
  from c2cwsgiutils.profile import Profiler

  @Profile('/my_file.prof')
  my_function():
      ...
  ```

  Or with the `with` statement:

  ```python
  from c2cwsgiutils.profile import Profiler

  with Profile('/my_file.prof'):
      ...
  ```

  Then open your file with SnakeViz:

  ```bash
  docker cp container_name:/my_file.prof .
  pip install --user snakeviz
  snakeviz my_file.prof
  ```



  The `c2cwsgiutils.db.init` allows you to setup a DB session that has two engines for accessing a
  master/slave PostgresQL setup. The slave engine (read only) will be used automatically for `GET` and `OPTIONS`
  requests and the master engine (read write) will be used for the other queries.

  To use that, your `production.ini` must look like that:

  ```ini
  sqlalchemy.url = s
  sqlalchemy.pool_recycle = s
  sqlalchemy.pool_size = s
  sqlalchemy.max_overflow = s

  sqlalchemy_slave.url = s
  sqlalchemy_slave.pool_recycle = s
  sqlalchemy_slave.pool_size = s
  sqlalchemy_slave.max_overflow = s
  ```

  And your code that initializes the DB connection must look like that:

  ```python
  import c2cwsgiutils.db

  def main(config):
      c2cwsgiutils.db.init(config, 'sqlalchemy', 'sqlalchemy_slave', force_slave=[
          "POST /api/hello"
      ])[0]
  ```

  You can use the `force_slave` and `force_master` parameters to override the defaults and force a route to use
  the master or the slave engine.



  To enable health checks, you must add some setup in your WSGI main (usually after the DB connections are
  setup). For example:

  ```python
  from c2cwsgiutils.health_check import HealthCheck

  def custom_check(request):
      global not_happy
      if not_happy:
          raise Exception("I'm not happy")
      return "happy"

  health_check = HealthCheck(config)
  health_check.add_db_session_check(models.DBSession, at_least_one_model=models.Hello)
  health_check.add_url_check('http://localhost:8080/api/hello')
  health_check.add_custom_check('custom', custom_check, 2)
  health_check.add_alembic_check(models.DBSession, '/app/alembic.ini', 3)
  ```

  Then, the URL `{C2C_BASE_PATH}/health_check?max_level=3` can be used to run the health checks and get a report
  looking like that (in case of error):

  ```json
  {
    "status": 500,
    "successes": {
      "db_engine_sqlalchemy": { "timing": 0.002 },
      "db_engine_sqlalchemy_slave": { "timing": 0.003 },
      "http://localhost/api/hello": { "timing": 0.01 },
      "alembic_app_alembic.ini_alembic": { "timing": 0.005, "result": "4a8c1bb4e775" }
    },
    "failures": {
      "custom": {
        "message": "I'm not happy",
        "timing": 0.001
      }
    }
  }
  ```

  The levels are:

  - 0: Don't add checks at this level. This max_level is used for doing a simple ping.
  - 1: Checks for anything vital for the usefulness of the service (DB, redis, ...). This is the max_level set
    by default and used by load balancers to determine if the service is alive.
  - \>=2: Use those at your convenience. Pingdom and CO are usually setup at max_level=100. So stay below.

  The URL `{C2C_BASE_PATH}/health_check?checks=<check_name>` can be used to run the health checks on some
  checks, coma separated list.

  When you instantiate the `HealthCheck` class, two checks may be automatically enabled:

  - If redis is configured, check that redis is reachable.
  - If redis is configured and the version information is available, check that the version matches
    across all instances.

  Look at the documentation of the `c2cwsgiutils.health_check.HealthCheck` class for more information.



  A command is provided that can generate Doxygen graphs of an SQLAlchemy ORM model.
  See [acceptance_tests/app/models_graph.py](acceptance_tests/app/models_graph.py) how it's used.



  If the `/app/versions.json` exists, a view is added (`{C2C_BASE_PATH}/versions.json`) to query the current
  version of a app. This file is generated by calling the `c2cwsgiutils-genversion [$GIT_TAG] $GIT_HASH`
  command line. Usually done in the [Dockerfile](acceptance_tests/app/Dockerfile) of the WSGI application.



  [Prometheus client](https://github.com/prometheus/client_python) is integrated in c2cwsgiutils.

  It will work in multi process mode with the limitation listed in the
  [`prometheus_client` documentation](https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn).

  To enable it you should provide the `C2C_PROMETHEUS_PORT` environment variable.
  For security reason, this port should not be exposed.

  We can customize it with the following environment variables:

  - `C2C_PROMETHEUS_PREFIX`: to customize the prefix, default is `c2cwsggiutils-`.
  - `C2C_PROMETHEUS_PACKAGES` the packages that will be present in the version information, default is `c2cwsgiutils,pyramid,gunicorn,sqlalchemy`.
  - `C2C_PROMETHEUS_APPLICATION_PACKAGE` the packages that will be present in the version information as application.

  And you should add in your `gunicorn.conf.py`:

  ```python
  from prometheus_client import multiprocess


  def on_starting(server):
      from c2cwsgiutils import prometheus

      del server

      prometheus.start()


  def post_fork(server, worker):
      from c2cwsgiutils import prometheus

      del server, worker

      prometheus.cleanup()


  def child_exit(server, worker):
      del server

      multiprocess.mark_process_dead(worker.pid)
  ```

  In your `Dockerfile` you should add:

  ```dockerfile
  RUN mkdir -p /prometheus-metrics \
      && chmod a+rwx /prometheus-metrics
  ENV PROMETHEUS_MULTIPROC_DIR=/prometheus-metrics
  ```



  See [official documentation](https://github.com/prometheus/client_python#custom-collectors).

  Related to the Unix process.

  ```python
  from c2cwsgiutils import broadcast, prometheus

  prometheus.MULTI_PROCESS_COLLECTOR_BROADCAST_CHANNELS.append("prometheus_collector_custom")
  broadcast.subscribe("c2cwsgiutils_prometheus_collect_gc", _broadcast_collector_custom)
  my_custom_collector_instance = MyCustomCollector()


  def _broadcast_collector_custom() -> List[prometheus.SerializedGauge]:
      """Get the collected GC gauges."""

      return prometheus.serialize_collected_data(my_custom_collector_instance)
  ```

  Related to the host, use that in the `gunicorn.conf.py`:

  ```python
  def on_starting(server):
      from c2cwsgiutils import prometheus

      del server

      registry = CollectorRegistry()
      registry.register(MyCollector())
      prometheus.start(registry)
  ```



  Look at the `c2cwsgiutils-stats-db` utility if you want to generate statistics (gauges) about the
  row counts.



  With c2cwsgiutils each instance (Pod) has its own metrics, so we need to aggregate them to have the metrics for the service you probably need to use `sum by (<fields>) (<metric>)` to get the metric (especially for counters).



  To have the application initialized in a script you should use the
  `c2cwsgiutils.setup_process.bootstrap_application_from_options` function.

  Example of `main` function:

  ```python
  def main() -> None:
      parser = argparse.ArgumentParser(description="My scrypt.")
      
      c2cwsgiutils.setup_process.fill_arguments(parser)
      args = parser.parse_args()
      env = c2cwsgiutils.setup_process.bootstrap_application_from_options(args)
      settings = env["registry"].settings

      
  ```

  If you need an access to the database you should add:

  ```python
      engine = c2cwsgiutils.db.get_engine(settings)
      session_factory = c2cwsgiutils.db.get_session_factory(engine)
      with transaction.manager:
          
  ```

  If you need the database connection without the application context, you can replace:

  ```python
      env = c2cwsgiutils.setup_process.bootstrap_application_from_options(args)
      settings = env["registry"].settings
  ```

  by:

  ```python
      loader = pyramid.scripts.common.get_config_loader(args.config_uri)
      loader.setup_logging(parse_vars(args.config_vars) if args.config_vars else None)
      settings = loader.get_settings()
  ```



  To enable the debugging interface, you must set the `C2C_DEBUG_VIEW_ENABLED` environment variable. Then you can
  have dumps of a few things:

  - every threads' stacktrace: `{C2C_BASE_PATH}/debug/stacks?secret={C2C_SECRET}`
  - memory usage: `{C2C_BASE_PATH}/debug/memory?secret={C2C_SECRET}&limit=30&analyze_type=builtins.dict&python_internals_map=false`
  - object ref: `{C2C_BASE_PATH}/debug/show_refs.dot?secret={C2C_SECRET}&analyze_type=gunicorn.app.wsgiapp.WSGIApplication&analyze_id=12345&max_depth=3&too_many=10&filter=1024&no_extra_info&backrefs`
    `analyze_type` and `analyze_id` should not ve used toogether, you can use it like:
    ```bash
    curl "<URL>" > /tmp/show_refs.dot
    dot -Lg -Tpng /tmp/show_refs.dot > /tmp/show_refs.png
    ```
  - memory increase when calling another API: `{C2C_BASE_PATH}/debug/memory_diff?path={path_info}&secret={C2C_SECRET}&limit=30&no_warmup`
  - sleep the given number of seconds (to test load balancer timeouts): `{C2C_BASE_PATH}/debug/sleep?secret={C2C_SECRET}&time=60.2`
  - see the HTTP headers received by WSGI: `{C2C_BASE_PATH}/debug/headers?secret={C2C_SECRET}&status=500`
  - return an HTTP error: `{C2C_BASE_PATH}/debug/error?secret={C2C_SECRET}&status=500`

  To ease local development, the views are automatically reloaded when files change. In addition, the filesystem is mounted by the `docker-compose.override.yaml` file. Make sure not to use such file / mechanism in production.



  Some c2cwsgiutils APIs effect or query the state of the WSGI server. Since only one process out of the 5
  (by default) time the number of servers gets a query, only this one will be affected. To avoid that, you
  can configure c2cwsgiutils to use Redis pub/sub to broadcast those requests and collect the answers.

  The impacted APIs are:

  - `{C2C_BASE_PATH}/debug/stacks`
  - `{C2C_BASE_PATH}/debug/memory`
  - `{C2C_BASE_PATH}/logging/level`
  - `{C2C_BASE_PATH}/sql_profiler`

  The configuration parameters are:

  - `C2C_REDIS_URL` (`c2c.redis_url`): The URL to the Redis single instance to use
  - `C2C_REDIS_OPTIONS`: The Redis options, comma separated list of <key>=<value>, the value is parsed as YAML
  - `C2C_REDIS_SENTINELS`: The coma separated list of Redis host:port sentinel instances to use
  - `C2C_REDIS_SERVICENAME`: The redis service name in case of using sentinels
  - `C2C_REDIS_DB`: The redis database number in case of using sentinels
  - `C2C_BROADCAST_PREFIX` (`c2c.broadcast_prefix`): The prefix to add to the channels being used (must be
    different for 2 different services)

  If not configured, only the process receiving the request is impacted.



  To have CORS compliant views, define your views like that:

  ```python
  from c2cwsgiutils import services
  hello_service = services.create("hello", "/hello", cors_credentials=True)

  @hello_service.get()
  def hello_get(request):
      return {'hello': True}
  ```



  c2cwsgiutils can install exception handling views that will catch any exception raised by the
  application views and will transform it into a JSON response with a HTTP status corresponding to the error.

  You can enable this by setting `C2C_ENABLE_EXCEPTION_HANDLING` (`c2c.enable_exception_handling`) to "1".

  In development mode (`DEVELOPMENT=1`), all the details (SQL statement, stacktrace, ...) are sent to the
  client. In production mode, you can still get them by sending the secret defined in `C2C_SECRET` in the query.

  If you want to use pyramid_debugtoolbar, you need to disable exception handling and configure it like that:

  ```ini
  pyramid.includes =
      pyramid_debugtoolbar
  debugtoolbar.enabled = true
  debugtoolbar.hosts = 0.0.0.0/0
  debugtoolbar.intercept_exc = debug
  debugtoolbar.show_on_exc_only = true
  c2c.enable_exception_handling = 0
  ```



  Some JSON renderers are available:

  - `json`: the normal JSON renderer (default).
  - `fast_json`: a faster JSON renderer using ujson.
  - `cornice_json`: the normal JSON renderer wrapped around cornice CorniceRenderer.
  - `cornice_fast_json`: a faster JSON renderer wrapped around cornice CorniceRenderer.

  Both pretty prints the rendered JSON. While this adds significant amount of whitespace, the difference in
  bytes transmitted on the network is negligible thanks to gzip compression.

  The `fast_json` renderer is using ujson which is faster, but doesn't offer the ability to change the rendering
  of some types (the `default` parameter of json.dumps). This will interact badly with `papyrus` and such.

  The cornice versions should be used to avoid the "'JSON' object has no attribute 'render_errors'" error.



  The stacktraces can be sent to a sentry.io service for collection. To enable it, you must set the `SENTRY_URL`
  (`c2c.sentry_url`) to point the the project's public DSN.

  A few other environment variables can be used to tune the info sent with each report:

  - `SENTRY_EXCLUDES` (`c2c.sentry.excludes`): list of loggers (colon separated, without spaces) to exclude for sentry
  - `GIT_HASH` (`c2c.git_hash`): will be used for the release
  - `SENTRY_CLIENT_RELEASE`: If not equal to "latest", will be taken for the release instead of the GIT_HASH
  - `SENTRY_CLIENT_ENVIRONMENT`: the environment (dev, int, prod, ...)
  - `SENTRY_CLIENT_IGNORE_EXCEPTIONS`: list (coma separated) of exceptions to ignore (defaults to SystemExit)
  - `SENTRY_TAG_...`: to add other custom tags
  - `SENTRY_LEVEL`: starting from what logging level to send events to Sentry (defaults to ERROR)
  - `SENTRY_TRACES_SAMPLE_RATE`: The percentage of events to send to sentry in order to compute the performance. Value between 0 and 1 (default is 0)
  - `SENTRY_INTEGRATION_LOGGING`: If set to 0, the Sentry integration will not log anything (default is 1)
  - `SENTRY_INTEGRATION_PYRAMID`: If set to 0, the Sentry integration with Pyramid will not be enabled (default is 1)
  - `SENTRY_INTEGRATION_SQLALCHEMY`: If set to 0, the Sentry integration with SQLAlchemy will not be enabled (default is 1)
  - `SENTRY_INTEGRATION_REDIS`: If set to 0, the Sentry integration with Redis will not be enabled (default is 1)
  - `SENTRY_INTEGRATION_ASYNCIO`: If set to 0, the Sentry integration with asyncio will not be enabled (default is 1)



  You will need `docker` (>=1.12.0), `docker compose` and
  `make` installed on the machine to play with this project.
  Check available versions of `docker-engine` with
  `apt-get policy docker-engine` and eventually force install the
  up-to-date version using a command similar to
  `apt-get install docker-engine=1.12.3-0~xenial`.

  To lint and test everything, run the following command:

  ```shell
  make
  ```

  Make sure you are strict with the version numbers:

  - bug fix version change: Nothing added, removed or changed in the API and only bug fix
    version number changes in the dependencies
  - minor version change: The API must remain backward compatible and only minor version
    number changes in the dependencies
  - major version change: The API and the dependencies are not backward compatible

  To make a release:

  - Change the the version in [setup.py](setup.py).
  - Commit and push to master.
  - Tag the GIT commit.
  - Add the new branch name in the `.github/workflows/rebuild.yaml` and
    `.github/workflows/audit.yaml` files.



  Pserve will not set the headers in the environment then if you are behind a reverse proxy, you will have
  wrong values in client information, you can force them by using the environment variables:
  `C2CWSGIUTILS_FORCE_PROTO`, `C2CWSGIUTILS_FORCE_HOST` `C2CWSGIUTILS_FORCE_SERVER_NAME` and
  `C2CWSGIUTILS_FORCE_REMOTE_ADDR`.





  To test the screenshots, you need to install `node` with `npm`, to do that add the following lines in your `Dockerfile`:

  ```dockerfile
  RUN --mount=type=cache,target=/var/lib/apt/lists \
      --mount=type=cache,target=/var/cache,sharing=locked \
      apt-get install --yes --no-install-recommends gnupg \
      && . /etc/os-release \
      && echo "deb https://deb.nodesource.com/node_18.x ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/nodesource.list \
      && curl --silent https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - \
      && apt-get update \
      && apt-get install --assume-yes --no-install-recommends 'nodejs=18.*' \
          libx11-6 libx11-xcb1 libxcomposite1 libxcursor1 \
          libxdamage1 libxext6 libxi6 libxtst6 libnss3 libcups2 libxss1 libxrandr2 libasound2 libatk1.0-0 \
          libatk-bridge2.0-0 libpangocairo-1.0-0 libgtk-3.0 libxcb-dri3-0 libgbm1 libxshmfence1
  ```

  To do the image test call `check_screenshot` e.g.:

  ```python
  from c2cwsgiutils.acceptance import image

  def test_screenshot(app_connection):
      image.check_screenshot(
          app_connection.base_url + "my-path",
          width=800,
          height=600,
          result_folder="results",
          expected_filename=os.path.join(os.path.dirname(__file__), "my-check.expected.png"),
      )
  ```



  Install the pre-commit hooks:

  ```bash
  pip install pre-commit
  pre-commit install --allow-missing-config
  ```


pkgId: c9093e2278c00f4c879f1870f8aea04bcb43145e58079cc56e9f26841fbf94d3
name: python-c2dataviewer
arch: src
version: 1.13.0
epoch: 0
release: 1
summary: Python based data viewer for next generation of APS control system (C2)
url: https://github.com/epics-extensions/c2dataviewer
rpm_license: EPICS
rpm_group: Unspecified
rpm_buildhost: localhost
location_href: Packages/python-c2dataviewer-1.13.0-1.src.rpm
checksum_type: sha256
description:
  <p align="center">
    <h1 align="center">C2DataViewer: EPICS7 pvObject Data Viewer</h1>
  </p>

  C2DataViewer is a Python based data viewer for next generation of APS control system (C2).
  It is developed with pyqtgraph, PyQt, and uses pvaPy as pvAccess Python binding.
  This is a viewer for pvData structured data objects as transported by pvAccess. Some use cases are a 'scope viewer',
  and Area Detector images from the AD pva plugin.


  ```bash
  > python -m pip install c2dataviewer
  > c2dv -h
  ```



  - [Image app](documentation/imageapp.md) - displays Area detector images from the AD pva plugin
  - [Scope app](documentation/scopeapp.md) - plots pvAccess waveform data
  - [Striptool](documentation/striptool.md) - plots channel access and pvAccess scalar data



  - [Building and packaging](documentation/building.md)


