Daf
DataAxesFormats.DataAxesFormats
—
Module
The
DataAxesFormats
package provides a uniform generic interface for accessing 1D and 2D data arranged along some set of axes. This is a much-needed generalization of the
AnnData
functionality. The key features are:
- The data model
StorageTypesinclude (1) some axes with named entries, (2) vector data indexed by a single axis, (3) matrix data indexed by a pair of axes, and also (4) scalar data (anything not tied to some axis). - Explicit control over 2D data
MatrixLayouts(row or column major), with support for both dense and sparse matrices, both of which are crucial for performance. - Out of the box, allow storing the data in memory (using
MemoryDaf), directly inside HDF5 files (usingH5df), or as a collection of simple files in a directory (usingFilesDaf), which works nicely with tools likemakefor automating computation pipelines. - Import and export to/from
AnnDataFormatfor interoperability with non-Daftools. - Implementation with a focus on memory-mapping to allow for efficient processing of large data sets (in theory, larger than the system's memory). In particular, merely opening a data set is a fast operation (almost) regardless of its size.
- Well-defined interfaces for implementing additional storage
Formats. - Creating
Chainsof data sets, allowing zero-copy reuse of common data between multiple computation pipelines. -
Concatmultiple data sets into a single data set along one or more axes. - A
Querylanguage for accessing the data, providing features such as slicing, aggregation and filtering, and makingViewsandCopiesbased on these queries. - Self documenting
Computationswith an explicitContractsdescribing and enforcing the inputs and outputs, andAdaptersfor applying the computation to data of a different format.
The top-level
DataAxesFormats
module re-exports all(most) everything from the sub-modules, so you can directly access any exported symbol by
using DataAxesFormats
(or
import DataAxesFormats: MemoryDaf
), instead of having to import or use qualified names (such as
DataAxesFormats.MemoryFormat.MemoryDaf
).
The
Daf
data sets type hierarchy looks like this:
DafReader (abstract type)
├─ DafReadOnly (abstract type)
│ ├─ DafReadOnlyWrapper (created by read_only)
│ ├─ DafView (created by viewer)
│ └─ DafChainReader (created by chain_reader)
└─ DafWriter (abstract type)
├─ DafChainWriter (created by chain_writer)
├─ MemoryDaf
├─ FilesDaf
└─ H5df
Index
-
DataAxesFormats.Adapters -
DataAxesFormats.AnnDataFormat -
DataAxesFormats.Chains -
DataAxesFormats.Computations -
DataAxesFormats.Concat -
DataAxesFormats.Contracts -
DataAxesFormats.Copies -
DataAxesFormats.DataAxesFormats -
DataAxesFormats.ExampleData -
DataAxesFormats.FilesFormat -
DataAxesFormats.Formats -
DataAxesFormats.GenericFunctions -
DataAxesFormats.GenericLocks -
DataAxesFormats.GenericLogging -
DataAxesFormats.GenericTypes -
DataAxesFormats.Groups -
DataAxesFormats.H5dfFormat -
DataAxesFormats.Keys -
DataAxesFormats.MatrixLayouts -
DataAxesFormats.MemoryFormat -
DataAxesFormats.Messages -
DataAxesFormats.Operations -
DataAxesFormats.Queries -
DataAxesFormats.ReadOnly -
DataAxesFormats.Readers -
DataAxesFormats.Reconstruction -
DataAxesFormats.Registry -
DataAxesFormats.StorageTypes -
DataAxesFormats.Tokens -
DataAxesFormats.Views -
DataAxesFormats.Writers -
DataAxesFormats.Computations.CONTRACT -
DataAxesFormats.Computations.CONTRACT1 -
DataAxesFormats.Computations.CONTRACT2 -
DataAxesFormats.Computations.DEFAULT -
DataAxesFormats.FilesFormat.MAJOR_VERSION -
DataAxesFormats.FilesFormat.MINOR_VERSION -
DataAxesFormats.H5dfFormat.MAJOR_VERSION -
DataAxesFormats.H5dfFormat.MINOR_VERSION -
DataAxesFormats.MatrixLayouts.Columns -
DataAxesFormats.MatrixLayouts.Rows -
DataAxesFormats.Queries.AXIS_MASK -
DataAxesFormats.Queries.COUNTS_MATRIX -
DataAxesFormats.Queries.GROUP_BY -
DataAxesFormats.Queries.LOOKUP_PROPERTY -
DataAxesFormats.Queries.MASK_OPERATION -
DataAxesFormats.Queries.MASK_SLICE -
DataAxesFormats.Queries.MATRIX_COLUMN -
DataAxesFormats.Queries.MATRIX_ENTRY -
DataAxesFormats.Queries.MATRIX_LOOKUP -
DataAxesFormats.Queries.MATRIX_QUERY -
DataAxesFormats.Queries.MATRIX_ROW -
DataAxesFormats.Queries.NAMES_QUERY -
DataAxesFormats.Queries.POST_PROCESS -
DataAxesFormats.Queries.QUERY_OPERATORS -
DataAxesFormats.Queries.REDUCE_MATRIX -
DataAxesFormats.Queries.REDUCE_VECTOR -
DataAxesFormats.Queries.SCALAR_QUERY -
DataAxesFormats.Queries.VECTOR_ENTRY -
DataAxesFormats.Queries.VECTOR_FETCH -
DataAxesFormats.Queries.VECTOR_LOOKUP -
DataAxesFormats.Queries.VECTOR_PROPERTY -
DataAxesFormats.Queries.VECTOR_QUERY -
DataAxesFormats.Tokens.SPACE_REGEX -
DataAxesFormats.Tokens.VALUE_REGEX -
DataAxesFormats.Views.ALL_AXES -
DataAxesFormats.Views.ALL_MATRICES -
DataAxesFormats.Views.ALL_SCALARS -
DataAxesFormats.Views.ALL_VECTORS -
DataAxesFormats.Views.VIEW_ALL_AXES -
DataAxesFormats.Views.VIEW_ALL_DATA -
DataAxesFormats.Views.VIEW_ALL_MATRICES -
DataAxesFormats.Views.VIEW_ALL_SCALARS -
DataAxesFormats.Views.VIEW_ALL_VECTORS -
DataAxesFormats.Chains.ReadOnlyChain -
DataAxesFormats.Chains.WriteChain -
DataAxesFormats.Concat.MergeAction -
DataAxesFormats.Concat.MergeData -
DataAxesFormats.Concat.MergeDatum -
DataAxesFormats.Contracts.AxisSpecification -
DataAxesFormats.Contracts.Contract -
DataAxesFormats.Contracts.ContractAxes -
DataAxesFormats.Contracts.ContractAxis -
DataAxesFormats.Contracts.ContractDaf -
DataAxesFormats.Contracts.ContractData -
DataAxesFormats.Contracts.ContractDatum -
DataAxesFormats.Contracts.ContractExpectation -
DataAxesFormats.Contracts.DataSpecification -
DataAxesFormats.Copies.DataTypes -
DataAxesFormats.Copies.EmptyData -
DataAxesFormats.FilesFormat.FilesDaf -
DataAxesFormats.Formats.CacheGroup -
DataAxesFormats.Formats.DafReader -
DataAxesFormats.Formats.DafWriter -
DataAxesFormats.Formats.FormatReader -
DataAxesFormats.Formats.FormatWriter -
DataAxesFormats.Formats.Internal -
DataAxesFormats.GenericFunctions.AbnormalHandler -
DataAxesFormats.GenericLocks.QueryReadWriteLock -
DataAxesFormats.GenericTypes.Maybe -
DataAxesFormats.GenericTypes.Unsure -
DataAxesFormats.H5dfFormat.H5df -
DataAxesFormats.Keys.AxisKey -
DataAxesFormats.Keys.DataKey -
DataAxesFormats.Keys.MatrixKey -
DataAxesFormats.Keys.PropertyKey -
DataAxesFormats.Keys.ScalarKey -
DataAxesFormats.Keys.TensorKey -
DataAxesFormats.Keys.VectorKey -
DataAxesFormats.MemoryFormat.MemoryDaf -
DataAxesFormats.Operations.Abs -
DataAxesFormats.Operations.Clamp -
DataAxesFormats.Operations.Convert -
DataAxesFormats.Operations.Count -
DataAxesFormats.Operations.Fraction -
DataAxesFormats.Operations.GeoMean -
DataAxesFormats.Operations.Log -
DataAxesFormats.Operations.Max -
DataAxesFormats.Operations.Mean -
DataAxesFormats.Operations.Median -
DataAxesFormats.Operations.Min -
DataAxesFormats.Operations.Mode -
DataAxesFormats.Operations.Quantile -
DataAxesFormats.Operations.Round -
DataAxesFormats.Operations.Significant -
DataAxesFormats.Operations.Std -
DataAxesFormats.Operations.StdN -
DataAxesFormats.Operations.Sum -
DataAxesFormats.Operations.Var -
DataAxesFormats.Operations.VarN -
DataAxesFormats.Queries.And -
DataAxesFormats.Queries.AndNot -
DataAxesFormats.Queries.AsAxis -
DataAxesFormats.Queries.Axis -
DataAxesFormats.Queries.ComparisonOperation -
DataAxesFormats.Queries.CountBy -
DataAxesFormats.Queries.Fetch -
DataAxesFormats.Queries.FrameColumn -
DataAxesFormats.Queries.FrameColumns -
DataAxesFormats.Queries.GroupBy -
DataAxesFormats.Queries.IfMissing -
DataAxesFormats.Queries.IfNot -
DataAxesFormats.Queries.IsEqual -
DataAxesFormats.Queries.IsGreater -
DataAxesFormats.Queries.IsGreaterEqual -
DataAxesFormats.Queries.IsLess -
DataAxesFormats.Queries.IsLessEqual -
DataAxesFormats.Queries.IsMatch -
DataAxesFormats.Queries.IsNotEqual -
DataAxesFormats.Queries.IsNotMatch -
DataAxesFormats.Queries.Lookup -
DataAxesFormats.Queries.MaskSlice -
DataAxesFormats.Queries.Names -
DataAxesFormats.Queries.Or -
DataAxesFormats.Queries.OrNot -
DataAxesFormats.Queries.Query -
DataAxesFormats.Queries.QuerySequence -
DataAxesFormats.Queries.QueryString -
DataAxesFormats.Queries.SquareMaskColumn -
DataAxesFormats.Queries.SquareMaskRow -
DataAxesFormats.Queries.Xor -
DataAxesFormats.Queries.XorNot -
DataAxesFormats.ReadOnly.DafReadOnly -
DataAxesFormats.ReadOnly.DafReadOnlyWrapper -
DataAxesFormats.Reconstruction.PropertiesDefaults -
DataAxesFormats.Registry.EltwiseOperation -
DataAxesFormats.Registry.QueryOperation -
DataAxesFormats.Registry.ReductionOperation -
DataAxesFormats.StorageTypes.StorageFloat -
DataAxesFormats.StorageTypes.StorageInteger -
DataAxesFormats.StorageTypes.StorageMatrix -
DataAxesFormats.StorageTypes.StorageReal -
DataAxesFormats.StorageTypes.StorageScalar -
DataAxesFormats.StorageTypes.StorageScalarBase -
DataAxesFormats.StorageTypes.StorageSigned -
DataAxesFormats.StorageTypes.StorageUnsigned -
DataAxesFormats.StorageTypes.StorageVector -
DataAxesFormats.Tokens.Token -
DataAxesFormats.Views.DafView -
DataAxesFormats.Views.ViewAxes -
DataAxesFormats.Views.ViewAxis -
DataAxesFormats.Views.ViewData -
DataAxesFormats.Views.ViewDatum -
DataAxesFormats.Adapters.adapter -
DataAxesFormats.AnnDataFormat.anndata_as_daf -
DataAxesFormats.AnnDataFormat.daf_as_anndata -
DataAxesFormats.Chains.chain_reader -
DataAxesFormats.Chains.chain_writer -
DataAxesFormats.Computations.function_contract -
DataAxesFormats.Computations.function_default -
DataAxesFormats.Concat.concatenate! -
DataAxesFormats.Contracts.contractor -
DataAxesFormats.Contracts.verify_input -
DataAxesFormats.Contracts.verify_output -
DataAxesFormats.Copies.copy_all! -
DataAxesFormats.Copies.copy_axis! -
DataAxesFormats.Copies.copy_matrix! -
DataAxesFormats.Copies.copy_scalar! -
DataAxesFormats.Copies.copy_tensor! -
DataAxesFormats.Copies.copy_vector! -
DataAxesFormats.ExampleData.example_daf -
DataAxesFormats.Formats.empty_cache! -
DataAxesFormats.Formats.format_add_axis! -
DataAxesFormats.Formats.format_axes_set -
DataAxesFormats.Formats.format_axis_length -
DataAxesFormats.Formats.format_axis_vector -
DataAxesFormats.Formats.format_delete_axis! -
DataAxesFormats.Formats.format_delete_matrix! -
DataAxesFormats.Formats.format_delete_scalar! -
DataAxesFormats.Formats.format_delete_vector! -
DataAxesFormats.Formats.format_description_footer -
DataAxesFormats.Formats.format_description_header -
DataAxesFormats.Formats.format_filled_empty_sparse_matrix! -
DataAxesFormats.Formats.format_filled_empty_sparse_vector! -
DataAxesFormats.Formats.format_get_empty_dense_matrix! -
DataAxesFormats.Formats.format_get_empty_dense_vector! -
DataAxesFormats.Formats.format_get_empty_sparse_matrix! -
DataAxesFormats.Formats.format_get_empty_sparse_vector! -
DataAxesFormats.Formats.format_get_matrix -
DataAxesFormats.Formats.format_get_scalar -
DataAxesFormats.Formats.format_get_vector -
DataAxesFormats.Formats.format_has_axis -
DataAxesFormats.Formats.format_has_matrix -
DataAxesFormats.Formats.format_has_scalar -
DataAxesFormats.Formats.format_has_vector -
DataAxesFormats.Formats.format_matrices_set -
DataAxesFormats.Formats.format_relayout_matrix! -
DataAxesFormats.Formats.format_scalars_set -
DataAxesFormats.Formats.format_set_matrix! -
DataAxesFormats.Formats.format_set_scalar! -
DataAxesFormats.Formats.format_set_vector! -
DataAxesFormats.Formats.format_vectors_set -
DataAxesFormats.GenericFunctions.dedent -
DataAxesFormats.GenericFunctions.handle_abnormal -
DataAxesFormats.GenericLocks.has_read_lock -
DataAxesFormats.GenericLocks.has_write_lock -
DataAxesFormats.GenericLocks.read_lock -
DataAxesFormats.GenericLocks.read_unlock -
DataAxesFormats.GenericLocks.with_read_lock -
DataAxesFormats.GenericLocks.with_write_lock -
DataAxesFormats.GenericLocks.write_lock -
DataAxesFormats.GenericLocks.write_unlock -
DataAxesFormats.GenericLogging.setup_logger -
DataAxesFormats.Groups.collect_group_members -
DataAxesFormats.Groups.compact_groups! -
DataAxesFormats.Groups.group_names -
DataAxesFormats.MatrixLayouts.axis_name -
DataAxesFormats.MatrixLayouts.bestify -
DataAxesFormats.MatrixLayouts.check_efficient_action -
DataAxesFormats.MatrixLayouts.copy_array -
DataAxesFormats.MatrixLayouts.densify -
DataAxesFormats.MatrixLayouts.depict -
DataAxesFormats.MatrixLayouts.inefficient_action_handler -
DataAxesFormats.MatrixLayouts.major_axis -
DataAxesFormats.MatrixLayouts.minor_axis -
DataAxesFormats.MatrixLayouts.other_axis -
DataAxesFormats.MatrixLayouts.relayout -
DataAxesFormats.MatrixLayouts.relayout! -
DataAxesFormats.MatrixLayouts.require_major_axis -
DataAxesFormats.MatrixLayouts.require_minor_axis -
DataAxesFormats.MatrixLayouts.sparsify -
DataAxesFormats.MatrixLayouts.transposer -
DataAxesFormats.Messages.depict_percent -
DataAxesFormats.Messages.unique_name -
DataAxesFormats.Operations.error_invalid_parameter_value -
DataAxesFormats.Operations.float_dtype_for -
DataAxesFormats.Operations.int_dtype_for -
DataAxesFormats.Operations.parse_float_dtype_value -
DataAxesFormats.Operations.parse_int_dtype_value -
DataAxesFormats.Operations.parse_number_dtype_value -
DataAxesFormats.Operations.parse_number_value -
DataAxesFormats.Operations.parse_parameter_value -
DataAxesFormats.Operations.sum_dtype_for -
DataAxesFormats.Operations.unsigned_dtype_for -
DataAxesFormats.Queries.full_vector_query -
DataAxesFormats.Queries.get_frame -
DataAxesFormats.Queries.get_query -
DataAxesFormats.Queries.guess_typed_value -
DataAxesFormats.Queries.is_axis_query -
DataAxesFormats.Queries.query_requires_relayout -
DataAxesFormats.Queries.query_result_dimensions -
DataAxesFormats.ReadOnly.is_read_only_array -
DataAxesFormats.ReadOnly.read_only -
DataAxesFormats.ReadOnly.read_only_array -
DataAxesFormats.Readers.axes_set -
DataAxesFormats.Readers.axis_dict -
DataAxesFormats.Readers.axis_indices -
DataAxesFormats.Readers.axis_length -
DataAxesFormats.Readers.axis_vector -
DataAxesFormats.Readers.axis_version_counter -
DataAxesFormats.Readers.description -
DataAxesFormats.Readers.get_matrix -
DataAxesFormats.Readers.get_scalar -
DataAxesFormats.Readers.get_vector -
DataAxesFormats.Readers.has_axis -
DataAxesFormats.Readers.has_matrix -
DataAxesFormats.Readers.has_scalar -
DataAxesFormats.Readers.has_vector -
DataAxesFormats.Readers.matrices_set -
DataAxesFormats.Readers.matrix_version_counter -
DataAxesFormats.Readers.scalars_set -
DataAxesFormats.Readers.vector_version_counter -
DataAxesFormats.Readers.vectors_set -
DataAxesFormats.Reconstruction.reconstruct_axis! -
DataAxesFormats.Registry.compute_eltwise -
DataAxesFormats.Registry.compute_reduction -
DataAxesFormats.Registry.reduction_result_type -
DataAxesFormats.Registry.register_query_operation -
DataAxesFormats.StorageTypes.sparse_matrix_csc -
DataAxesFormats.StorageTypes.sparse_vector -
DataAxesFormats.Tokens.decode_expression -
DataAxesFormats.Tokens.encode_expression -
DataAxesFormats.Tokens.escape_value -
DataAxesFormats.Tokens.is_value_char -
DataAxesFormats.Tokens.tokenize -
DataAxesFormats.Tokens.unescape_value -
DataAxesFormats.Views.viewer -
DataAxesFormats.Writers.add_axis! -
DataAxesFormats.Writers.delete_axis! -
DataAxesFormats.Writers.delete_matrix! -
DataAxesFormats.Writers.delete_scalar! -
DataAxesFormats.Writers.delete_vector! -
DataAxesFormats.Writers.empty_dense_matrix! -
DataAxesFormats.Writers.empty_dense_vector! -
DataAxesFormats.Writers.empty_sparse_matrix! -
DataAxesFormats.Writers.empty_sparse_vector! -
DataAxesFormats.Writers.relayout_matrix! -
DataAxesFormats.Writers.set_matrix! -
DataAxesFormats.Writers.set_scalar! -
DataAxesFormats.Writers.set_vector! -
DataAxesFormats.Computations.@computation -
DataAxesFormats.GenericLogging.@logged -
DataAxesFormats.MatrixLayouts.@assert_matrix -
DataAxesFormats.MatrixLayouts.@assert_vector -
DataAxesFormats.Queries.@q_str -
DataAxesFormats.Registry.@query_operation