Commit graph

972 commits

Author SHA1 Message Date
Alex Good
837c07b23a
Correctly encode compressed changes in sync messages
Sync messages encode changes as length prefixed byte arrays. We were
calculating the length using the uncompressed bytes of a change but
encoding the bytes of the change using the (possibly) compressed bytes.
This meant that if a change was large enough to compress then it  would
fail to decode. Switch to using uncompressed bytes in sync messages.
2022-10-02 18:59:41 +01:00
Alex Good
3d59e61cd6
Allow empty changes when loading document format
The logic for loading compressed document chunks has a check that the
`max_op` of a change is valid. This check was overly strict in that it
checked that the max op was strictly larger than the max op of a
previous strange - this rejects valid documents which contain changes
with no ops in them, in which case the max op can be equal to the max op
of the previous change. Loosen the logic to allow empty changes.
2022-09-30 19:00:48 +01:00
Alex Good
e57548f6e2
Fix broken encode/decode change
Previous ceremonies to appease clippy resulted in the
encodeChange/decodeChange wasm functions being slightly broken. Here we
fix them.
2022-09-29 15:49:31 -05:00
Alex Good
c7e370a1df
Appease clippy 2022-09-28 17:18:37 -05:00
Alex Good
427002caf3 Correctly load documents with deleted objects
The logic for reconstructing changes from the compressed document format
records operations which set a key in an object so that it can later
reconstruct delete operations from the successor list of the document
format operations. The logic to do this was only recording set
operations and not `make*` operations. This meant that delete operations
targeting `make*` operations could not be loaded correctly.

Correctly record `make*` operations for later use in constructing delete
operations.
2022-09-12 12:38:57 +01:00
Alex Good
fc9cb17b34
Use the local automerge-wasm in automerge-js tests
Somehow the `devDependencies` for `automerge-js` dependended on the
released `automerge-wasm` package, rather than the local version, which
means that the JS tests are not actually testing the current
implementation. Depend on the local `automerge-wasm` package to fix
this.
2022-09-08 16:27:30 +01:00
Alex Good
f586c82557 OpSet::visualise: add argument to filter by obj ID
Occasionally one needs to debug problems in a document with a large
number of objects. In this case it is unhelpful to print a graphviz of
the whole opset because there are too many objects. Add a
`Option<Vec<ObjId>>` argument to `OpSet::visualise` to filter the
objects which are visualised.
2022-09-08 12:48:53 +01:00
+merlan #flirora
649b75deb1 Correct documentation for AutoSerde 2022-09-05 21:11:13 +01:00
Alex Good
eba7038bd2 Allow for empty head indices when decoding doc
The compressed document format includes at the end of the document chunk
the indicies of the heads of the document. Older versions of the
javascript implementation do not include these indicies so we allow them
to be omitted when decoding.

Whilst we're here add some tracing::trace logs to make it easier to
understand where parsing is failing.
2022-09-02 14:59:51 +01:00
Alex Good
dd69f6f7b4
Add readme field to automerge/Cargo.toml 2022-09-01 12:27:34 +01:00
Alex Good
e295a55b41 Add #[derive(Eq)] to satisfy clippy
The latest clippy (90.1.65 for me) added a lint which checks for types
that implement `PartialEq` and could implement `Eq`
(`derive_partial_eq_without_eq`). Add a `derive(Eq)` in a bunch of
places to satisfy this lint.
2022-09-01 12:24:00 +01:00
Orion Henry
c2ed212dbc
Merge pull request #422 from automerge/fix-transaction-put-doc
Update docs for Transaction::put
2022-08-29 13:35:42 -05:00
Orion Henry
1817e98ec9
Merge pull request #418 from jkankiewicz/normalize_C_API_header_include
Expose `Vec<automerge::Change>` initialization and `automerge::AutoCommit::with_actor()` to the C API
2022-08-29 13:35:01 -05:00
Alex Good
a0eb4218d8
Update docs for Transaction::put
Fixes #420
2022-08-27 11:59:14 +01:00
Orion Henry
9879fd9342 copy pasta typo fix 2022-08-26 14:19:28 -05:00
Orion Henry
59bde120ee automerge-js adding trace to out of date errors 2022-08-26 14:17:56 -05:00
Jason Kankiewicz
22f720c465 Emphasize that an AMbyteSpan is only a view onto
the memory that it references.
2022-08-25 13:51:15 -07:00
Orion Henry
e6cd366aa0 automerge-js 0.1.12 2022-08-24 19:12:47 -05:00
Orion Henry
6d05cbd9e3 fix indexOf 2022-08-23 12:13:32 -05:00
Peter van Hardenberg
43bdd60904 the fields in a doc are not docs themselves 2022-08-23 09:31:09 -07:00
Orion Henry
363ad7d59a automerge-js ts fixes 2022-08-23 11:12:22 -05:00
Jason Kankiewicz
7da1832b52 Fix documentation bug caused by missing /. 2022-08-23 06:04:22 -07:00
Jason Kankiewicz
5e37ebfed0 Add AMchangesInit() for @rkuhn in #411.
Expose `automerge::AutoCommit::with_actor()` through `AMcreate()`.
Add notes to clarify the purpose of `AMfreeStack()`, `AMpop()`,
`AMpush()`, `AMpushCallback()`, and `AMresultStack`.
2022-08-23 05:34:45 -07:00
Jason Kankiewicz
1ed67a7658 Add missing documentation for the AMvalue.unknown
variant, the `AMunknownValue.bytes` member and the
`AMunknownValue.type_code` member.
2022-08-22 23:31:55 -07:00
Jason Kankiewicz
3ddde2fff2 Normalize the header include statement for all C
source files.
Normalize the header include statement within the documentation.
Limit `AMpush()` usage within the quickstart example to variable
assignment.
2022-08-22 22:28:23 -07:00
Orion Henry
b4705691c2
Merge pull request #355 from automerge/storage-v2
Storage v2
2022-08-22 18:18:50 -05:00
Alex Good
9ac8827219
Remove storage-v2 feature flag
Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:21:21 +01:00
Alex Good
9c86c09aaa
Rename Change::compressed_bytes -> Change::bytes 2022-08-22 21:18:11 +01:00
Jason Kankiewicz
632da04d60
Add the -DFEATURE_FLAG_STORAGE_V2 CMake option
for toggling the "storage-v2" feature flag in a Cargo invocation.
Correct the `AMunknownValue` struct misnomer.
Ease the rebasing of changes to the `AMvalue` struct declaration with
pending upstream changes to same.
2022-08-22 21:18:07 +01:00
Alex Good
8f2d4a494f
Test entire workspace for storage-v2 in CI
Now that all crates support the storage-v2 feature flag of the automerge
crate we update CI to run tests for '--workspace --all-features'

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:48 +01:00
Alex Good
db4cb52750
Add a storage-v2 feature flag to edit-trace
Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:48 +01:00
Alex Good
fc94d43e53
Expose storage-v2 in automerge-wasm
Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
d53d107076
Expose storage-v2 in automerge-c
Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
63dca26fe2
Additional tests for storage-v2
Various tests were required to cover edge cases in the new storage-v2
implementation.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
252a7eb8a5
Add automerge::Automerge::save_nocompress
For some usecases the overhead of compressed columns in the document
format is not worth it. Add `Automerge::save_nocompress` to save without
compressing columns.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
34e919a4c8
Plumb in storage-v2
This is achieved by liberal use of feature flags. Main additions are:

* Build the OpSet more efficiently when loading from compressed
  document storage using a DocObserver as implemented in
  `automerge::op_tree::load`
* Reimplement the parsing login in the various types in
  `automerge::sync`

There are numerous other small changes required to get the types to line
up.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
fc7657bcc6
Add a wrapper to implement Deserialize for Automerge
It is useful to be able to generate a `serde::Value` representation of
an automerge document. We can do this without an intermediate type by
iterating over the keys of the document recursively. Add
`autoeserde::AutoSerde` to implement this.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
771733deac
Implement storage-v2
Implement parsing the binary format using the new parser library and the
new encoding types. This is superior to the previous parsing
implementation in that invalid data should never cause panics and it
exposes and interface to construct an OpSet from a saved document much
more efficiently.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
3a3df45b85
Access change fields through field accessors
The representation of changes in storage-v2 is different to the existing
representation so add accessor methods to the fields of `Change` and
make all accesses go through them. This allows the change representation
in storage-v2 to be a drop-in.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:42 +01:00
Orion Henry
d28767e689 automerge-js v0.1.10 2022-08-22 15:13:08 -05:00
Alex Good
de997e2c50
Reimplement columnar decoding types
The existing implementation of the columnar format elides a lot of error
handling (by converting `Err` to `None`) and doesn't allow writing to a
single chunk of memory when encoding. Implement a new set of encoding and
decoding primitives which handle errors more robustly and allow us to
use a single chunk of memory when reading and writing.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
782f351322
Add types to convert between different Op types
Op IDs in the OpSet are represented using an index into a set of actor
IDs. This is efficient but requires conversion when reading and
writing from storage (where the set of actors might be different from
ths in the OpSet). Add a trait for converting between different
representations of an OpID.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
e1295b9daa
Add a simple parser combinator library
We have parsing needs which are slightly more complex than just reading
stuff from a buffer, but not complex enough to justify a dependency on a
parsing library. Implement a simple parser combinator library for use in
parsing the binary storage format.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
d785c319b8
Add ScalarValue::Unknown
The colunar storage format allows for values which we do not know the
type of. In order that we can handle these types in a forward compatible
way we add ScalarValue::Unknown.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:04:19 +01:00
Orion Henry
88f8976d0a automerge-js 0.1.9 2022-08-22 14:58:13 -05:00
Alex Good
56563a4a60
Add a storage-v2 feature flag
The new storage implementation is sufficiently large a change that it
warrants a period of testing. To facilitate testing the new and old
implementations side by side we slightly abuse cargo's feature flags and
add a storage-v2 feature which enables the new storage and disables the
old storage.

Note that this commit doesn't use `--all-features` when building the
workspace in scripts/ci/build-test. This will be rectified in a later
commit once the storage-v2 feature is integrated into the other crates
in the workspace.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-20 17:36:26 +01:00
Orion Henry
ff90327b52
Merge pull request #414 from jkankiewicz/port_WASM_basic_tests_to_C
Port the WASM API's basic tests to C
2022-08-11 19:00:11 -05:00
Orion Henry
ece66396e8
Merge pull request #417 from tombh/readme-update
Readme updates
2022-08-11 18:58:44 -05:00
Orion Henry
d1a926bcbe fix ownKeys bug in automerge-js 2022-08-11 18:49:42 -05:00
Orion Henry
1a955e1f0d fix some typescript errors - depricate default export of the wasm package 2022-08-11 18:24:21 -05:00