Commit graph

393 commits

Author SHA1 Message Date
Orion Henry
e382e9d952 move automerge-js onto the applyPatches model 2022-09-28 17:08:24 -05:00
Orion Henry
f3befbfc98 cleanup for PR, remove comments, document, clean up patch merge 2022-09-28 17:01:02 -05:00
Orion Henry
53f5d9304f fmt & clippy 2022-09-06 15:16:49 -05:00
Orion Henry
3015c4bff2 map and array insert, delete for apply() 2022-09-06 15:09:43 -05:00
Orion Henry
7b354ba465 move op observer into transaction 2022-09-06 15:09:35 -05:00
+merlan #flirora
649b75deb1 Correct documentation for AutoSerde 2022-09-05 21:11:13 +01:00
Alex Good
eba7038bd2 Allow for empty head indices when decoding doc
The compressed document format includes at the end of the document chunk
the indicies of the heads of the document. Older versions of the
javascript implementation do not include these indicies so we allow them
to be omitted when decoding.

Whilst we're here add some tracing::trace logs to make it easier to
understand where parsing is failing.
2022-09-02 14:59:51 +01:00
Alex Good
e295a55b41 Add #[derive(Eq)] to satisfy clippy
The latest clippy (90.1.65 for me) added a lint which checks for types
that implement `PartialEq` and could implement `Eq`
(`derive_partial_eq_without_eq`). Add a `derive(Eq)` in a bunch of
places to satisfy this lint.
2022-09-01 12:24:00 +01:00
Alex Good
a0eb4218d8
Update docs for Transaction::put
Fixes #420
2022-08-27 11:59:14 +01:00
Alex Good
9ac8827219
Remove storage-v2 feature flag
Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:21:21 +01:00
Alex Good
9c86c09aaa
Rename Change::compressed_bytes -> Change::bytes 2022-08-22 21:18:11 +01:00
Alex Good
252a7eb8a5
Add automerge::Automerge::save_nocompress
For some usecases the overhead of compressed columns in the document
format is not worth it. Add `Automerge::save_nocompress` to save without
compressing columns.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
34e919a4c8
Plumb in storage-v2
This is achieved by liberal use of feature flags. Main additions are:

* Build the OpSet more efficiently when loading from compressed
  document storage using a DocObserver as implemented in
  `automerge::op_tree::load`
* Reimplement the parsing login in the various types in
  `automerge::sync`

There are numerous other small changes required to get the types to line
up.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
fc7657bcc6
Add a wrapper to implement Deserialize for Automerge
It is useful to be able to generate a `serde::Value` representation of
an automerge document. We can do this without an intermediate type by
iterating over the keys of the document recursively. Add
`autoeserde::AutoSerde` to implement this.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
771733deac
Implement storage-v2
Implement parsing the binary format using the new parser library and the
new encoding types. This is superior to the previous parsing
implementation in that invalid data should never cause panics and it
exposes and interface to construct an OpSet from a saved document much
more efficiently.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:47 +01:00
Alex Good
3a3df45b85
Access change fields through field accessors
The representation of changes in storage-v2 is different to the existing
representation so add accessor methods to the fields of `Change` and
make all accesses go through them. This allows the change representation
in storage-v2 to be a drop-in.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:16:42 +01:00
Alex Good
de997e2c50
Reimplement columnar decoding types
The existing implementation of the columnar format elides a lot of error
handling (by converting `Err` to `None`) and doesn't allow writing to a
single chunk of memory when encoding. Implement a new set of encoding and
decoding primitives which handle errors more robustly and allow us to
use a single chunk of memory when reading and writing.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
782f351322
Add types to convert between different Op types
Op IDs in the OpSet are represented using an index into a set of actor
IDs. This is efficient but requires conversion when reading and
writing from storage (where the set of actors might be different from
ths in the OpSet). Add a trait for converting between different
representations of an OpID.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
e1295b9daa
Add a simple parser combinator library
We have parsing needs which are slightly more complex than just reading
stuff from a buffer, but not complex enough to justify a dependency on a
parsing library. Implement a simple parser combinator library for use in
parsing the binary storage format.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:06:35 +01:00
Alex Good
d785c319b8
Add ScalarValue::Unknown
The colunar storage format allows for values which we do not know the
type of. In order that we can handle these types in a forward compatible
way we add ScalarValue::Unknown.

Signed-off-by: Alex Good <alex@memoryandthought.me>
2022-08-22 21:04:19 +01:00
Jason Kankiewicz
a22afdd70d Expose automerge::AutoCommit::get_change_by_hash()
as `AMgetChangeByHash()`.
Add the `AM_CHANGE_HASH_SIZE` macro define constant for
`AMgetChangeByHash()`.
Replace the literal `32` with the `automerge::types::HASH_SIZE` constant.
Expose `automerge::AutoCommit::splice()` as `AMsplice()`.
Add the `automerge::error::AutomergeError::InvalidValueType` variant for
`AMsplice()`.
Add push functionality to `AMspliceText()`.
Fix some documentation content bugs.
Fix some documentation formatting bugs.
2022-08-06 15:04:46 -07:00
Jason Kankiewicz
877744d40b Add equality comparison to the AM* types from
which it was missing.
Add equality comparison to `automerge::sync::message`.
Defer `std::ffi::CString` creation until necessary.
2022-07-25 01:33:50 -07:00
Alex Good
d71a734e49 Add OpIds to enforce ordering of Op::succ and Op::pred
The ordering of opids in the successor and predecessors of an op is
relevant when encoding because inconsistent ordering changes the
hashgraph. This means we must maintain the invariant that opids are
encoded in ascending lamport order. We have been maintaining this
invariant in the encoding implementation - however, this is not ideal
because it requires allocating for every op in the change when we commit
a transaction.

Add `types::OpIds` and use it in place of `Vec<OpId>` for `Op::succ` and
`Op::pred`. `OpIds` maintains the invariant that the IDs it contains
must be ordered with respect to some comparator function - which is
always `OpSetMetadata::lamport_cmp`. Remove the sorting of opids in
SuccEncoder::append.
2022-07-17 20:58:47 +01:00
Adel Salakh
f14a61e581 Sort successors in SuccEncoder
Makes SuccEncoder sort successors in Lamport clock order.
Such an ordering is expected by automerge js when loading documents,
otherwise some documents fail to load with a "operation IDs are not in
ascending order" error.
2022-07-13 11:25:12 +01:00
Andrew Jeffery
6ea5982c16 Change parents to return result if objid is not an object
There is easy confusion when calling parents with the id of a scalar,
wanting it to get the parent object first but that is not implemented.
To get the parent object of a scalar id would mean searching every
object for the OpId which may get too expensive when lots of objects are
around, this may be reconsidered later but the result would still be
useful to indicate when the id doesn't exist in the document vs has no
parents.
2022-07-12 18:36:47 +01:00
Jason Kankiewicz
32baae1a31 Hoisted InvalidChangeHashSlice into the
`Automerge` namespace.
2022-06-20 01:09:50 -07:00
Jason Kankiewicz
ac3709e670 Hoisted InvalidActorId into the automerge
namespace.
2022-06-14 00:38:55 -07:00
Jason Kankiewicz
71d8a7e717 Removed the superfluous AutomergeError::HexDecode
variant.
2022-06-14 00:37:42 -07:00
Orion Henry
0c9e77b644 added a test to ensure we dont break counter serialization 2022-06-09 12:45:20 +02:00
Jerome Gravel-Niquet
b20d04b0f2
serialize Counter with it's current value instead of start value 2022-06-08 14:00:03 -04:00
Andrew Jeffery
a569611d83 Use clock_at for filter_changes 2022-05-26 19:03:09 +01:00
Andrew Jeffery
03a635a926 Extend last_sync_hashes 2022-05-26 19:03:09 +01:00
Andrew Jeffery
97a5144d59 Reduce the amount of shuffling data for changes_to_send 2022-05-26 19:03:09 +01:00
Andrew Jeffery
03289510d6 Remove cloning their_have in sync 2022-05-26 19:03:09 +01:00
Andrew Jeffery
dae6509e13 Update autocommit's apply_changes to take an iterator 2022-05-26 09:02:59 +01:00
Andrew Jeffery
587adf7418 Add Eq to ObjType 2022-05-24 09:48:55 +01:00
Orion Henry
c353abfe4e
Merge pull request #375 from jeffa5/get-changes-opt
Get changes opt
2022-05-22 10:30:24 -07:00
Andrew Jeffery
2c1a71e143 Use expect for getting clock 2022-05-20 18:01:46 +01:00
Andrew Jeffery
8b1c3c73cd Use BTreeSet for sync::State to allow deriving Hash 2022-05-20 16:13:10 +01:00
Andrew Jeffery
3a8e833187 Document num_ops on change 2022-05-20 10:05:08 +01:00
Andrew Jeffery
1355a024a7 Use actor_index to get state in update_history 2022-05-20 10:05:08 +01:00
Andrew Jeffery
e5b527e17d Remove old functions 2022-05-20 10:05:08 +01:00
Andrew Jeffery
36857e0f6b Store seq in clock to remove binary_search_by_key 2022-05-20 10:05:08 +01:00
Andrew Jeffery
b7c50e47b9 Just use get_changes_clock 2022-05-20 10:05:08 +01:00
Andrew Jeffery
933bf5ee07 Return an error when getting clock for missing hash 2022-05-20 10:05:08 +01:00
Andrew Jeffery
c2765885fd Maintain incremental clocks 2022-05-20 10:05:08 +01:00
Andrew Jeffery
5e088ee9e0 Document clock module and add merge function 2022-05-20 10:05:08 +01:00
Andrew Jeffery
1b34892585 Add num_ops to change to quickly get the len 2022-05-20 10:05:08 +01:00
Andrew Jeffery
0de37d292d Sort change results from clock search 2022-05-20 10:05:08 +01:00
Andrew Jeffery
b9a6b3129f Add method to get changes by clock 2022-05-20 10:05:08 +01:00