Commit graph

120 commits

Author SHA1 Message Date
Orion Henry
83a8499559 merge fix 2023-03-11 09:08:55 -06:00
Orion Henry
5f6b0e00be first draft of diff() api 2023-03-10 13:15:44 -06:00
Orion Henry
271d5cbead cleanup based on alex's comments 2023-03-10 12:57:13 -06:00
Orion Henry
aa0fdc7d2d Merge remote-tracking branch 'origin/main' into marks_port 2023-03-10 12:06:58 -06:00
Conrad Irwin
b34b46fa16
smaller automerge c (#545)
* Fix automerge-c tests on mac

* Generate significantly smaller automerge-c builds

This cuts the size of libautomerge_core.a from 25Mb to 1.6Mb on macOS
and 53Mb to 2.7Mb on Linux.

As a side-effect of setting codegen-units = 1 for all release builds the
optimized wasm files are also 100kb smaller.
2023-03-09 15:09:43 +00:00
Conrad Irwin
7b747b8341
Error instead of corrupt large op counters (#543)
Since b78211ca6, OpIds have been silently truncated to 2**32. This
causes corruption in the case the op id overflows.

This change converts the silent error to a panic, and guards against the
panic on the codepath found by the fuzzer.
2023-03-07 16:49:04 +00:00
Conrad Irwin
2c1970f664
Fix panic on invalid action (#541)
We make the validation on parsing operations in the encoded changes stricter to avoid a possible panic when applying changes.
2023-03-04 12:09:08 +00:00
christine betts
63b761c0d1
Suppress clippy warning in parse.rs + bump toolchain (#542)
* Fix rust error in parse.rs
* Bump toolchain to 1.67.0
2023-03-03 22:42:40 +00:00
Orion Henry
1c28e9656a add get_marks_at() 2023-02-28 17:17:14 -06:00
Orion Henry
8e818910d1 Merge remote-tracking branch 'origin/main' into marks_port 2023-02-28 14:47:44 -06:00
Orion Henry
ba491c6f72 move wasm observer into automerge 2023-02-28 13:25:07 -06:00
Conrad Irwin
44fa7ac416
Don't panic on missing deps of change chunks (#538)
* Fix doubly-reported ops in load of change chunks

Since c3c04128f5, observers have been
called twice when calling Automerge::load() with change chunks.

* Better handle change chunks with missing deps

Before this change Automerge::load would panic if you passed a change
chunk that was missing a dependency, or multiple change chunks not in
strict dependency order. After this change these cases will error
instead.
2023-02-27 20:12:09 +00:00
Orion Henry
1a539a3d79 add unmark / remove spans/raw_spans/attr 2023-02-27 12:33:55 -06:00
Jason Kankiewicz
8de2fa9bd4
C API 2 (#530)
The AMvalue union, AMlistItem struct, AMmapItem struct, and AMobjItem struct are gone, replaced by the AMitem struct.

The AMchangeHashes, AMchanges, AMlistItems, AMmapItems, AMobjItems, AMstrs, and AMsyncHaves iterators are gone, replaced by the AMitems iterator.

The AMitem struct is opaque, getting and setting values is now achieved exclusively through function calls.

The AMitemsNext(), AMitemsPrev(), and AMresultItem() functions return a pointer to an AMitem struct so you ultimately get the same thing whether you're iterating over a sequence or calling AMmapGet() or AMlistGet().

Calling AMitemResult() on an AMitem struct will produce a new AMresult struct referencing its storage so now the AMresult struct for an iterator can be subsequently freed without affecting the AMitem structs that were filtered out of it.

The storage for a set of AMitem structs can be recombined into a single AMresult struct by passing pointers to their corresponding AMresult structs to AMresultCat().

For C/C++ programmers, I've added AMstrCmp(), AMstrdup(), AM{idxType,objType,status,valType}ToString() and AM{idxType,objType,status,valType}FromString(). It's also now possible to pass arbitrary parameters through AMstack{Item,Items,Result}() to a callback function.
2023-02-25 18:47:00 +00:00
Orion Henry
af9b006bb0 marks name->key, range->start,end, patch callback propogates expceitons and has heads 2023-02-24 17:57:45 -06:00
Orion Henry
01c721e640 remove a fixme 2023-02-24 14:10:29 -06:00
Orion Henry
3beccfb5ee track mark data by reference 2023-02-24 13:38:10 -06:00
Orion Henry
9a6840392e let insert take a conflict flag 2023-02-24 12:26:11 -06:00
Orion Henry
d7d2c29dc7
Merge branch 'main' into marks_port 2023-02-24 09:59:19 -08:00
Orion Henry
2c6e54390b rewrote observe_current_state - added get_marks() api call - cleaned up js interface 2023-02-24 11:36:14 -06:00
Orion Henry
d7f93c5aca get mark patches working on load 2023-02-23 17:51:45 -06:00
Orion Henry
e1d81e01fc mark patch callbacks and js mark api 2023-02-16 19:35:39 -06:00
Philip Schatz
407faefa6e
A few setup fixes (#529)
* include deno in dependencies

* install javascript dependencies

* remove redundant operation
2023-02-15 09:23:02 +00:00
Alex Good
c92d042c87 @automerge/automerge-wasm@0.1.24 and @automerge/automerge@2.0.2-alpha.2 2023-02-14 17:59:23 +00:00
Alex Good
c6a32d8368 Correct logic when skip = B and fix formatting
A few tests were failing which exposed the fact that if skip is `B` (the
out factor of the OpTree) then we set `skip = None` and this causes us
to attempt to return `Skip` in a non root node. I ported the failing
test from JS to Rust and fixed the problem.

I also fixed the formatting issues.
2023-02-14 11:29:55 -06:00
Orion Henry
a9612371e0 rework how skip works to push the logic into node 2023-02-14 11:29:55 -06:00
Conrad Irwin
e2bb0eb6b9 Use our leb128 parser for values
This ensures that values in automerge documents are encoded correctly,
and that no extra data is smuggled in any LEB fields.
2023-02-14 11:29:55 -06:00
Alex Good
9271b20cf5 Correct logic when skip = B and fix formatting
A few tests were failing which exposed the fact that if skip is `B` (the
out factor of the OpTree) then we set `skip = None` and this causes us
to attempt to return `Skip` in a non root node. I ported the failing
test from JS to Rust and fixed the problem.

I also fixed the formatting issues.
2023-02-14 17:21:59 +00:00
Orion Henry
5e82dbc3c8 rework how skip works to push the logic into node 2023-02-14 17:21:59 +00:00
Orion Henry
a006a32e3f get tests passing + unmark 2023-02-13 10:56:05 -06:00
Conrad Irwin
2cd7427f35 Use our leb128 parser for values
This ensures that values in automerge documents are encoded correctly,
and that no extra data is smuggled in any LEB fields.
2023-02-09 15:46:22 +00:00
Alex Good
a02f70f2b8
Use new columns instead of existing ones
The previous approach of using the key and insert columns of existing
ops was leading to quite confusing code. There's no real cost to
introducing new columns so I've switched the code to do that instead.

Introduce an `expand` and a `mark_name` column. `expand` is a boolean
column and `mark_name` is a RLE encoded string column. Neither of these
columns are encoded if they are empty.

Also move the `MarkData::name` property to use strings interned in
`OpSetMetadata::props` rather than representing the string directly on
the basis that we probably will have a lot of repeated mark names and
we do a bunch of equality checks on them while searching so this will
probably speed things up a bit.

Introduced a new `MaybeBooleanEncoder` (and associated `MaybeBooleanDecoder` and
`MaybeBooleanRange`) types to represent a boolean column which is
entirely skipped if all it contains are `false` values. This allows us
to omit encoding the `expand` column for groups of ops which only ever
set it to `false` which in turn makes us backwards compatible when not
using marks.
2023-02-09 15:09:52 +00:00
Alex Good
f281213a47
tests passing 2023-02-09 10:55:19 +00:00
Alex Good
a44ceacb1c
everything compiles 2023-02-09 10:26:26 +00:00
Orion Henry
290c9e6872 attempt to finish - two issues outstanding 2023-02-08 14:59:41 -06:00
Alex Good
9a7dba09a4
wip 2023-02-08 18:20:38 +00:00
Alex Good
2345176526
DocOp 2023-02-08 17:53:58 +00:00
Orion Henry
9bc424d776 first test passing - needs serialization 2023-02-08 10:43:33 -06:00
Orion Henry
61f9604d0c merge with main 2023-02-06 11:34:13 -06:00
Alex Good
a24d536d16 Move automerge::SequenceTree to automerge_wasm::SequenceTree
The `SequenceTree` is only ever used in `automerge_wasm` so move it
there.
2023-02-05 11:08:33 +00:00
Alex Good
c5fde2802f @automerge/automerge-wasm@0.1.24 and @automerge/automerge@2.0.2-alpha.1 2023-02-03 16:31:46 +00:00
Alex Good
13a775ed9a Speed up loading by generating clocks on demand
Context: currently we store a mapping from ChangeHash -> Clock, where
`Clock` is the set of (ActorId, (Sequence number, max Op)) pairs derived
from the given change and it's dependencies. This clock is used to
determine what operations are visible at a given set of heads.

Problem: populating this mapping for documents with large histories
containing many actors can be very slow as for each change we have to
allocate and merge a bunch of hashmaps.

Solution: instead of creating the clocks on load, create an adjacency
list based representation of the change graph and then derive the clock
from this graph when it is needed. Traversing even large graphs is still
almost as fast as looking up the clock in a hashmap.
2023-02-03 16:15:15 +00:00
Alex Good
1e33c9d9e0 Use Automerge::load instead of load_incremental if empty
Problem: when running the sync protocol for a new document the API
requires that the user create an empty document and then call
`receive_sync_message` on that document. This results in the OpObserver
for the new document being called with every single op in the document
history. For documents with a large history this can be extremely time
consuming, but the OpObserver doesn't need to know about all the hidden
states.

Solution: Modify `Automerge::load_with` and
`Automerge::apply_changes_with` to check if the document is empty before
applying changes. If the document _is_ empty then we don't call the
observer for every change, but instead use
`automerge::observe_current_state` to notify the observer of the new
state once all the changes have been applied.
2023-02-03 10:01:12 +00:00
Alex Good
c3c04128f5 Only observe the current state on load
Problem: When loading a document whilst passing an `OpObserver` we call
the OpObserver for every change in the loaded document. This slows down
the loading process for two reasons: 1) we have to make a call to the
observer for every op 2) we cannot just stream the ops into the OpSet in
topological order but must instead buffer them to pass to the observer.

Solution: Construct the OpSet first, then only traverse the visible ops
in the OpSet, calling the observer. For documents with a deep history
this results in vastly fewer calls to the observer and also allows us to
construct the OpSet much more quickly. It is slightly different
semantically because the observer never gets notified of changes which
are not visible, but that shouldn't matter to most observers.
2023-02-03 10:01:12 +00:00
Alex Good
da55dfac7a refactor: make fields of Automerge private
The fields of `automerge::Automerge` were crate public, which made it
hard to change the structure of `Automerge` with confidence. Make all
fields private and put them behind accessors where necessary to allow
for easy internal changes.
2023-02-03 10:01:12 +00:00
alexjg
9195e9cb76
Fix deny errors (#518)
* Ignore deny errors on duplicate windows-sys

* Delete spurious lockfile in automerge-cli
2023-02-02 15:02:53 +00:00
Conrad Irwin
a6959e70e8
More robust leb128 parsing (#515)
Before this change i64 decoding did not work for negative numbers (not a
real problem because it is only used for the timestamp of a change),
and both u64 and i64 would allow overlong LEB encodings.
2023-01-31 17:54:54 +00:00
alexjg
de5af2fffa
automerge-rs 0.3.0 and automerge-test 0.2.0 (#512) 2023-01-30 19:58:35 +00:00
alexjg
08801ab580
automerge-rs: Introduce ReadDoc and SyncDoc traits and add documentation (#511)
The Rust API has so far grown somewhat organically driven by the needs of the
javascript implementation. This has led to an API which is quite awkward and
unfamiliar to Rust programmers. Additionally there is no documentation to speak
of. This commit is the first movement towards cleaning things up a bit. We touch
a lot of files but the changes are all very mechanical. We introduce a few
traits to abstract over the common operations between `Automerge` and
`AutoCommit`, and add a whole bunch of documentation.

* Add a `ReadDoc` trait to describe methods which read value from a document.
  make `Transactable` extend `ReadDoc`
* Add a `SyncDoc` trait to describe methods necessary for synchronizing
  documents.
* Put the `SyncDoc` implementation for `AutoCommit` behind `AutoCommit::sync` to
  ensure that any open transactions are closed before taking part in the sync
  protocol
* Split `OpObserver` into two traits: `OpObserver` + `BranchableObserver`.
  `BranchableObserver` captures the methods which are only needed for observing
  transactions.
* Add a whole bunch of documentation.

The main changes Rust users will need to make is:

* Import the `ReadDoc` trait wherever you are using the methods which have been
  moved to it. Optionally change concrete paramters on functions to `ReadDoc`
  constraints.
* Likewise import the `SyncDoc` trait wherever you are doing synchronisation
  work
* If you are using the `AutoCommit::*_sync_message` methods you will need to add
  a call to `AutoCommit::sync()` first. E.g. `doc.generate_sync_message` becomes
  `doc.sync().generate_sync_message`
* If you have an implementation of `OpObserver` which you are using in an
  `AutoCommit` then split it into an implementation of `OpObserver` and
  `BranchableObserver`
2023-01-30 19:37:03 +00:00
alexjg
58a7a06b75
@automerge/automerge-wasm@0.1.23 and @automerge/automerge@2.0.1-alpha.6 (#509) 2023-01-27 20:27:11 +00:00