12cdeee7ec
Rename `count` to `numMessagesPerBatch`
7 years ago
38ac9972e8
Improve termination condition
7 years ago
5b1a0c45df
Replace `getAll` with cursor
...
This reduces the likelihood of IPC message size error.
7 years ago
ed77378293
Fix bug in termination condition
7 years ago
052fb841f6
Allow database name override in migrations
7 years ago
1e04083813
Support database upgrades in `Database.open`
7 years ago
f50ac1c99e
Reduce excessive logging about database status
7 years ago
77b1f3c505
Replace startup with attachment background migration
...
Iterates through `Message::id` index.
7 years ago
057762806e
Add support for attachment background migration without index
7 years ago
2e4893f4c1
Clarify implications of introducing new migrations
7 years ago
7413b787b6
Make `migrations` private
7 years ago
b07c66eaa7
Generate attachments from local files
...
This ensures our benchmarking is closer to real-world usage, e.g. images, video,
etc. that are not compressible.
7 years ago
064a40544c
Fix error message formatting
7 years ago
442f5d7295
Use `done` to match JavaScript iterators
...
See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators
7 years ago
a4f0c5242f
Document why batch size is one
7 years ago
805031ade8
Conditionally run post-attachment migrations
...
Introduce placeholder migrations for Backbone models so they never implicitly
run migrations whenever they are `fetch`ed. We prefer to run our migrations
explicitly upon app startup and then let Backbone models be (slightly) dumb(er)
models, without inadvertently triggering migrations.
7 years ago
bfbeedab5c
Temporarily disable post-attachment migration migrations
7 years ago
d9be6a0f94
Destructure Lodash `require`s
7 years ago
b7b6195cfc
Extract IIFE into separate function
7 years ago
11f98474ba
Capture how many messages we have to process
7 years ago
0fdc1140dd
Add `Database.getCount` function
7 years ago
1f8556b049
Remove unused `createCollection`
7 years ago
c67c2a858a
Remove Backbone references for attachment migration
...
It has to run without any other migrations interfering.
7 years ago
0c40f35623
Document disadvantage of fetching messages without index
7 years ago
d3c9de4712
Add debug module
...
Lets us generate large conversations with large attachments.
7 years ago
30037e5308
Reduce attachment migration batch size to 1
...
This prevents ‘Maximum IPC message size exceeded’ due to IDB `getAll` operation.
- https://github.com/zincbase/zincdb/issues/17
- https://cs.chromium.org/chromium/src/content/browser/indexed_db/indexed_db_database.cc?q=%22Maximum+IPC+message+size+exceeded%22&sq=package:chromium&l=1160
7 years ago
21147a20a0
Add `sleep` module
7 years ago
08f73b8420
Remove last processed index after attachment migration
7 years ago
696a144ab7
Add `settings.deleteItem`
7 years ago
5910f84af4
Remove outdated documentation
7 years ago
efe3cd67fc
Allow attachment migration run on higher database version
7 years ago
921c3dba7c
Skip migrations that have already been applied
7 years ago
417511ffd2
Add `database.getVersion`
7 years ago
c5c94bc3ab
Extract `getMigrationVersions`
7 years ago
6aea36240d
Rename `closeDatabase` to `closeDatabaseConnection`
7 years ago
016432826b
Extract `database` and `settings` modules
7 years ago
5bea894abd
Close database connection via Backbone IDB adapter
7 years ago
f50e9ae364
Log closing connection of database
7 years ago
4ff8bc3357
Use `camelCase` for non-constructors
7 years ago
3720c3f3bb
Improve log message
7 years ago
ce5b450fdb
Log `targetSchemaVersion`
7 years ago
f7f24b5822
Log total number of processed messages
7 years ago
eca930770c
Remove hard-coded database connection settings
7 years ago
1df6dc8378
Abort processing if there are no more messages
7 years ago
d5d0eabdfd
Remove usage of `storage` module
7 years ago
8966e80284
Improve identifier names
7 years ago
3e2d575506
Document `MessageDataMigrator` module design
7 years ago
3c57dbfb56
Extract `NUM_MESSAGES_PER_BATCH`
7 years ago
85788d3c4a
Match `items` storage format to Backbone adapter
7 years ago
7de7fcf561
Avoid `no-param-reassign` violation
7 years ago
070235b59b
Implement `MessageDataMigrator.processAll`
...
Upgrades schema of all messags upon startup.
7 years ago
178a3cc262
Reduce work for verifying transaction completion
7 years ago
b6e978f74c
Implement `MessagesDataMigrator.processAll`
7 years ago
457bf7ab9d
Add `createCollection` function
7 years ago
8ea257ad4d
Use double quotes for identifiers in error messages
7 years ago
172616ca4f
Add log message for dummy migration 18
7 years ago
579b01283e
Replace `wrapDeferred` with `deferredToPromise`
7 years ago
40c40c800a
Prefer `exports`
7 years ago
da144edc56
Manually close database connection after migration
7 years ago
106ce21c49
Remove redundant log message
7 years ago
fcd30cd919
Close database after migration
...
This is not 100% reliable as database connections are closed in a separate
thread according to the documentation:
- https://developer.mozilla.org/en-US/docs/Web/API/IDBDatabase/close
- https://stackoverflow.com/a/18639298
- 80c7a06d5c/backbone-indexeddb.js (L558-L565)
7 years ago
c765422fa1
Extract `deferredToPromise`
7 years ago
e2f1339ab9
Explicitly run post-attachment migrations
7 years ago
d7c8d33edb
Extract `runMigrations`
7 years ago
d16178638e
Split database migrations into pre- and post-attachment migration
...
- Run light-weight migrations before attachment migration.
- Run regular migrations after attachments have been moved to disk.
7 years ago
90de9d85a3
Remove type annotation
7 years ago
4c10fcfa17
Reduce threshold until optimization appears to 1s
7 years ago
add19aa732
Add initialization view
7 years ago
9b112489cd
Upgrade 2 messages every 30 seconds
...
This results in 4 messages per minute or 240 messages per hour.
7 years ago
7c2fde4433
Add `Signal.Workflow` namespace
7 years ago
4a7cc22de3
🎨 Use `window.requestIdleCallback`
7 years ago
4651cbc2eb
Improve ``Message.upgradeSchema` preconditions
7 years ago
ca28f3030b
Add `MessageMigration` module
7 years ago
8e762797c0
Remove attachment validation
...
Turns out `fileName` is optional and maybe even `contentType`. We should revisit
this at a later point in time.
7 years ago
09ee2d4ea2
Switch to using `requestIdleCallback`
...
Example:
https://developer.mozilla.org/en-US/docs/Web/API/Background_Tasks_API#Example
7 years ago
d13668544d
Rename `IdleListener` to `IdleDetector`
7 years ago
3140e4d66d
Implement basic `IdleListener`
7 years ago
867f73b80a
Use double quotes for identifiers in error messages
7 years ago
70cdd2b350
Add `stringToArrayBuffer` utility
...
This way we can avoid an extra dependency.
7 years ago
2ad0fd7db6
Use Lodash `omit` instead of `delete`
7 years ago
279b3f81c7
Add `Attachment.deleteData`
7 years ago
9a540d6d18
Load attachment data before sending message
7 years ago
6355c54114
Return relative path from `writeAttachmentData`
...
This will make our app more robust to changes in location on the file system.
7 years ago
2cd3d5ac16
Add `Message` schema version 3
7 years ago
aa82a2a5fb
Add `context` to `Message.upgradeSchema`
...
Allows us to pass in Electron/Node.js specific functions. This allows better
unit testing in Mocha where we don’t have access to Electron APIs.
7 years ago
ebe2a769c9
Add `Attachment.migrateDataToFileSystem`
7 years ago
cea42bde7d
Encryption support for backup and restore
...
Also moved to the _ prefix in backup.js for all private methods exported
for testing.
7 years ago
6d8f4b7b6e
Backup: zipped messages.json, flat attachments dir
...
Backup creates, in a target directory:
- An attachments folder, with all attachments, each named for their
parent message's id - a GUID. If there is more than one attachment
in a given message, each attachment beyond the first will end with
'-N', zero-indexed.
- A file named messages.zip. It contains exactly what went to disk in
the original export code, but zipped up.
Export is now only 'light,' and in this new messages.zip format.
Import supports both the new format and the old format. If the target
directory has a messages.zip file, we'll treat it as the new format.
Next up: Encrypting attachments and the messages.zip!
7 years ago
153a22f46b
Exclude unnecessary data in export: profiles, replayable errors
7 years ago
c0c758d459
Backup: All exports are now 'light' exports
7 years ago
e583434366
Refactor: Move Backup under window.Signal
7 years ago
c88381efe3
Use `async` / `await` to improve readability
7 years ago
2642844c27
Rewrite migration 17 without `idb`
...
We ran into issues when doing async operations inside of an IndexedDB
`onupgradeneeded` handler. The errors were ‘The transaction is not active’ or
‘Transaction has finished’. The following documentation confirmed that
transactions are committed/terminated when control returns to the event loop:
Spec
- https://www.w3.org/TR/IndexedDB/#transaction-lifetime-concept
- https://www.w3.org/TR/IndexedDB/#upgrade-transaction-construct
Stack Overflow
- https://stackoverflow.com/a/11059085
- https://stackoverflow.com/a/27338944
Since the initial database migration is so critical, I decided to avoid `idb`
with promise support for IndexedDB for now, but will reconsider using it for
other tasks in the future to improve readability of IndexedDB code.
7 years ago
a619d48fac
Update schema version history
7 years ago
0e20e8e2ea
Use `Errors.toLogFormat`
7 years ago
a5edbf8328
Initialize schema as first step in `upgradeSchema`
7 years ago
5d927b73e6
Use `while` loop for IDB cursor iteration
...
Previously, I messily combined promises and callbacks because I thought we
were affected by the microtask issue:
https://github.com/gasi/idb#iteratecursor--iteratekeycursor
ESLint’s `more/no-then` encouraged me to revisit this and it works as expected.
7 years ago
182e6ffe10
Add version 17 migration
7 years ago
752c8f97e6
🎨 Format ternaries
7 years ago
8dfaa5619f
Prefix private functions with underscore
7 years ago
Daniel Gasienica
Scott Nonnenberg
Scott Nonnenberg