prosody-modules
281a864e7472
mod_storage_xmlarchive/README.markdown

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody-modules

Annotate

mod_storage_xmlarchive/README.markdown @ 4282:281a864e7472

mod_pubsub_feeds: Don't skip publishing items after an existing one I encountered a feed which was backwards, such that older entries were considered first and then it would skip newer entries. This may however run into trouble if the feed contains more items than what's persisted in pubsub.
author Kim Alvefur <zash@zash.se>
date Mon, 30 Nov 2020 15:17:29 +0100
parent 3987:dddcf094af5c
child 4538:591c643d55b2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
1 ---
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
2 labels:
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
3 - 'Stage-Beta'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
4 - 'Type-Storage'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
5 - ArchiveStorage
2359
33e85070dc60 mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents: 2291
diff changeset
6 summary: XML file based archive storage
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
7 ---
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
9 Introduction
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
10 ============
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
11
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
12 This module implements stanza archives using files, similar to the
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
13 default "internal" storage. Unlike "internal", it saves messages in two
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
14 files per day (and per user), one containing metadata and one containing
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
15 the actual messages in XML format (hence the name).
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
16
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
17 Splitting data per day improves performance for larger archives as it
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
18 does not have to look through data from other days.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
20 Configuration
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
21 =============
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
22
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
23 To use this with [mod\_mam] add this to your config:
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
24
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
25 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
26 storage = {
3451
7b97abf26612 mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents: 3434
diff changeset
27 archive = "xmlarchive"
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
28 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
29 ```
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
30
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]:
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
32
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
33 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
34 storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
35 muc_log = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
36 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
37 ```
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
38
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
39 Refer to [Prosodys data storage documentation][doc:storage] for more
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
40 information.
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
41
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
42 Note that this module does not implement the "keyval" storage method and
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
43 can't be used by anything other than archives.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
44
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
45 Compatibility
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
46 =============
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
47
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
48 ------ ---------------
3987
dddcf094af5c mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents: 3460
diff changeset
49 trunk Should work
dddcf094af5c mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents: 3460
diff changeset
50 0.11 Works
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
51 0.10 Works
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
52 0.9 Should work
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
53 0.8 Does not work
2291
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2048
diff changeset
54 ------ ---------------
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
55
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
56 Conversion to or from internal storage
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
57 --------------------------------------
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
58
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
59 This module stores data in a way that overlaps with the more recent
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
60 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate]
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
61 will not be able to cleanly convert to or from the `xmlarchive` format.
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
62
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
63 To mitigate this, an migration command has been added to
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
64 `mod_storage_xmlarchive`:
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
65
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
66 ``` bash
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
67 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
68 ```
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
69
2818
88474dd1af48 Various READMEs: s/eg/e.g.g/
Kim Alvefur <zash@zash.se>
parents: 2817
diff changeset
70 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2`
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
71 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the
2855
7713cd4fff8f mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents: 2854
diff changeset
72 user or MUC room to be migrated, which can be repeated.
2854
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
73
3459
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
74 ::: {.alert .alert-danger}
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
75 Since this is a destructive command, don't forget to backup your data
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
76 first.
3460
f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents: 3459
diff changeset
77
f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents: 3459
diff changeset
78 Prosody should *not* be running while converting data.
3459
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
79 :::
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
80
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
81
2854
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
82 Data structure
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
83 ==============
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
84
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
85 Data is split in three kinds of files and messages are grouped by day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
86 Prosodys `util.datamanager` is used, so all special characters in these
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
87 filenames are escaped and reside under `hostname/store` in Prosodys Data
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
88 directory, commonly `/var/lib/prosody`.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
89
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
90 `username.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
91 : A list of dates in `YYYY-MM-DD` format.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
92
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
93 `username@YYYY-MM-DD.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
94 : Index containing metadata for messages stored on that day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
95
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
96 `username@YYYY-MM-DD.xml`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
97 : Messages in textual XML format, separated by newlines.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
98
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
99 This makes it fairly simple and fast to find messages by timestamp.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
100 Queries that are not time based, but limited to a specific contact may
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
101 be expensive as potentially the entire archive will be read.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
102
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
103 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
104 archive id just as simple as time based queries.
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
105
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
106 ## Limitations
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
107
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
108 - Only XML stanzas can be stored.
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
109 - The deletion method only supports removing entire days at a time.