Software /
code /
prosody-modules
Annotate
mod_storage_xmlarchive/README.markdown @ 5408:3989c57cc551
mod_http_oauth2: Allow configuring links to policy and terms in metadata
These are for the Authorization Server, here the same as the XMPP
server.
author | Kim Alvefur <zash@zash.se> |
---|---|
date | Tue, 02 May 2023 17:01:02 +0200 |
parent | 4538:591c643d55b2 |
child | 5763:100110d539d3 |
rev | line source |
---|---|
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
1 --- |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
2 labels: |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
3 - 'Stage-Beta' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
4 - 'Type-Storage' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
5 - ArchiveStorage |
2359
33e85070dc60
mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents:
2291
diff
changeset
|
6 summary: XML file based archive storage |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
7 --- |
1782 | 8 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
9 Introduction |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
10 ============ |
1782 | 11 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
12 This module implements stanza archives using files, similar to the |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
13 default "internal" storage. Unlike "internal", it saves messages in two |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
14 files per day (and per user), one containing metadata and one containing |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
15 the actual messages in XML format (hence the name). |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
16 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
17 Splitting data per day improves performance for larger archives as it |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
18 does not have to look through data from other days. |
1782 | 19 |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
20 Configuration |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
21 ============= |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
22 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
23 To use this with [mod\_mam] add this to your config: |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
24 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
25 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
26 storage = { |
3451
7b97abf26612
mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents:
3434
diff
changeset
|
27 archive = "xmlarchive" |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
28 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
29 ``` |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
30 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]: |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
32 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
33 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
34 storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
35 muc_log = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
36 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
37 ``` |
1782 | 38 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
39 Refer to [Prosodys data storage documentation][doc:storage] for more |
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
40 information. |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
41 |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
42 Note that this module does not implement the "keyval" storage method and |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
43 can't be used by anything other than archives. |
1782 | 44 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
45 Compatibility |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
46 ============= |
1782 | 47 |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
48 ------ --------------- |
3987
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
49 trunk Should work |
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
50 0.11 Works |
4538
591c643d55b2
mod_storage_xmlarchive: Insert micropauses in long-running queries
Kim Alvefur <zash@zash.se>
parents:
3987
diff
changeset
|
51 0.10 Should work |
591c643d55b2
mod_storage_xmlarchive: Insert micropauses in long-running queries
Kim Alvefur <zash@zash.se>
parents:
3987
diff
changeset
|
52 0.9 Does not work |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
53 ------ --------------- |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
54 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
55 Conversion to or from internal storage |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
56 -------------------------------------- |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
57 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
58 This module stores data in a way that overlaps with the more recent |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
59 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate] |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
60 will not be able to cleanly convert to or from the `xmlarchive` format. |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
61 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
62 To mitigate this, an migration command has been added to |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
63 `mod_storage_xmlarchive`: |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
64 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
65 ``` bash |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
66 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
67 ``` |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
68 |
2818 | 69 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2` |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
70 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the |
2855
7713cd4fff8f
mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents:
2854
diff
changeset
|
71 user or MUC room to be migrated, which can be repeated. |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
72 |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
73 ::: {.alert .alert-danger} |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
74 Since this is a destructive command, don't forget to backup your data |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
75 first. |
3460
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
76 |
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
77 Prosody should *not* be running while converting data. |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
78 ::: |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
79 |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
80 |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
81 Data structure |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
82 ============== |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
83 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
84 Data is split in three kinds of files and messages are grouped by day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
85 Prosodys `util.datamanager` is used, so all special characters in these |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
86 filenames are escaped and reside under `hostname/store` in Prosodys Data |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
87 directory, commonly `/var/lib/prosody`. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
88 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
89 `username.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
90 : A list of dates in `YYYY-MM-DD` format. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
91 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
92 `username@YYYY-MM-DD.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
93 : Index containing metadata for messages stored on that day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
94 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
95 `username@YYYY-MM-DD.xml` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
96 : Messages in textual XML format, separated by newlines. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
97 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
98 This makes it fairly simple and fast to find messages by timestamp. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
99 Queries that are not time based, but limited to a specific contact may |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
100 be expensive as potentially the entire archive will be read. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
101 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
102 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
103 archive id just as simple as time based queries. |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
104 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
105 ## Limitations |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
106 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
107 - Only XML stanzas can be stored. |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
108 - The deletion method only supports removing entire days at a time. |