prosody-modules
fe8222112cf4
mod_storage_xmlarchive/README.md

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody-modules

Annotate

mod_storage_xmlarchive/README.md @ 6199:fe8222112cf4

mod_conversejs: Serve base app at / This makes things slightly less awkward for the browser to figure out which URLs belong to a PWA. The app's "start URL" was previously without the '/' and therefore was not considered within the scope of the PWA. Now the canonical app URL will always have a '/'. Prosody/mod_http should take care of redirecting existing links without the trailing / to the new URL. If you have an installation at https://prosody/conversejs then it is now at https://prosody/conversejs/ (the first URL will now redirect to the second URL if you use it). The alternative would be to make the PWA scope include the parent, i.e. the whole of https://prosody/ in this case. This might get messy if other PWAs are provided by the same site or Prosody installation, however.
author Matthew Wild <mwild1@gmail.com>
date Tue, 11 Feb 2025 13:18:38 +0000
parent 6010:cf877e16df70
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
1 ---
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
2 labels:
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
3 - 'Stage-Beta'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
4 - 'Type-Storage'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
5 - ArchiveStorage
2359
33e85070dc60 mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents: 2291
diff changeset
6 summary: XML file based archive storage
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
7 ---
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
9 Introduction
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
10 ============
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
11
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
12 This module implements stanza archives using files, similar to the
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
13 default "internal" storage. Unlike "internal", it saves messages in two
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
14 files per day (and per user), one containing metadata and one containing
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
15 the actual messages in XML format (hence the name).
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
16
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
17 Splitting data per day improves performance for larger archives as it
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
18 does not have to look through data from other days.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
20 Configuration
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
21 =============
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
22
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
23 To use this with [mod\_mam] add this to your config:
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
24
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
25 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
26 storage = {
3451
7b97abf26612 mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents: 3434
diff changeset
27 archive = "xmlarchive"
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
28 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
29 ```
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
30
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]:
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
32
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
33 ``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
34 storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
35 muc_log = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
36 }
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
37 ```
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
38
1969
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
39 Refer to [Prosodys data storage documentation][doc:storage] for more
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1940
diff changeset
40 information.
1803
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
41
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1782
diff changeset
42 Note that this module does not implement the "keyval" storage method and
1940
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1803
diff changeset
43 can't be used by anything other than archives.
1782
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
44
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
45
6010
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
46 ### Conversion to or from internal storage
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
47
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
48 This module stores data in a way that overlaps with the more recent
2817
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2815
diff changeset
49 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate]
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
50 will not be able to cleanly convert to or from the `xmlarchive` format.
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
51
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
52 To mitigate this, an migration command has been added to
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
53 `mod_storage_xmlarchive`:
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
54
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
55 ``` bash
5764
5232d12eb74d mod_storage_xmlarchive: Pass hostname to converter for converting all users
Kim Alvefur <zash@zash.se>
parents: 5763
diff changeset
56 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID+
2815
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
57 ```
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2359
diff changeset
58
2818
88474dd1af48 Various READMEs: s/eg/e.g.g/
Kim Alvefur <zash@zash.se>
parents: 2817
diff changeset
59 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2`
5763
100110d539d3 mod_storage_xmlarchive: Migrate all users/rooms if no JID argument given
Kim Alvefur <zash@zash.se>
parents: 4538
diff changeset
60 for MAM and `muc_log` for MUC logs. Finally, `$JID` is one or more JID
5764
5232d12eb74d mod_storage_xmlarchive: Pass hostname to converter for converting all users
Kim Alvefur <zash@zash.se>
parents: 5763
diff changeset
61 of the users or MUC rooms to be migrated.
5232d12eb74d mod_storage_xmlarchive: Pass hostname to converter for converting all users
Kim Alvefur <zash@zash.se>
parents: 5763
diff changeset
62
5232d12eb74d mod_storage_xmlarchive: Pass hostname to converter for converting all users
Kim Alvefur <zash@zash.se>
parents: 5763
diff changeset
63 To migrate all users/rooms on a particular host, pass a bare hostname.
2854
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
64
3459
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
65 ::: {.alert .alert-danger}
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
66 Since this is a destructive command, don't forget to backup your data
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
67 first.
3460
f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents: 3459
diff changeset
68
f25b0623b276 mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents: 3459
diff changeset
69 Prosody should *not* be running while converting data.
3459
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
70 :::
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
71
471855552f49 mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents: 3451
diff changeset
72
2854
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
73 Data structure
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
74 ==============
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
75
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
76 Data is split in three kinds of files and messages are grouped by day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
77 Prosodys `util.datamanager` is used, so all special characters in these
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
78 filenames are escaped and reside under `hostname/store` in Prosodys Data
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
79 directory, commonly `/var/lib/prosody`.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
80
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
81 `username.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
82 : A list of dates in `YYYY-MM-DD` format.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
83
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
84 `username@YYYY-MM-DD.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
85 : Index containing metadata for messages stored on that day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
86
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
87 `username@YYYY-MM-DD.xml`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
88 : Messages in textual XML format, separated by newlines.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
89
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
90 This makes it fairly simple and fast to find messages by timestamp.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
91 Queries that are not time based, but limited to a specific contact may
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
92 be expensive as potentially the entire archive will be read.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
93
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
94 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2818
diff changeset
95 archive id just as simple as time based queries.
3434
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
96
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
97 ## Limitations
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
98
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
99 - Only XML stanzas can be stored.
a6722a35f35e mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents: 2855
diff changeset
100 - The deletion method only supports removing entire days at a time.
6010
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
101
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
102 Compatibility
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
103 =============
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
104
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
105 ------ ---------------
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
106 trunk Works
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
107 0.12 Works
cf877e16df70 mod_storage_xmlarchive: Reorganize Readme Layout
Menel <menel@snikket.de>
parents: 6003
diff changeset
108 ------ ---------------