Software /
code /
prosody-modules
Annotate
mod_storage_xmlarchive/README.markdown @ 4930:13070c6a7ce8
mod_http_muc_log: Fix exception on lack of trailing slash in room path
A request to /room leads to the match call returning nil which in turn
calls nodeprep(nil). In Prosody 0.11.x this does nothing and simply
returns the nil, while in 0.12 it is an error.
Now it redirects to the calendar view at /room/ - even for non-existant
rooms.
Discovered at a deployment with http_paths = { muc_log = "/" } and
requests to /robots.txt and similar, which now result in a uses redirect
before returning 404.
author | Kim Alvefur <zash@zash.se> |
---|---|
date | Fri, 22 Apr 2022 14:29:32 +0200 |
parent | 4538:591c643d55b2 |
child | 5763:100110d539d3 |
rev | line source |
---|---|
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
1 --- |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
2 labels: |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
3 - 'Stage-Beta' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
4 - 'Type-Storage' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
5 - ArchiveStorage |
2359
33e85070dc60
mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents:
2291
diff
changeset
|
6 summary: XML file based archive storage |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
7 --- |
1782 | 8 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
9 Introduction |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
10 ============ |
1782 | 11 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
12 This module implements stanza archives using files, similar to the |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
13 default "internal" storage. Unlike "internal", it saves messages in two |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
14 files per day (and per user), one containing metadata and one containing |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
15 the actual messages in XML format (hence the name). |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
16 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
17 Splitting data per day improves performance for larger archives as it |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
18 does not have to look through data from other days. |
1782 | 19 |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
20 Configuration |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
21 ============= |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
22 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
23 To use this with [mod\_mam] add this to your config: |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
24 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
25 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
26 storage = { |
3451
7b97abf26612
mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents:
3434
diff
changeset
|
27 archive = "xmlarchive" |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
28 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
29 ``` |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
30 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
31 To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]: |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
32 |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
33 ``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
34 storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
35 muc_log = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
36 } |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
37 ``` |
1782 | 38 |
1969
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
39 Refer to [Prosodys data storage documentation][doc:storage] for more |
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1940
diff
changeset
|
40 information. |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
41 |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
42 Note that this module does not implement the "keyval" storage method and |
1940
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1803
diff
changeset
|
43 can't be used by anything other than archives. |
1782 | 44 |
1803
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
45 Compatibility |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1782
diff
changeset
|
46 ============= |
1782 | 47 |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
48 ------ --------------- |
3987
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
49 trunk Should work |
dddcf094af5c
mod_storage_xmlarchive: Update compat table
Kim Alvefur <zash@zash.se>
parents:
3460
diff
changeset
|
50 0.11 Works |
4538
591c643d55b2
mod_storage_xmlarchive: Insert micropauses in long-running queries
Kim Alvefur <zash@zash.se>
parents:
3987
diff
changeset
|
51 0.10 Should work |
591c643d55b2
mod_storage_xmlarchive: Insert micropauses in long-running queries
Kim Alvefur <zash@zash.se>
parents:
3987
diff
changeset
|
52 0.9 Does not work |
2291
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2048
diff
changeset
|
53 ------ --------------- |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
54 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
55 Conversion to or from internal storage |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
56 -------------------------------------- |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
57 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
58 This module stores data in a way that overlaps with the more recent |
2817
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2815
diff
changeset
|
59 archive support in `mod_storage_internal`, meaning e.g. [mod_migrate] |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
60 will not be able to cleanly convert to or from the `xmlarchive` format. |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
61 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
62 To mitigate this, an migration command has been added to |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
63 `mod_storage_xmlarchive`: |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
64 |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
65 ``` bash |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
66 prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
67 ``` |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
68 |
2818 | 69 Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2` |
2815
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2359
diff
changeset
|
70 for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the |
2855
7713cd4fff8f
mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents:
2854
diff
changeset
|
71 user or MUC room to be migrated, which can be repeated. |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
72 |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
73 ::: {.alert .alert-danger} |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
74 Since this is a destructive command, don't forget to backup your data |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
75 first. |
3460
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
76 |
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3459
diff
changeset
|
77 Prosody should *not* be running while converting data. |
3459
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
78 ::: |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
79 |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3451
diff
changeset
|
80 |
2854
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
81 Data structure |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
82 ============== |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
83 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
84 Data is split in three kinds of files and messages are grouped by day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
85 Prosodys `util.datamanager` is used, so all special characters in these |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
86 filenames are escaped and reside under `hostname/store` in Prosodys Data |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
87 directory, commonly `/var/lib/prosody`. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
88 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
89 `username.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
90 : A list of dates in `YYYY-MM-DD` format. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
91 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
92 `username@YYYY-MM-DD.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
93 : Index containing metadata for messages stored on that day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
94 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
95 `username@YYYY-MM-DD.xml` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
96 : Messages in textual XML format, separated by newlines. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
97 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
98 This makes it fairly simple and fast to find messages by timestamp. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
99 Queries that are not time based, but limited to a specific contact may |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
100 be expensive as potentially the entire archive will be read. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
101 |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
102 Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2818
diff
changeset
|
103 archive id just as simple as time based queries. |
3434
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
104 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
105 ## Limitations |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
106 |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
107 - Only XML stanzas can be stored. |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2855
diff
changeset
|
108 - The deletion method only supports removing entire days at a time. |