prosody
9751c17f5281
plugins/mod_storage_internal.lua

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody

Annotate

plugins/mod_storage_internal.lua @ 9884:9751c17f5281

mod_storage_internal,_sql: Expose archive capabilities feature set This was planned to be added long ago but was forgotten.
author Kim Alvefur <zash@zash.se>
date Fri, 22 Mar 2019 18:01:24 +0100
parent 9883:f76bd399267c
child 9885:64e16d1e91f6
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
1 local cache = require "util.cache";
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
2 local datamanager = require "core.storagemanager".olddm;
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
3 local array = require "util.array";
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
4 local datetime = require "util.datetime";
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
5 local st = require "util.stanza";
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
6 local now = require "util.time".now;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
7 local id = require "util.id".medium;
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
8
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
9 local host = module.host;
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
10
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
11 local archive_item_limit = module:get_option_number("storage_archive_item_limit", 1000);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
12 local archive_item_count_cache = cache.new(module:get_option("storage_archive_item_limit_cache_size", 1000));
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
13
5122
b41c33dc7c36 mod_storage_*: Don't explicitly set driver name, to ease copying/renaming modules.
Waqas Hussain <waqas20@gmail.com>
parents: 5121
diff changeset
14 local driver = {};
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
15
5153
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents: 5133
diff changeset
16 function driver:open(store, typ)
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
17 local mt = self[typ or "keyval"]
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
18 if not mt then
6283
7cf6d3a2c855 mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types
Kim Alvefur <zash@zash.se>
parents: 5153
diff changeset
19 return nil, "unsupported-store";
7cf6d3a2c855 mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types
Kim Alvefur <zash@zash.se>
parents: 5153
diff changeset
20 end
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
21 return setmetatable({ store = store, type = typ }, mt);
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
22 end
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
23
8020
342ce07836de mod_storage_internal: Ignore unused 'self' argument [luacheck]
Kim Alvefur <zash@zash.se>
parents: 8019
diff changeset
24 function driver:stores(username) -- luacheck: ignore 212/self
8019
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
25 return datamanager.stores(username, host);
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
26 end
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
27
8020
342ce07836de mod_storage_internal: Ignore unused 'self' argument [luacheck]
Kim Alvefur <zash@zash.se>
parents: 8019
diff changeset
28 function driver:purge(user) -- luacheck: ignore 212/self
8019
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
29 return datamanager.purge(user, host);
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
30 end
925098aad268 mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents: 8018
diff changeset
31
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
32 local keyval = { };
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
33 driver.keyval = { __index = keyval };
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
34
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
35 function keyval:get(user)
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
36 return datamanager.load(user, host, self.store);
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
37 end
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
38
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
39 function keyval:set(user, data)
4085
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
40 return datamanager.store(user, host, self.store, data);
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
41 end
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
42
8018
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents: 6283
diff changeset
43 function keyval:users()
5153
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents: 5133
diff changeset
44 return datamanager.users(host, self.store, self.type);
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents: 5133
diff changeset
45 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents: 5133
diff changeset
46
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
47 local archive = {};
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
48 driver.archive = { __index = archive };
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
49
9884
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
50 archive.caps = {
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
51 total = true;
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
52 quota = archive_item_limit;
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
53 truncate = true;
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
54 };
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents: 9883
diff changeset
55
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
56 function archive:append(username, key, value, when, with)
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
57 when = when or now();
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
58 if not st.is_stanza(value) then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
59 return nil, "unsupported-datatype";
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
60 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
61 value = st.preserialize(st.clone(value));
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
62 value.when = when;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
63 value.with = with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
64 value.attr.stamp = datetime.datetime(when);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
65 value.attr.stamp_legacy = datetime.legacy(when);
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
66
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
67 local item_count = archive_item_count_cache:get(username);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
68
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
69 if key then
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
70 local items, err = datamanager.list_load(username, host, self.store);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
71 if not items and err then return items, err; end
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
72
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
73 -- Check the quota
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
74 item_count = items and #items or 0;
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
75 archive_item_count_cache:set(username, item_count);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
76 if item_count >= archive_item_limit then
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
77 module:log("debug", "%s reached or over quota, not adding to store", username);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
78 return nil, "quota-limit";
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
79 end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
80
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
81 if items then
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
82 -- Filter out any item with the same key as the one being added
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
83 items = array(items);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
84 items:filter(function (item)
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
85 return item.key ~= key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
86 end);
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
87
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
88 value.key = key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
89 items:push(value);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
90 local ok, err = datamanager.list_store(username, host, self.store, items);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
91 if not ok then return ok, err; end
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
92 archive_item_count_cache:set(username, #items);
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
93 return key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
94 end
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
95 else
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
96 if not item_count then -- Item count not cached?
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
97 -- We need to load the list to get the number of items currently stored
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
98 local items, err = datamanager.list_load(username, host, self.store);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
99 if not items and err then return items, err; end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
100 item_count = items and #items or 0;
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
101 archive_item_count_cache:set(username, item_count);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
102 end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
103 if item_count >= archive_item_limit then
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
104 module:log("debug", "%s reached or over quota, not adding to store", username);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
105 return nil, "quota-limit";
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
106 end
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
107 key = id();
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
108 end
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
109
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
110 module:log("debug", "%s has %d items out of %d limit", username, item_count, archive_item_limit);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
111
8309
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
112 value.key = key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents: 8173
diff changeset
113
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
114 local ok, err = datamanager.list_append(username, host, self.store, value);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
115 if not ok then return ok, err; end
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
116 archive_item_count_cache:set(username, item_count+1);
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
117 return key;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
118 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
119
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
120 function archive:find(username, query)
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
121 local items, err = datamanager.list_load(username, host, self.store);
8173
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
122 if not items then
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
123 if err then
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
124 return items, err;
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
125 else
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
126 return function () end, 0;
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
127 end
3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents: 8143
diff changeset
128 end
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
129 local count = #items;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
130 local i = 0;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
131 if query then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
132 items = array(items);
8089
4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents: 8023
diff changeset
133 if query.key then
4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents: 8023
diff changeset
134 items:filter(function (item)
4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents: 8023
diff changeset
135 return item.key == query.key;
4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents: 8023
diff changeset
136 end);
4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents: 8023
diff changeset
137 end
8021
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
138 if query.with then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
139 items:filter(function (item)
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
140 return item.with == query.with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
141 end);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
142 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
143 if query.start then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
144 items:filter(function (item)
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
145 return item.when >= query.start;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
146 end);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
147 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
148 if query["end"] then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
149 items:filter(function (item)
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
150 return item.when <= query["end"];
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
151 end);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
152 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
153 count = #items;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
154 if query.reverse then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
155 items:reverse();
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
156 if query.before then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
157 for j = 1, count do
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
158 if (items[j].key or tostring(j)) == query.before then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
159 i = j;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
160 break;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
161 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
162 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
163 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
164 elseif query.after then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
165 for j = 1, count do
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
166 if (items[j].key or tostring(j)) == query.after then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
167 i = j;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
168 break;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
169 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
170 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
171 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
172 if query.limit and #items - i > query.limit then
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
173 items[i+query.limit+1] = nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
174 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
175 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
176 return function ()
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
177 i = i + 1;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
178 local item = items[i];
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
179 if not item then return; end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
180 local key = item.key or tostring(i);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
181 local when = item.when or datetime.parse(item.attr.stamp);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
182 local with = item.with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
183 item.key, item.when, item.with = nil, nil, nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
184 item.attr.stamp = nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
185 item.attr.stamp_legacy = nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
186 item = st.deserialize(item);
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
187 return key, item, when, with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
188 end, count;
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
189 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents: 8020
diff changeset
190
8023
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
191 function archive:dates(username)
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
192 local items, err = datamanager.list_load(username, host, self.store);
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
193 if not items then return items, err; end
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
194 return array(items):pluck("when"):map(datetime.date):unique();
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
195 end
5a9d491cc714 mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents: 8022
diff changeset
196
8022
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
197 function archive:delete(username, query)
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
198 if not query or next(query) == nil then
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
199 archive_item_count_cache:set(username, nil);
8022
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
200 return datamanager.list_store(username, host, self.store, nil);
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
201 end
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
202 local items, err = datamanager.list_load(username, host, self.store);
8143
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
203 if not items then
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
204 if err then
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
205 return items, err;
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
206 end
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
207 archive_item_count_cache:set(username, 0);
8143
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
208 -- Store is empty
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
209 return 0;
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents: 8142
diff changeset
210 end
8022
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
211 items = array(items);
8142
13e075549cb3 mod_storage_internal: Correctly calculate number of deleted items (fixes #912)
Kim Alvefur <zash@zash.se>
parents: 8089
diff changeset
212 local count_before = #items;
8310
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
213 if query then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
214 if query.key then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
215 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
216 return item.key ~= query.key;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
217 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
218 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
219 if query.with then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
220 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
221 return item.with ~= query.with;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
222 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
223 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
224 if query.start then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
225 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
226 return item.when < query.start;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
227 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
228 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
229 if query["end"] then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
230 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
231 return item.when > query["end"];
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
232 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
233 end
9105
e735c9865f42 mod_storage_internal: Only apply truncate if there are more items than requested
Kim Alvefur <zash@zash.se>
parents: 8402
diff changeset
234 if query.truncate and #items > query.truncate then
8393
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
235 if query.reverse then
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
236 -- Before: { 1, 2, 3, 4, 5, }
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
237 -- After: { 1, 2, 3 }
8394
4892c22403d5 mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents: 8393
diff changeset
238 for i = #items, query.truncate + 1, -1 do
4892c22403d5 mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents: 8393
diff changeset
239 items[i] = nil;
8393
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
240 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
241 else
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
242 -- Before: { 1, 2, 3, 4, 5, }
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
243 -- After: { 3, 4, 5 }
8394
4892c22403d5 mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents: 8393
diff changeset
244 local offset = #items - query.truncate;
4892c22403d5 mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents: 8393
diff changeset
245 for i = 1, #items do
4892c22403d5 mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents: 8393
diff changeset
246 items[i] = items[i+offset];
8393
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
247 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
248 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents: 8310
diff changeset
249 end
8310
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents: 8309
diff changeset
250 end
8142
13e075549cb3 mod_storage_internal: Correctly calculate number of deleted items (fixes #912)
Kim Alvefur <zash@zash.se>
parents: 8089
diff changeset
251 local count = count_before - #items;
8402
469afa02947b mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents: 8394
diff changeset
252 if count == 0 then
469afa02947b mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents: 8394
diff changeset
253 return 0; -- No changes, skip write
469afa02947b mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents: 8394
diff changeset
254 end
8022
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
255 local ok, err = datamanager.list_store(username, host, self.store, items);
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
256 if not ok then return ok, err; end
9883
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents: 9105
diff changeset
257 archive_item_count_cache:set(username, #items);
8022
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
258 return count;
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
259 end
05e201468f27 mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents: 8021
diff changeset
260
5121
b5a5643f8572 core.storagemanager, mod_storage_*: "data-driver" -> "storage-provider", to allow using module:provides().
Waqas Hussain <waqas20@gmail.com>
parents: 5039
diff changeset
261 module:provides("storage", driver);