Software /
code /
prosody
Annotate
plugins/mod_storage_internal.lua @ 11517:f7275c2c58fa
mod_c2s: Fix traceback if session was destroyed while opening stream (thanks Ge0rG)
Could happen with the 'opportunistic_writes' setting, since then the
stream opening is written directly to the socket, which can in turn
trigger session destruction if the socket somehow got closed just after
the other sent their stream header.
Error happens later when it tries to `hosts[session.host == nil].events`
author | Kim Alvefur <zash@zash.se> |
---|---|
date | Wed, 14 Apr 2021 16:02:47 +0200 |
parent | 11278:c3907f05bed4 |
child | 11765:1cac469b18d0 |
rev | line source |
---|---|
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
1 local cache = require "util.cache"; |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
2 local datamanager = require "core.storagemanager".olddm; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
3 local array = require "util.array"; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
4 local datetime = require "util.datetime"; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
5 local st = require "util.stanza"; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
6 local now = require "util.time".now; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
7 local id = require "util.id".medium; |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
8 local jid_join = require "util.jid".join; |
11278
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
9 local set = require "util.set"; |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
10 |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
11 local host = module.host; |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
12 |
9888
5669bb11ac7b
mod_storage_internal: Increase default quota to 10 000
Kim Alvefur <zash@zash.se>
parents:
9885
diff
changeset
|
13 local archive_item_limit = module:get_option_number("storage_archive_item_limit", 10000); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
14 local archive_item_count_cache = cache.new(module:get_option("storage_archive_item_limit_cache_size", 1000)); |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
15 |
5122
b41c33dc7c36
mod_storage_*: Don't explicitly set driver name, to ease copying/renaming modules.
Waqas Hussain <waqas20@gmail.com>
parents:
5121
diff
changeset
|
16 local driver = {}; |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
17 |
5153
688aeac0012a
mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents:
5133
diff
changeset
|
18 function driver:open(store, typ) |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
19 local mt = self[typ or "keyval"] |
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
20 if not mt then |
6283
7cf6d3a2c855
mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types
Kim Alvefur <zash@zash.se>
parents:
5153
diff
changeset
|
21 return nil, "unsupported-store"; |
7cf6d3a2c855
mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types
Kim Alvefur <zash@zash.se>
parents:
5153
diff
changeset
|
22 end |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
23 return setmetatable({ store = store, type = typ }, mt); |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
24 end |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
25 |
8020
342ce07836de
mod_storage_internal: Ignore unused 'self' argument [luacheck]
Kim Alvefur <zash@zash.se>
parents:
8019
diff
changeset
|
26 function driver:stores(username) -- luacheck: ignore 212/self |
8019
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
27 return datamanager.stores(username, host); |
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
28 end |
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
29 |
8020
342ce07836de
mod_storage_internal: Ignore unused 'self' argument [luacheck]
Kim Alvefur <zash@zash.se>
parents:
8019
diff
changeset
|
30 function driver:purge(user) -- luacheck: ignore 212/self |
8019
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
31 return datamanager.purge(user, host); |
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
32 end |
925098aad268
mod_storage_internal: Reorder methods
Kim Alvefur <zash@zash.se>
parents:
8018
diff
changeset
|
33 |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
34 local keyval = { }; |
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
35 driver.keyval = { __index = keyval }; |
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
36 |
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
37 function keyval:get(user) |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
38 return datamanager.load(user, host, self.store); |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
39 end |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
40 |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
41 function keyval:set(user, data) |
4085
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
42 return datamanager.store(user, host, self.store, data); |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
43 end |
7699cef04740
storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
44 |
8018
9545d0a9401f
mod_storage_internal: Separate driver from keyval implementation
Kim Alvefur <zash@zash.se>
parents:
6283
diff
changeset
|
45 function keyval:users() |
5153
688aeac0012a
mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents:
5133
diff
changeset
|
46 return datamanager.users(host, self.store, self.type); |
688aeac0012a
mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents:
5133
diff
changeset
|
47 end |
688aeac0012a
mod_storage_internal, datamanager: Add support for iterating over users with data in a store
Kim Alvefur <zash@zash.se>
parents:
5133
diff
changeset
|
48 |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
49 local archive = {}; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
50 driver.archive = { __index = archive }; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
51 |
9884
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
52 archive.caps = { |
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
53 total = true; |
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
54 quota = archive_item_limit; |
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
55 truncate = true; |
11275
b8fada57faf0
mod_storage_internal: Add support for full ID range query
Kim Alvefur <zash@zash.se>
parents:
10926
diff
changeset
|
56 full_id_range = true; |
11278
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
57 ids = true; |
9884
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
58 }; |
9751c17f5281
mod_storage_internal,_sql: Expose archive capabilities feature set
Kim Alvefur <zash@zash.se>
parents:
9883
diff
changeset
|
59 |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
60 function archive:append(username, key, value, when, with) |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
61 when = when or now(); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
62 if not st.is_stanza(value) then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
63 return nil, "unsupported-datatype"; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
64 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
65 value = st.preserialize(st.clone(value)); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
66 value.when = when; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
67 value.with = with; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
68 value.attr.stamp = datetime.datetime(when); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
69 value.attr.stamp_legacy = datetime.legacy(when); |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
70 |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
71 local cache_key = jid_join(username, host, self.store); |
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
72 local item_count = archive_item_count_cache:get(cache_key); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
73 |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
74 if key then |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
75 local items, err = datamanager.list_load(username, host, self.store); |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
76 if not items and err then return items, err; end |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
77 |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
78 -- Check the quota |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
79 item_count = items and #items or 0; |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
80 archive_item_count_cache:set(cache_key, item_count); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
81 if item_count >= archive_item_limit then |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
82 module:log("debug", "%s reached or over quota, not adding to store", username); |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
83 return nil, "quota-limit"; |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
84 end |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
85 |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
86 if items then |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
87 -- Filter out any item with the same key as the one being added |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
88 items = array(items); |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
89 items:filter(function (item) |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
90 return item.key ~= key; |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
91 end); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
92 |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
93 value.key = key; |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
94 items:push(value); |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
95 local ok, err = datamanager.list_store(username, host, self.store, items); |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
96 if not ok then return ok, err; end |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
97 archive_item_count_cache:set(cache_key, #items); |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
98 return key; |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
99 end |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
100 else |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
101 if not item_count then -- Item count not cached? |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
102 -- We need to load the list to get the number of items currently stored |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
103 local items, err = datamanager.list_load(username, host, self.store); |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
104 if not items and err then return items, err; end |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
105 item_count = items and #items or 0; |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
106 archive_item_count_cache:set(cache_key, item_count); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
107 end |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
108 if item_count >= archive_item_limit then |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
109 module:log("debug", "%s reached or over quota, not adding to store", username); |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
110 return nil, "quota-limit"; |
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
111 end |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
112 key = id(); |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
113 end |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
114 |
9895
6bd65bff03b5
mod_storage_internal: Include store name when reporting quota status
Kim Alvefur <zash@zash.se>
parents:
9888
diff
changeset
|
115 module:log("debug", "%s has %d items out of %d limit in store %s", username, item_count, archive_item_limit, self.store); |
9883
f76bd399267c
mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733)
Matthew Wild <mwild1@gmail.com>
parents:
9105
diff
changeset
|
116 |
8309
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
117 value.key = key; |
5281c479955a
mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql)
Kim Alvefur <zash@zash.se>
parents:
8173
diff
changeset
|
118 |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
119 local ok, err = datamanager.list_append(username, host, self.store, value); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
120 if not ok then return ok, err; end |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
121 archive_item_count_cache:set(cache_key, item_count+1); |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
122 return key; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
123 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
124 |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
125 function archive:find(username, query) |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
126 local items, err = datamanager.list_load(username, host, self.store); |
8173
3ff99d49082f
mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents:
8143
diff
changeset
|
127 if not items then |
3ff99d49082f
mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents:
8143
diff
changeset
|
128 if err then |
3ff99d49082f
mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents:
8143
diff
changeset
|
129 return items, err; |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
130 elseif query then |
10024
4a0d990253a0
mod_storage_internal: Return appropriate error even with empty archive
Kim Alvefur <zash@zash.se>
parents:
10018
diff
changeset
|
131 if query.before or query.after then |
4a0d990253a0
mod_storage_internal: Return appropriate error even with empty archive
Kim Alvefur <zash@zash.se>
parents:
10018
diff
changeset
|
132 return nil, "item-not-found"; |
4a0d990253a0
mod_storage_internal: Return appropriate error even with empty archive
Kim Alvefur <zash@zash.se>
parents:
10018
diff
changeset
|
133 end |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
134 if query.total then |
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
135 return function () end, 0; |
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
136 end |
8173
3ff99d49082f
mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents:
8143
diff
changeset
|
137 end |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
138 return function () end; |
8173
3ff99d49082f
mod_storage_internal: Return a noop iterator if archive is empty (fixes #920)
Kim Alvefur <zash@zash.se>
parents:
8143
diff
changeset
|
139 end |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
140 local count = nil; |
10926
c55bd98a54f8
mod_storage_internal, mod_storage_memory: Add support for query.before
Matthew Wild <mwild1@gmail.com>
parents:
10844
diff
changeset
|
141 local i, last_key = 0; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
142 if query then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
143 items = array(items); |
8089
4ba8cb75d925
mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents:
8023
diff
changeset
|
144 if query.key then |
4ba8cb75d925
mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents:
8023
diff
changeset
|
145 items:filter(function (item) |
4ba8cb75d925
mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents:
8023
diff
changeset
|
146 return item.key == query.key; |
4ba8cb75d925
mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents:
8023
diff
changeset
|
147 end); |
4ba8cb75d925
mod_storage_internal: Support the 'key' archive query field
Kim Alvefur <zash@zash.se>
parents:
8023
diff
changeset
|
148 end |
11278
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
149 if query.ids then |
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
150 local ids = set.new(query.ids); |
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
151 items:filter(function (item) |
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
152 return ids:contains(item.key); |
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
153 end); |
c3907f05bed4
mod_storage_internal: Support query for set of IDs
Kim Alvefur <zash@zash.se>
parents:
11275
diff
changeset
|
154 end |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
155 if query.with then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
156 items:filter(function (item) |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
157 return item.with == query.with; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
158 end); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
159 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
160 if query.start then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
161 items:filter(function (item) |
10843
8fcd46ee9bf5
mod_storage_internal: Fix error in time limited queries on items without 'when' field, fixes #1557
Kim Alvefur <zash@zash.se>
parents:
9105
diff
changeset
|
162 local when = item.when or datetime.parse(item.attr.stamp); |
8fcd46ee9bf5
mod_storage_internal: Fix error in time limited queries on items without 'when' field, fixes #1557
Kim Alvefur <zash@zash.se>
parents:
9105
diff
changeset
|
163 return when >= query.start; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
164 end); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
165 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
166 if query["end"] then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
167 items:filter(function (item) |
10843
8fcd46ee9bf5
mod_storage_internal: Fix error in time limited queries on items without 'when' field, fixes #1557
Kim Alvefur <zash@zash.se>
parents:
9105
diff
changeset
|
168 local when = item.when or datetime.parse(item.attr.stamp); |
8fcd46ee9bf5
mod_storage_internal: Fix error in time limited queries on items without 'when' field, fixes #1557
Kim Alvefur <zash@zash.se>
parents:
9105
diff
changeset
|
169 return when <= query["end"]; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
170 end); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
171 end |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
172 if query.total then |
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
173 count = #items; |
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
174 end |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
175 if query.reverse then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
176 items:reverse(); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
177 if query.before then |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
178 local found = false; |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
179 for j = 1, #items do |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
180 if (items[j].key or tostring(j)) == query.before then |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
181 found = true; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
182 i = j; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
183 break; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
184 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
185 end |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
186 if not found then |
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
187 return nil, "item-not-found"; |
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
188 end |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
189 end |
11275
b8fada57faf0
mod_storage_internal: Add support for full ID range query
Kim Alvefur <zash@zash.se>
parents:
10926
diff
changeset
|
190 last_key = query.after; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
191 elseif query.after then |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
192 local found = false; |
9997
7c4631d7b6fb
mod_storage_internal,memory: Only return total count if requested
Kim Alvefur <zash@zash.se>
parents:
9903
diff
changeset
|
193 for j = 1, #items do |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
194 if (items[j].key or tostring(j)) == query.after then |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
195 found = true; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
196 i = j; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
197 break; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
198 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
199 end |
10018
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
200 if not found then |
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
201 return nil, "item-not-found"; |
7408b9473729
mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325)
Kim Alvefur <zash@zash.se>
parents:
10001
diff
changeset
|
202 end |
11275
b8fada57faf0
mod_storage_internal: Add support for full ID range query
Kim Alvefur <zash@zash.se>
parents:
10926
diff
changeset
|
203 last_key = query.before; |
b8fada57faf0
mod_storage_internal: Add support for full ID range query
Kim Alvefur <zash@zash.se>
parents:
10926
diff
changeset
|
204 elseif query.before then |
b8fada57faf0
mod_storage_internal: Add support for full ID range query
Kim Alvefur <zash@zash.se>
parents:
10926
diff
changeset
|
205 last_key = query.before; |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
206 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
207 if query.limit and #items - i > query.limit then |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
208 items[i+query.limit+1] = nil; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
209 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
210 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
211 return function () |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
212 i = i + 1; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
213 local item = items[i]; |
10926
c55bd98a54f8
mod_storage_internal, mod_storage_memory: Add support for query.before
Matthew Wild <mwild1@gmail.com>
parents:
10844
diff
changeset
|
214 if not item or (last_key and item.key == last_key) then |
c55bd98a54f8
mod_storage_internal, mod_storage_memory: Add support for query.before
Matthew Wild <mwild1@gmail.com>
parents:
10844
diff
changeset
|
215 return; |
c55bd98a54f8
mod_storage_internal, mod_storage_memory: Add support for query.before
Matthew Wild <mwild1@gmail.com>
parents:
10844
diff
changeset
|
216 end |
8021
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
217 local key = item.key or tostring(i); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
218 local when = item.when or datetime.parse(item.attr.stamp); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
219 local with = item.with; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
220 item.key, item.when, item.with = nil, nil, nil; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
221 item.attr.stamp = nil; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
222 item.attr.stamp_legacy = nil; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
223 item = st.deserialize(item); |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
224 return key, item, when, with; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
225 end, count; |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
226 end |
83f18982bcfd
mod_storage_internal: Add basic archive store implementation
Kim Alvefur <zash@zash.se>
parents:
8020
diff
changeset
|
227 |
10837
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
228 function archive:get(username, wanted_key) |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
229 local iter, err = self:find(username, { key = wanted_key }) |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
230 if not iter then return iter, err; end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
231 for key, stanza, when, with in iter do |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
232 if key == wanted_key then |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
233 return stanza, when, with; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
234 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
235 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
236 return nil, "item-not-found"; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
237 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
238 |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
239 function archive:set(username, key, new_value, new_when, new_with) |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
240 local items, err = datamanager.list_load(username, host, self.store); |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
241 if not items then |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
242 if err then |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
243 return items, err; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
244 else |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
245 return nil, "item-not-found"; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
246 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
247 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
248 |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
249 for i = 1, #items do |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
250 local old_item = items[i]; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
251 if old_item.key == key then |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
252 local item = st.preserialize(st.clone(new_value)); |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
253 |
10842
5a6ba2f38e2b
mod_storage_internal: Fix keeping old timestamp in archive map API
Kim Alvefur <zash@zash.se>
parents:
10837
diff
changeset
|
254 local when = new_when or old_item.when or datetime.parse(old_item.attr.stamp); |
10837
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
255 item.key = key; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
256 item.when = when; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
257 item.with = new_with or old_item.with; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
258 item.attr.stamp = datetime.datetime(when); |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
259 item.attr.stamp_legacy = datetime.legacy(when); |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
260 items[i] = item; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
261 return datamanager.list_store(username, host, self.store, items); |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
262 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
263 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
264 |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
265 return nil, "item-not-found"; |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
266 end |
f23363380599
mod_storage_internal: Implement key-value API
Kim Alvefur <zash@zash.se>
parents:
10223
diff
changeset
|
267 |
8023
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
268 function archive:dates(username) |
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
269 local items, err = datamanager.list_load(username, host, self.store); |
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
270 if not items then return items, err; end |
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
271 return array(items):pluck("when"):map(datetime.date):unique(); |
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
272 end |
5a9d491cc714
mod_storage_internal: Add the dates method
Kim Alvefur <zash@zash.se>
parents:
8022
diff
changeset
|
273 |
9903
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
274 function archive:summary(username, query) |
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
275 local iter, err = self:find(username, query) |
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
276 if not iter then return iter, err; end |
10220
1e2b444acb72
mod_storage_*: Tweak :summary API to allow future expansion with more fields
Kim Alvefur <zash@zash.se>
parents:
10024
diff
changeset
|
277 local counts = {}; |
10222
51f145094648
mod_storage_*: Also include timestmap of first message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10221
diff
changeset
|
278 local earliest = {}; |
10221
068692cb9e78
mod_storage_*: Include timestamp of latest message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10220
diff
changeset
|
279 local latest = {}; |
10223
d185c4961ee0
mod_storage_internal: Include last text message
Kim Alvefur <zash@zash.se>
parents:
10222
diff
changeset
|
280 local body = {}; |
d185c4961ee0
mod_storage_internal: Include last text message
Kim Alvefur <zash@zash.se>
parents:
10222
diff
changeset
|
281 for _, stanza, when, with in iter do |
10220
1e2b444acb72
mod_storage_*: Tweak :summary API to allow future expansion with more fields
Kim Alvefur <zash@zash.se>
parents:
10024
diff
changeset
|
282 counts[with] = (counts[with] or 0) + 1; |
10222
51f145094648
mod_storage_*: Also include timestmap of first message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10221
diff
changeset
|
283 if earliest[with] == nil then |
51f145094648
mod_storage_*: Also include timestmap of first message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10221
diff
changeset
|
284 earliest[with] = when; |
51f145094648
mod_storage_*: Also include timestmap of first message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10221
diff
changeset
|
285 end |
10221
068692cb9e78
mod_storage_*: Include timestamp of latest message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10220
diff
changeset
|
286 latest[with] = when; |
10223
d185c4961ee0
mod_storage_internal: Include last text message
Kim Alvefur <zash@zash.se>
parents:
10222
diff
changeset
|
287 body[with] = stanza:get_child_text("body") or body[with]; |
9903
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
288 end |
10220
1e2b444acb72
mod_storage_*: Tweak :summary API to allow future expansion with more fields
Kim Alvefur <zash@zash.se>
parents:
10024
diff
changeset
|
289 return { |
1e2b444acb72
mod_storage_*: Tweak :summary API to allow future expansion with more fields
Kim Alvefur <zash@zash.se>
parents:
10024
diff
changeset
|
290 counts = counts; |
10222
51f145094648
mod_storage_*: Also include timestmap of first message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10221
diff
changeset
|
291 earliest = earliest; |
10221
068692cb9e78
mod_storage_*: Include timestamp of latest message in :summary API
Kim Alvefur <zash@zash.se>
parents:
10220
diff
changeset
|
292 latest = latest; |
10223
d185c4961ee0
mod_storage_internal: Include last text message
Kim Alvefur <zash@zash.se>
parents:
10222
diff
changeset
|
293 body = body; |
10220
1e2b444acb72
mod_storage_*: Tweak :summary API to allow future expansion with more fields
Kim Alvefur <zash@zash.se>
parents:
10024
diff
changeset
|
294 }; |
9903
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
295 end |
2c5546cc5c70
mod_storage_internal: Implement a summary API returning message counts per contact
Kim Alvefur <zash@zash.se>
parents:
9895
diff
changeset
|
296 |
10001
55f5588d71c6
mod_storage_internal: Add support for iterating over users in archive stores
Kim Alvefur <zash@zash.se>
parents:
9997
diff
changeset
|
297 function archive:users() |
55f5588d71c6
mod_storage_internal: Add support for iterating over users in archive stores
Kim Alvefur <zash@zash.se>
parents:
9997
diff
changeset
|
298 return datamanager.users(host, self.store, "list"); |
55f5588d71c6
mod_storage_internal: Add support for iterating over users in archive stores
Kim Alvefur <zash@zash.se>
parents:
9997
diff
changeset
|
299 end |
55f5588d71c6
mod_storage_internal: Add support for iterating over users in archive stores
Kim Alvefur <zash@zash.se>
parents:
9997
diff
changeset
|
300 |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
301 function archive:delete(username, query) |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
302 local cache_key = jid_join(username, host, self.store); |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
303 if not query or next(query) == nil then |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
304 archive_item_count_cache:set(cache_key, nil); |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
305 return datamanager.list_store(username, host, self.store, nil); |
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
306 end |
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
307 local items, err = datamanager.list_load(username, host, self.store); |
8143
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
308 if not items then |
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
309 if err then |
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
310 return items, err; |
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
311 end |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
312 archive_item_count_cache:set(cache_key, 0); |
8143
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
313 -- Store is empty |
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
314 return 0; |
c4c159953c72
mod_storage_internal: Handle case of empty item store when deleting (fixes #910)
Kim Alvefur <zash@zash.se>
parents:
8142
diff
changeset
|
315 end |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
316 items = array(items); |
8142
13e075549cb3
mod_storage_internal: Correctly calculate number of deleted items (fixes #912)
Kim Alvefur <zash@zash.se>
parents:
8089
diff
changeset
|
317 local count_before = #items; |
8310
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
318 if query then |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
319 if query.key then |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
320 items:filter(function (item) |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
321 return item.key ~= query.key; |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
322 end); |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
323 end |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
324 if query.with then |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
325 items:filter(function (item) |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
326 return item.with ~= query.with; |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
327 end); |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
328 end |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
329 if query.start then |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
330 items:filter(function (item) |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
331 return item.when < query.start; |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
332 end); |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
333 end |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
334 if query["end"] then |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
335 items:filter(function (item) |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
336 return item.when > query["end"]; |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
337 end); |
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
338 end |
9105
e735c9865f42
mod_storage_internal: Only apply truncate if there are more items than requested
Kim Alvefur <zash@zash.se>
parents:
8402
diff
changeset
|
339 if query.truncate and #items > query.truncate then |
8393
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
340 if query.reverse then |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
341 -- Before: { 1, 2, 3, 4, 5, } |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
342 -- After: { 1, 2, 3 } |
8394
4892c22403d5
mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents:
8393
diff
changeset
|
343 for i = #items, query.truncate + 1, -1 do |
4892c22403d5
mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents:
8393
diff
changeset
|
344 items[i] = nil; |
8393
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
345 end |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
346 else |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
347 -- Before: { 1, 2, 3, 4, 5, } |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
348 -- After: { 3, 4, 5 } |
8394
4892c22403d5
mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents:
8393
diff
changeset
|
349 local offset = #items - query.truncate; |
4892c22403d5
mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents:
8393
diff
changeset
|
350 for i = 1, #items do |
4892c22403d5
mod_storage_internal: Optimize truncation
Kim Alvefur <zash@zash.se>
parents:
8393
diff
changeset
|
351 items[i] = items[i+offset]; |
8393
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
352 end |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
353 end |
b6a7b83f8d87
mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store
Kim Alvefur <zash@zash.se>
parents:
8310
diff
changeset
|
354 end |
8310
1759491b53db
mod_storage_internal: Add more extensive query support to archive:delete method
Kim Alvefur <zash@zash.se>
parents:
8309
diff
changeset
|
355 end |
8142
13e075549cb3
mod_storage_internal: Correctly calculate number of deleted items (fixes #912)
Kim Alvefur <zash@zash.se>
parents:
8089
diff
changeset
|
356 local count = count_before - #items; |
8402
469afa02947b
mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents:
8394
diff
changeset
|
357 if count == 0 then |
469afa02947b
mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents:
8394
diff
changeset
|
358 return 0; -- No changes, skip write |
469afa02947b
mod_storage_internal: Skip write if no items matched a deletion query
Kim Alvefur <zash@zash.se>
parents:
8394
diff
changeset
|
359 end |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
360 local ok, err = datamanager.list_store(username, host, self.store, items); |
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
361 if not ok then return ok, err; end |
9885
64e16d1e91f6
mod_storage_internal,_sql: Key item count cache on both username and store
Kim Alvefur <zash@zash.se>
parents:
9884
diff
changeset
|
362 archive_item_count_cache:set(cache_key, #items); |
8022
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
363 return count; |
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
364 end |
05e201468f27
mod_storage_internal: Add support for removing archived items
Kim Alvefur <zash@zash.se>
parents:
8021
diff
changeset
|
365 |
5121
b5a5643f8572
core.storagemanager, mod_storage_*: "data-driver" -> "storage-provider", to allow using module:provides().
Waqas Hussain <waqas20@gmail.com>
parents:
5039
diff
changeset
|
366 module:provides("storage", driver); |