Software `/` code `/` prosody

Annotate

util/datamanager.lua @ 13228:616c578c644f

mod_storage_internal: Use integer option method for cache size Missed this one in previous sweep

author	Kim Alvefur <zash@zash.se>
date	Tue, 18 Jul 2023 12:36:47 +0200
parent	13186:affaf6d08d26
child	13231:6a11d92ae436

rev	line source
1523 841d61be198f Remove version number from copyright headers Matthew Wild <mwild1@gmail.com> parents: 1462 diff changeset	1 -- Prosody IM
2923 b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	2 -- Copyright (C) 2008-2010 Matthew Wild
b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	3 -- Copyright (C) 2008-2010 Waqas Hussain
4452 7de17ca4de14 util/datamanager: Use package.config to figure out directory seperator James Callahan <james@chatid.com> parents: 4112 diff changeset	4 --
758 b1885732e979 GPL->MIT! Matthew Wild <mwild1@gmail.com> parents: 732 diff changeset	5 -- This project is MIT/X11 licensed. Please see the
b1885732e979 GPL->MIT! Matthew Wild <mwild1@gmail.com> parents: 732 diff changeset	6 -- COPYING file in the source package for more information.
519 cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	7 --
cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	8
cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	9
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	10 local string = string;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	11 local format = string.format;
5051 71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	12 local setmetatable = setmetatable;
71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	13 local ipairs = ipairs;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	14 local char = string.char;
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	15 local pcall = pcall;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	16 local log = require "prosody.util.logger".init("datamanager");
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	17 local io_open = io.open;
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	18 local os_remove = os.remove;
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	19 local os_rename = os.rename;
5051 71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	20 local tonumber = tonumber;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	21 local floor = math.floor;
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	22 local next = next;
7432 92f721226753 util.datamanager: Import tostring and type (fix global access) Kim Alvefur <zash@zash.se> parents: 7202 diff changeset	23 local type = type;
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	24 local t_insert = table.insert;
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	25 local t_concat = table.concat;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	26 local envloadfile = require"prosody.util.envload".envloadfile;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	27 local envload = require"prosody.util.envload".envload;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	28 local serialize = require "prosody.util.serialization".serialize;
3086 931acb1188b1 util.datamanager: When failing to load a file, and the file exists, return nil, error. Waqas Hussain <waqas20@gmail.com> parents: 2925 diff changeset	29 local lfs = require "lfs";
9993 02a41315d275 Fix various spelling mistakes [codespell] Kim Alvefur <zash@zash.se> parents: 8555 diff changeset	30 -- Extract directory separator from package.config (an undocumented string that comes with lua)
7995 b99404a340fc util.datamanager: Split long line [luacheck] Kim Alvefur <zash@zash.se> parents: 7929 diff changeset	31 local path_separator = assert ( package.config:match ( "^([^\n]+)" ) , "package.config not in standard form" )
b99404a340fc util.datamanager: Split long line [luacheck] Kim Alvefur <zash@zash.se> parents: 7929 diff changeset	32
4093 36555949bd16 util.datamanager: Use prosody.paths.data as the initial value for data_path Matthew Wild <mwild1@gmail.com> parents: 3723 diff changeset	33 local prosody = prosody;
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	34
13186 affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	35 local blocksize = 0x1000;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	36 local raw_mkdir = lfs.mkdir;
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	37 local atomic_append;
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	38 local remove_blocks;
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	39 local ENOENT = 2;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	40 pcall(function()
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	41 local pposix = require "prosody.util.pposix";
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	42 raw_mkdir = pposix.mkdir or raw_mkdir; -- Doesn't trample on umask
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	43 atomic_append = pposix.atomic_append;
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	44 remove_blocks = pposix.remove_blocks;
7927 7132abcf669e util.datamanager: Import value of ENOENT from util.pposix (if available) Kim Alvefur <zash@zash.se> parents: 7925 diff changeset	45 ENOENT = pposix.ENOENT or ENOENT;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	46 end);
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	47
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	48 local _ENV = nil;
8555 4f0f5b49bb03 vairious: Add annotation when an empty environment is set [luacheck] Kim Alvefur <zash@zash.se> parents: 8299 diff changeset	49 -- luacheck: std none
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	50
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	51 ---- utils -----
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	52 local encode, decode, store_encode;
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	53 do
7732 aba3dd84d9f0 util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimal Kim Alvefur <zash@zash.se> parents: 7674 diff changeset	54 local urlcodes = setmetatable({}, { __index = function (t, k) t[k] = char(tonumber(k, 16)); return t[k]; end });
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	55
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	56 decode = function (s)
7734 6a52415ed68a util.datamanager: Shorter pattern Kim Alvefur <zash@zash.se> parents: 7733 diff changeset	57 return s and (s:gsub("%%(%x%x)", urlcodes));
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	58 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	59
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	60 encode = function (s)
628 3712d36b6d25 Fixed URL encoding to generate %0x instead of %x Waqas Hussain <waqas20@gmail.com> parents: 615 diff changeset	61 return s and (s:gsub("%W", function (c) return format("%%%02x", c:byte()); end));
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	62 end
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	63
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	64 -- Special encode function for store names, which historically were unencoded.
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	65 -- All currently known stores use a-z and underscore, so this one preserves underscores.
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	66 store_encode = function (s)
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	67 return s and (s:gsub("[^_%w]", function (c) return format("%%%02x", c:byte()); end));
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	68 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	69 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	70
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	71 if not atomic_append then
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	72 function atomic_append(f, data)
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	73 local pos = f:seek();
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	74 if not f:write(data) or not f:flush() then
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	75 f:seek("set", pos);
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	76 f:write((" "):rep(#data));
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	77 f:flush();
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	78 return nil, "write-failed";
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	79 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	80 return true;
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	81 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	82 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	83
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	84 local _mkdir = {};
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	85 local function mkdir(path)
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	86 path = path:gsub("/", path_separator); -- TODO as an optimization, do this during path creation rather than here
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	87 if not _mkdir[path] then
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	88 raw_mkdir(path);
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	89 _mkdir[path] = true;
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	90 end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	91 return path;
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	92 end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	93
4108 e3e3aa286334 util.datamanager: Handle gracefully the lack of prosody.paths.data Matthew Wild <mwild1@gmail.com> parents: 4093 diff changeset	94 local data_path = (prosody and prosody.paths and prosody.paths.data) or ".";
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	95 local callbacks = {};
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	96
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	97 ------- API -------------
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	98
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	99 local function set_data_path(path)
1097 c5b33640a5f0 util.datamanager: Lower log level of 'Setting data path' to 'debug' Matthew Wild <mwild1@gmail.com> parents: 932 diff changeset	100 log("debug", "Setting data path to: %s", path);
452 613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	101 data_path = path;
613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	102 end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	103
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	104 local function callback(username, host, datastore, data)
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	105 for _, f in ipairs(callbacks) do
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	106 username, host, datastore, data = f(username, host, datastore, data);
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	107 if username == false then break; end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	108 end
4452 7de17ca4de14 util/datamanager: Use package.config to figure out directory seperator James Callahan <james@chatid.com> parents: 4112 diff changeset	109
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	110 return username, host, datastore, data;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	111 end
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	112 local function add_callback(func)
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	113 if not callbacks[func] then -- Would you really want to set the same callback more than once?
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	114 callbacks[func] = true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	115 callbacks[#callbacks+1] = func;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	116 return true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	117 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	118 end
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	119 local function remove_callback(func)
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	120 if callbacks[func] then
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	121 for i, f in ipairs(callbacks) do
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	122 if f == func then
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	123 callbacks[i] = nil;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	124 callbacks[f] = nil;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	125 return true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	126 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	127 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	128 end
932 1ed3e5fe165a Added: datamanager: Allow a callback to be installed which selectively prevents disk writes Waqas Hussain <waqas20@gmail.com> parents: 915 diff changeset	129 end
452 613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	130
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	131 local function getpath(username, host, datastore, ext, create)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	132 ext = ext or "dat";
2328 2804d81206d8 util.datamanager: Store data stores with no host in '_global' folder Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	133 host = (host and encode(host)) or "_global";
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	134 username = username and encode(username);
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	135 datastore = store_encode(datastore);
84 d0a0bac6815e Added: Datastore support for hosts and global data in addition to users Waqas Hussain <waqas20@gmail.com> parents: 0 diff changeset	136 if username then
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	137 if create then mkdir(mkdir(mkdir(data_path).."/"..host).."/"..datastore); end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	138 return format("%s/%s/%s/%s.%s", data_path, host, datastore, username, ext);
5243 07e8256efcda util.datamanager: Remove dead code path Kim Alvefur <zash@zash.se> parents: 5153 diff changeset	139 else
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	140 if create then mkdir(mkdir(data_path).."/"..host); end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	141 return format("%s/%s/%s.%s", data_path, host, datastore, ext);
84 d0a0bac6815e Added: Datastore support for hosts and global data in addition to users Waqas Hussain <waqas20@gmail.com> parents: 0 diff changeset	142 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	143 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	144
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	145 local function load(username, host, datastore)
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	146 local data, err, errno = envloadfile(getpath(username, host, datastore), {});
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	147 if not data then
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	148 if errno == ENOENT then
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	149 -- No such file, ok to ignore
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	150 return nil;
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	151 end
7996 f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	152 log("error", "Failed to load %s storage ('%s') for user: %s@%s", datastore, err, username or "nil", host or "nil");
f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	153 return nil, "Error reading storage";
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	154 end
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	155
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	156 local success, ret = pcall(data);
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	157 if not success then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	158 log("error", "Unable to load %s storage ('%s') for user: %s@%s", datastore, ret, username or "nil", host or "nil");
3086 931acb1188b1 util.datamanager: When failing to load a file, and the file exists, return nil, error. Waqas Hussain <waqas20@gmail.com> parents: 2925 diff changeset	159 return nil, "Error reading storage";
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	160 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	161 return ret;
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	162 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	163
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	164 local function atomic_store(filename, data)
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	165 local scratch = filename.."~";
10531 810abf34919a util.datamanager: Ignore unused 'errno' variable [luacheck] Kim Alvefur <zash@zash.se> parents: 9993 diff changeset	166 local f, ok, msg, errno; -- luacheck: ignore errno
810abf34919a util.datamanager: Ignore unused 'errno' variable [luacheck] Kim Alvefur <zash@zash.se> parents: 9993 diff changeset	167 -- TODO return util.error with code=errno?
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	168
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	169 f, msg, errno = io_open(scratch, "w");
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	170 if not f then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	171 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	172 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	173
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	174 ok, msg = f:write(data);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	175 if not ok then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	176 f:close();
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	177 os_remove(scratch);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	178 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	179 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	180
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	181 ok, msg = f:close();
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	182 if not ok then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	183 os_remove(scratch);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	184 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	185 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	186
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	187 return os_rename(scratch, filename);
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	188 end
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	189
6575 bdaadf70a48f util.datamanager: Check that the global 'prosody' exists before using it (fixes nil indexing in use outside of prosody) Kim Alvefur <zash@zash.se> parents: 5441 diff changeset	190 if prosody and prosody.platform ~= "posix" then
5065 acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	191 -- os.rename does not overwrite existing files on Windows
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	192 -- TODO We could use Transactional NTFS on Vista and above
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	193 function atomic_store(filename, data)
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	194 local f, err = io_open(filename, "w");
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	195 if not f then return f, err; end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	196 local ok, msg = f:write(data);
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	197 if not ok then f:close(); return ok, msg; end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	198 return f:close();
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	199 end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	200 end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	201
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	202 local function store(username, host, datastore, data)
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	203 if not data then
e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	204 data = {};
e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	205 end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	206
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	207 username, host, datastore, data = callback(username, host, datastore, data);
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	208 if username == false then
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	209 return true; -- Don't save this data at all
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	210 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	211
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	212 -- save the datastore
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	213 local d = "return " .. serialize(data) .. ";\n";
5441 6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	214 local mkdir_cache_cleared;
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	215 repeat
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	216 local ok, msg = atomic_store(getpath(username, host, datastore, nil, true), d);
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	217 if not ok then
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	218 if not mkdir_cache_cleared then -- We may need to recreate a removed directory
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	219 _mkdir = {};
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	220 mkdir_cache_cleared = true;
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	221 else
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	222 log("error", "Unable to write to %s storage ('%s') for user: %s@%s", datastore, msg, username or "nil", host or "nil");
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	223 return nil, "Error saving to storage";
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	224 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	225 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	226 if next(data) == nil then -- try to delete empty datastore
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	227 log("debug", "Removing empty %s datastore for user %s@%s", datastore, username or "nil", host or "nil");
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	228 os_remove(getpath(username, host, datastore));
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	229 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	230 -- we write data even when we are deleting because lua doesn't have a
12388 50fcd3879482 spelling: non-existing mistakes (thanks timeless) Kim Alvefur <zash@zash.se> parents: 12387 diff changeset	231 -- platform independent way of checking for nonexisting files
5441 6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	232 until ok;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	233 return true;
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	234 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	235
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	236 -- Append a blob of data to a file
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	237 local function append(username, host, datastore, ext, data)
7000 2b57f77985a3 util.datamanager: Make sure only strings are passed as data to append() Kim Alvefur <zash@zash.se> parents: 6999 diff changeset	238 if type(data) ~= "string" then return; end
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	239 local filename = getpath(username, host, datastore, ext, true);
6996 644b1bddc676 util.datamanager: No shadowing of variable [luacheck] Kim Alvefur <zash@zash.se> parents: 6995 diff changeset	240
8013 72cfbe377326 util.datamanager: Rearrange locals Kim Alvefur <zash@zash.se> parents: 8011 diff changeset	241 local f = io_open(filename, "r+");
5066 4be7093edde9 util.datamanager: Try to open in read+write mode, then retry with write mode if that fails (usually because it doesn't exist) Kim Alvefur <zash@zash.se> parents: 5057 diff changeset	242 if not f then
7928 9d014aca766f util.datamanager: Use atomic store function when writing the first entry of a list Kim Alvefur <zash@zash.se> parents: 7927 diff changeset	243 return atomic_store(filename, data);
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	244 -- File did probably not exist, let's create it
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	245 end
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	246
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	247 local pos = f:seek("end");
13186 affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	248 if (blocksize-(pos%blocksize)) < (#data%blocksize) then
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	249 -- pad to blocksize with newlines so that the next item is both on a new
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	250 -- block and a new line
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	251 atomic_append(f, ("\n"):rep(blocksize-(pos%blocksize)));
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	252 pos = f:seek("end");
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	253 end
6997 0ab228bc21c6 util.datamanager: Handle potential issues from fallocate Kim Alvefur <zash@zash.se> parents: 6996 diff changeset	254
8013 72cfbe377326 util.datamanager: Rearrange locals Kim Alvefur <zash@zash.se> parents: 8011 diff changeset	255 local ok, msg = atomic_append(f, data);
6997 0ab228bc21c6 util.datamanager: Handle potential issues from fallocate Kim Alvefur <zash@zash.se> parents: 6996 diff changeset	256
6998 86607fe755b6 util.datamanager: Handle potential error from :write() call Kim Alvefur <zash@zash.se> parents: 6997 diff changeset	257 if not ok then
86607fe755b6 util.datamanager: Handle potential error from :write() call Kim Alvefur <zash@zash.se> parents: 6997 diff changeset	258 f:close();
7001 2743759ca1b5 util.datamanager: Return extra location info Kim Alvefur <zash@zash.se> parents: 7000 diff changeset	259 return ok, msg, "write";
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	260 end
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	261
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	262 ok, msg = f:close();
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	263 if not ok then
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	264 return ok, msg, "close";
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	265 end
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	266
7001 2743759ca1b5 util.datamanager: Return extra location info Kim Alvefur <zash@zash.se> parents: 7000 diff changeset	267 return true, pos;
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	268 end
0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	269
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	270 local index_fmt, index_item_size, index_magic;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	271 if string.packsize then
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	272 index_fmt = "T"; -- offset to the end of the item, length can be derived from two index items
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	273 index_item_size = string.packsize(index_fmt);
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	274 index_magic = string.pack(index_fmt, 7767639 + 1); -- Magic string: T9 for "prosody", version number
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	275 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	276
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	277 local function list_append(username, host, datastore, data)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	278 if not data then return; end
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	279 if callback(username, host, datastore) == false then return true; end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	280 -- save the datastore
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	281
6993 dc0c6b8dc638 util.datamanager: Overwrite 'data' variable instead of shadownig it [luacheck] Kim Alvefur <zash@zash.se> parents: 6992 diff changeset	282 data = "item(" .. serialize(data) .. ");\n";
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	283 local ok, msg, where = append(username, host, datastore, "list", data);
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	284 if not ok then
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	285 log("error", "Unable to write to %s storage ('%s' in %s) for user: %s@%s",
ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	286 datastore, msg, where, username or "nil", host or "nil");
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	287 return ok, msg;
4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	288 end
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	289 if string.packsize then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	290 local offset = type(msg) == "number" and msg or 0;
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	291 local index_entry = string.pack(index_fmt, offset + #data);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	292 if offset == 0 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	293 index_entry = index_magic .. index_entry;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	294 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	295 local ok, off = append(username, host, datastore, "lidx", index_entry);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	296 off = off or 0;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	297 -- If this was the first item, then both the data and index offsets should
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	298 -- be zero, otherwise there's some kind of mismatch and we should drop the
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	299 -- index and recreate it from scratch
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	300 -- TODO Actually rebuild the index in this case?
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	301 if not ok or (off == 0 and offset ~= 0) or (off ~= 0 and offset == 0) then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	302 os_remove(getpath(username, host, datastore, "lidx"));
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	303 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	304 end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	305 return true;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	306 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	307
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	308 local function list_store(username, host, datastore, data)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	309 if not data then
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	310 data = {};
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	311 end
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	312 if callback(username, host, datastore) == false then return true; end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	313 -- save the datastore
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	314 local d = {};
8092 0a1c0f1107d2 util.datamanager: Use already known index instead of measuring length each iteration Kim Alvefur <zash@zash.se> parents: 8014 diff changeset	315 for i, item in ipairs(data) do
0a1c0f1107d2 util.datamanager: Use already known index instead of measuring length each iteration Kim Alvefur <zash@zash.se> parents: 8014 diff changeset	316 d[i] = "item(" .. serialize(item) .. ");\n";
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	317 end
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	318 os_remove(getpath(username, host, datastore, "lidx"));
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	319 local ok, msg = atomic_store(getpath(username, host, datastore, "list", true), t_concat(d));
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	320 if not ok then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	321 log("error", "Unable to write to %s storage ('%s') for user: %s@%s", datastore, msg, username or "nil", host or "nil");
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	322 return;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	323 end
915 0fe5bf7ab81d util.datamanager: Don't delete data when first entry in table is 'false'. My favourite bug so far. Matthew Wild <mwild1@gmail.com> parents: 896 diff changeset	324 if next(data) == nil then -- try to delete empty datastore
1732 f1282fad2f99 datamanager: Fixed logging errors on deletion of datastores not owned by a user@host Waqas Hussain <waqas20@gmail.com> parents: 1523 diff changeset	325 log("debug", "Removing empty %s datastore for user %s@%s", datastore, username or "nil", host or "nil");
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	326 os_remove(getpath(username, host, datastore, "list"));
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	327 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	328 -- we write data even when we are deleting because lua doesn't have a
12388 50fcd3879482 spelling: non-existing mistakes (thanks timeless) Kim Alvefur <zash@zash.se> parents: 12387 diff changeset	329 -- platform independent way of checking for nonexisting files
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	330 return true;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	331 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	332
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	333 local function build_list_index(username, host, datastore, items)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	334 log("debug", "Building index for (%s@%s/%s)", username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	335 local filename = getpath(username, host, datastore, "list");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	336 local fh, err, errno = io_open(filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	337 if not fh then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	338 return fh, err, errno;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	339 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	340 local prev_pos = 0; -- position before reading
13182 c48ae06e24d6 util.datamanager: Fix indexing first item if not at the very start Kim Alvefur <zash@zash.se> parents: 13181 diff changeset	341 local last_item_start = nil;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	342
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	343 if items and items[1] then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	344 local last_item = items[#items];
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	345 last_item_start = fh:seek("set", last_item.start + last_item.length);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	346 else
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	347 items = {};
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	348 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	349
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	350 for line in fh:lines() do
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	351 if line:sub(1, 4) == "item" then
13182 c48ae06e24d6 util.datamanager: Fix indexing first item if not at the very start Kim Alvefur <zash@zash.se> parents: 13181 diff changeset	352 if prev_pos ~= 0 and last_item_start then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	353 t_insert(items, { start = last_item_start; length = prev_pos - last_item_start });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	354 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	355 last_item_start = prev_pos
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	356 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	357 -- seek position is at the start of the next line within each loop iteration
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	358 -- so we need to collect the "current" position at the end of the previous
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	359 prev_pos = fh:seek()
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	360 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	361 if prev_pos ~= 0 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	362 t_insert(items, { start = last_item_start; length = prev_pos - last_item_start });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	363 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	364 return items;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	365 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	366
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	367 local function store_list_index(username, host, datastore, index)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	368 local data = { index_magic };
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	369 for i, v in ipairs(index) do
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	370 data[i + 1] = string.pack(index_fmt, v.start + v.length);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	371 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	372 local filename = getpath(username, host, datastore, "lidx");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	373 return atomic_store(filename, t_concat(data));
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	374 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	375
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	376 local index_mt = {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	377 __index = function(t, i)
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	378 if type(i) ~= "number" or i % 1 ~= 0 or i < 0 then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	379 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	380 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	381 if i <= 0 then
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	382 return 0
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	383 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	384 local fh = t.file;
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	385 local pos = (i - 1) * index_item_size;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	386 if fh:seek("set", pos) ~= pos then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	387 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	388 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	389 local data = fh:read(index_item_size * 2);
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	390 if not data or #data ~= index_item_size * 2 then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	391 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	392 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	393 local start, next_pos = string.unpack(index_fmt .. index_fmt, data);
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	394 if pos == 0 then
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	395 start = 0
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	396 end
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	397 local length = next_pos - start;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	398 local v = { start = start; length = length };
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	399 t[i] = v;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	400 return v;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	401 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	402 __len = function(t)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	403 -- Account for both the header and the fence post error
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	404 return floor(t.file:seek("end") / index_item_size) - 1;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	405 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	406 }
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	407
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	408 local function get_list_index(username, host, datastore)
13180 48622b89f570 util.datamanager: Fix missing separator in log line Kim Alvefur <zash@zash.se> parents: 13137 diff changeset	409 log("debug", "Loading index for (%s@%s/%s)", username, host, datastore);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	410 local index_filename = getpath(username, host, datastore, "lidx");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	411 local ih = io_open(index_filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	412 if ih then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	413 local magic = ih:read(#index_magic);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	414 if magic ~= index_magic then
13181 87487056bccb util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) Kim Alvefur <zash@zash.se> parents: 13180 diff changeset	415 log("debug", "Index %q has wrong version number (got %q, expected %q), rebuilding...", index_filename, magic, index_magic);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	416 -- wrong version or something
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	417 ih:close();
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	418 ih = nil;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	419 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	420 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	421
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	422 if ih then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	423 return setmetatable({ file = ih }, index_mt);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	424 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	425
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	426 local index, err = build_list_index(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	427 if not index then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	428 return index, err
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	429 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	430
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	431 -- TODO How to handle failure to store the index?
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	432 local dontcare = store_list_index(username, host, datastore, index); -- luacheck: ignore 211/dontcare
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	433 return index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	434 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	435
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	436 local function list_load_one(fh, start, length)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	437 if fh:seek("set", start) ~= start then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	438 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	439 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	440 local raw_data = fh:read(length)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	441 if not raw_data or #raw_data ~= length then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	442 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	443 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	444 local item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	445 local data, err, errno = envload(raw_data, "@list", {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	446 item = function(i)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	447 item = i;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	448 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	449 });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	450 if not data then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	451 return data, err, errno
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	452 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	453 local success, ret = pcall(data);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	454 if not success then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	455 return success, ret;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	456 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	457 return item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	458 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	459
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	460 local indexed_list_mt = {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	461 __index = function(t, i)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	462 if type(i) ~= "number" or i % 1 ~= 0 or i < 1 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	463 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	464 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	465 local ix = t.index[i];
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	466 if not ix then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	467 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	468 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	469 local item = list_load_one(t.file, ix.start, ix.length);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	470 return item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	471 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	472 __len = function(t)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	473 return #t.index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	474 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	475 }
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	476
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	477 local function list_load(username, host, datastore)
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	478 local items = {};
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	479 local data, err, errno = envloadfile(getpath(username, host, datastore, "list"), {item = function(i) t_insert(items, i); end});
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	480 if not data then
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	481 if errno == ENOENT then
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	482 -- No such file, ok to ignore
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	483 return nil;
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	484 end
7996 f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	485 log("error", "Failed to load %s storage ('%s') for user: %s@%s", datastore, err, username or "nil", host or "nil");
f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	486 return nil, "Error reading storage";
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	487 end
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	488
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	489 local success, ret = pcall(data);
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	490 if not success then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	491 log("error", "Unable to load %s storage ('%s') for user: %s@%s", datastore, ret, username or "nil", host or "nil");
3723 72a917d910a4 util.datamanager: Return an error string when pcall fails on a loaded list file. Waqas Hussain <waqas20@gmail.com> parents: 3722 diff changeset	492 return nil, "Error reading storage";
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	493 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	494 return items;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	495 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	496
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	497 local function list_open(username, host, datastore)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	498 if not index_magic then
13181 87487056bccb util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) Kim Alvefur <zash@zash.se> parents: 13180 diff changeset	499 log("debug", "Falling back from lazy loading to to loading full list for %s storage for user: %s@%s", datastore, username or "nil", host or "nil");
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	500 return list_load(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	501 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	502 local filename = getpath(username, host, datastore, "list");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	503 local file, err, errno = io_open(filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	504 if not file then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	505 if errno == ENOENT then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	506 return nil;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	507 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	508 return file, err, errno;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	509 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	510 local index, err = get_list_index(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	511 if not index then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	512 file:close()
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	513 return index, err;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	514 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	515 return setmetatable({ file = file; index = index }, indexed_list_mt);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	516 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	517
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	518 local function shift_index(index_filename, index, trim_to, offset)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	519 local index_scratch = index_filename .. "~";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	520 local new_index, err = io_open(index_scratch, "w");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	521 if not new_index then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	522 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	523 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	524 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	525
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	526 local ok, err = new_index:write(index_magic);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	527 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	528 new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	529 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	530 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	531 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	532 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	533
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	534 if not index.file or not index.file:seek("set", index_item_size * trim_to) then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	535 new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	536 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	537 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	538 return "deleted";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	539 else
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	540 local pack, unpack = string.pack, string.unpack;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	541 for item in index.file:lines(index_item_size) do
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	542 local ok, err = new_index:write(pack(index_fmt, unpack(index_fmt, item) - offset));
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	543 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	544 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	545 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	546 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	547 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	548 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	549 local ok, err = new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	550 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	551 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	552 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	553 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	554 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	555 return os_rename(index_scratch, index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	556 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	557 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	558
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	559 local function list_shift(username, host, datastore, trim_to)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	560 if trim_to == 1 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	561 return true
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	562 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	563 if type(trim_to) ~= "number" or trim_to < 1 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	564 return nil, "invalid-argument";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	565 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	566 local list_filename = getpath(username, host, datastore, "list");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	567 local index_filename = getpath(username, host, datastore, "lidx");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	568 local index, err = get_list_index(username, host, datastore);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	569 if not index then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	570 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	571 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	572
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	573 local new_first = index[trim_to];
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	574 if not new_first then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	575 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	576 return os_remove(list_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	577 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	578
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	579 local offset = new_first.start;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	580 if offset == 0 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	581 return true;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	582 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	583
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	584 if remove_blocks then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	585 local f, err = io_open(list_filename, "r+");
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	586 if not f then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	587 return f, err;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	588 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	589
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	590 local diff = 0;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	591 local block_offset = 0;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	592 if offset % 0x1000 ~= 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	593 -- Not an even block boundary, we will have to overwrite
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	594 diff = offset % 0x1000;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	595 block_offset = offset - diff;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	596 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	597
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	598 if block_offset == 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	599 log("debug", "")
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	600 else
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	601 local ok, err = remove_blocks(f, 0, block_offset);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	602 log("debug", "remove_blocks(%s, 0, %d)", f, block_offset);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	603 if not ok then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	604 log("warn", "Could not remove blocks from %q[%d, %d]: %s", list_filename, 0, block_offset, err);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	605 else
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	606 if diff ~= 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	607 -- overwrite unaligned leftovers
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	608 if f:seek("set", 0) then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	609 local wrote, err = f:write(string.rep("\n", diff));
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	610 if not wrote then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	611 log("error", "Could not blank out %q[%d, %d]: %s", list_filename, 0, diff, err);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	612 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	613 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	614 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	615 local ok, err = f:close();
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	616 shift_index(index_filename, index, trim_to, offset); -- Shift or delete the index
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	617 return ok, err;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	618 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	619 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	620 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	621
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	622 local r, err = io_open(list_filename, "r");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	623 if not r then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	624 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	625 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	626 local w, err = io_open(list_filename .. "~", "w");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	627 if not w then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	628 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	629 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	630 r:seek("set", offset);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	631 for block in r:lines(0x1000) do
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	632 local ok, err = w:write(block);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	633 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	634 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	635 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	636 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	637 r:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	638 local ok, err = w:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	639 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	640 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	641 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	642 shift_index(index_filename, index, trim_to, offset)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	643 return os_rename(list_filename .. "~", list_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	644 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	645
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	646
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	647 local type_map = {
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	648 keyval = "dat";
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	649 list = "list";
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	650 }
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	651
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	652 local function users(host, store, typ) -- luacheck: ignore 431/store
11369 87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	653 typ = "."..(type_map[typ or "keyval"] or typ);
10665 69acda92d5a9 util.datamanager: Fix iterating over "users" (thanks marc0s) Kim Alvefur <zash@zash.se> parents: 8555 diff changeset	654 local store_dir = format("%s/%s/%s", data_path, encode(host), store_encode(store));
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	655
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	656 local mode, err = lfs.attributes(store_dir, "mode");
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	657 if not mode then
6644 6cb6855f60df util.datamanager: Fix traceback due to %s in log message Kim Alvefur <zash@zash.se> parents: 6575 diff changeset	658 return function() log("debug", "%s", err or (store_dir .. " does not exist")) end
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	659 end
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	660 local next, state = lfs.dir(store_dir); -- luacheck: ignore 431/next 431/state
8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	661 return function(state) -- luacheck: ignore 431/state
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	662 for node in next, state do
11369 87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	663 if node:sub(-#typ, -1) == typ then
87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	664 return decode(node:sub(1, -#typ-1));
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	665 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	666 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	667 end, state;
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	668 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	669
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	670 local function stores(username, host, typ)
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	671 typ = type_map[typ or "keyval"];
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	672 local store_dir = format("%s/%s/", data_path, encode(host));
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	673
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	674 local mode, err = lfs.attributes(store_dir, "mode");
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	675 if not mode then
11288 5fd1f1b544a0 util.datamanager: Fix not passing variables as format string to logger (thanks semgrep) Kim Alvefur <zash@zash.se> parents: 10666 diff changeset	676 return function() log("debug", "Could not iterate over stores in %s: %s", store_dir, err); end
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	677 end
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	678 local next, state = lfs.dir(store_dir); -- luacheck: ignore 431/next 431/state
8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	679 return function(state) -- luacheck: ignore 431/state
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	680 for node in next, state do
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	681 if not node:match"^%." then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	682 if username == true then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	683 if lfs.attributes(store_dir..node, "mode") == "directory" then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	684 return decode(node);
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	685 end
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	686 elseif username then
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	687 local store_name = decode(node);
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	688 if lfs.attributes(getpath(username, host, store_name, typ), "mode") then
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	689 return store_name;
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	690 end
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	691 elseif lfs.attributes(node, "mode") == "file" then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	692 local file, ext = node:match("^(.*)%.([dalist]+)$");
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	693 if ext == typ then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	694 return decode(file)
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	695 end
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	696 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	697 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	698 end
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	699 end, state;
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	700 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	701
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	702 local function do_remove(path)
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	703 local ok, err = os_remove(path);
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	704 if not ok and lfs.attributes(path, "mode") then
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	705 return ok, err;
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	706 end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	707 return true
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	708 end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	709
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	710 local function purge(username, host)
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	711 local host_dir = format("%s/%s/", data_path, encode(host));
6681 0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	712 local ok, iter, state, var = pcall(lfs.dir, host_dir);
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	713 if not ok then
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	714 return ok, iter;
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	715 end
5095 dddbcd62183a util.datamanager: Collect errors when deleting all stores of a user, but ignore "no such file" Kim Alvefur <zash@zash.se> parents: 5069 diff changeset	716 local errs = {};
6681 0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	717 for file in iter, state, var do
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	718 if lfs.attributes(host_dir..file, "mode") == "directory" then
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	719 local store_name = decode(file);
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	720 local ok, err = do_remove(getpath(username, host, store_name));
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	721 if not ok then errs[#errs+1] = err; end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	722
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	723 local ok, err = do_remove(getpath(username, host, store_name, "list"));
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	724 if not ok then errs[#errs+1] = err; end
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	725 end
242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	726 end
5095 dddbcd62183a util.datamanager: Collect errors when deleting all stores of a user, but ignore "no such file" Kim Alvefur <zash@zash.se> parents: 5069 diff changeset	727 return #errs == 0, t_concat(errs, ", ");
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	728 end
242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	729
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	730 return {
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	731 set_data_path = set_data_path;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	732 add_callback = add_callback;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	733 remove_callback = remove_callback;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	734 getpath = getpath;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	735 load = load;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	736 store = store;
7002 9ab0d5e69c41 util.datamanager: Add append to public api Kim Alvefur <zash@zash.se> parents: 7001 diff changeset	737 append_raw = append;
7929 4017ea6d2599 util.datamanager: Expose atomic_store as store_raw Kim Alvefur <zash@zash.se> parents: 7928 diff changeset	738 store_raw = atomic_store;
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	739 list_append = list_append;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	740 list_store = list_store;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	741 list_load = list_load;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	742 users = users;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	743 stores = stores;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	744 purge = purge;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	745 path_decode = decode;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	746 path_encode = encode;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	747
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	748 build_list_index = build_list_index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	749 list_open = list_open;
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	750 list_shift = list_shift;
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	751 };

prosody

616c578c644f

util/datamanager.lua

Software / code / prosody

Annotate

util/datamanager.lua @ 13228:616c578c644f

Software `/` code `/` prosody