Software `/` code `/` prosody

Annotate

util/datamanager.lua @ 13231:6a11d92ae436

util.datamanager: Disable block alignment Until we have more test coverage. Somehow the index becomes incorrect after inserting padding, unclear why.

author	Kim Alvefur <zash@zash.se>
date	Fri, 21 Jul 2023 18:03:22 +0200
parent	13186:affaf6d08d26
child	13234:c886ea087b38

rev	line source
1523 841d61be198f Remove version number from copyright headers Matthew Wild <mwild1@gmail.com> parents: 1462 diff changeset	1 -- Prosody IM
2923 b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	2 -- Copyright (C) 2008-2010 Matthew Wild
b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	3 -- Copyright (C) 2008-2010 Waqas Hussain
4452 7de17ca4de14 util/datamanager: Use package.config to figure out directory seperator James Callahan <james@chatid.com> parents: 4112 diff changeset	4 --
758 b1885732e979 GPL->MIT! Matthew Wild <mwild1@gmail.com> parents: 732 diff changeset	5 -- This project is MIT/X11 licensed. Please see the
b1885732e979 GPL->MIT! Matthew Wild <mwild1@gmail.com> parents: 732 diff changeset	6 -- COPYING file in the source package for more information.
519 cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	7 --
cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	8
cccd610a0ef9 Insert copyright/license headers Matthew Wild <mwild1@gmail.com> parents: 456 diff changeset	9
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	10 local string = string;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	11 local format = string.format;
5051 71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	12 local setmetatable = setmetatable;
71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	13 local ipairs = ipairs;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	14 local char = string.char;
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	15 local pcall = pcall;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	16 local log = require "prosody.util.logger".init("datamanager");
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	17 local io_open = io.open;
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	18 local os_remove = os.remove;
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	19 local os_rename = os.rename;
5051 71253db26fda util.datamanager: Remove a few unused imports Kim Alvefur <zash@zash.se> parents: 5049 diff changeset	20 local tonumber = tonumber;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	21 local floor = math.floor;
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	22 local next = next;
7432 92f721226753 util.datamanager: Import tostring and type (fix global access) Kim Alvefur <zash@zash.se> parents: 7202 diff changeset	23 local type = type;
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	24 local t_insert = table.insert;
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	25 local t_concat = table.concat;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	26 local envloadfile = require"prosody.util.envload".envloadfile;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	27 local envload = require"prosody.util.envload".envload;
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	28 local serialize = require "prosody.util.serialization".serialize;
3086 931acb1188b1 util.datamanager: When failing to load a file, and the file exists, return nil, error. Waqas Hussain <waqas20@gmail.com> parents: 2925 diff changeset	29 local lfs = require "lfs";
9993 02a41315d275 Fix various spelling mistakes [codespell] Kim Alvefur <zash@zash.se> parents: 8555 diff changeset	30 -- Extract directory separator from package.config (an undocumented string that comes with lua)
7995 b99404a340fc util.datamanager: Split long line [luacheck] Kim Alvefur <zash@zash.se> parents: 7929 diff changeset	31 local path_separator = assert ( package.config:match ( "^([^\n]+)" ) , "package.config not in standard form" )
b99404a340fc util.datamanager: Split long line [luacheck] Kim Alvefur <zash@zash.se> parents: 7929 diff changeset	32
4093 36555949bd16 util.datamanager: Use prosody.paths.data as the initial value for data_path Matthew Wild <mwild1@gmail.com> parents: 3723 diff changeset	33 local prosody = prosody;
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	34
13186 affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	35 local blocksize = 0x1000;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	36 local raw_mkdir = lfs.mkdir;
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	37 local atomic_append;
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	38 local remove_blocks;
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	39 local ENOENT = 2;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	40 pcall(function()
12975 d10957394a3c util: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12388 diff changeset	41 local pposix = require "prosody.util.pposix";
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	42 raw_mkdir = pposix.mkdir or raw_mkdir; -- Doesn't trample on umask
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	43 atomic_append = pposix.atomic_append;
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	44 remove_blocks = pposix.remove_blocks;
7927 7132abcf669e util.datamanager: Import value of ENOENT from util.pposix (if available) Kim Alvefur <zash@zash.se> parents: 7925 diff changeset	45 ENOENT = pposix.ENOENT or ENOENT;
5118 0dc9e6c128c3 util.datamanager: Make the util.pposix dependency optional. Waqas Hussain <waqas20@gmail.com> parents: 5103 diff changeset	46 end);
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	47
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	48 local _ENV = nil;
8555 4f0f5b49bb03 vairious: Add annotation when an empty environment is set [luacheck] Kim Alvefur <zash@zash.se> parents: 8299 diff changeset	49 -- luacheck: std none
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	50
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	51 ---- utils -----
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	52 local encode, decode, store_encode;
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	53 do
7732 aba3dd84d9f0 util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimal Kim Alvefur <zash@zash.se> parents: 7674 diff changeset	54 local urlcodes = setmetatable({}, { __index = function (t, k) t[k] = char(tonumber(k, 16)); return t[k]; end });
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	55
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	56 decode = function (s)
7734 6a52415ed68a util.datamanager: Shorter pattern Kim Alvefur <zash@zash.se> parents: 7733 diff changeset	57 return s and (s:gsub("%%(%x%x)", urlcodes));
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	58 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	59
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	60 encode = function (s)
628 3712d36b6d25 Fixed URL encoding to generate %0x instead of %x Waqas Hussain <waqas20@gmail.com> parents: 615 diff changeset	61 return s and (s:gsub("%W", function (c) return format("%%%02x", c:byte()); end));
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	62 end
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	63
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	64 -- Special encode function for store names, which historically were unencoded.
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	65 -- All currently known stores use a-z and underscore, so this one preserves underscores.
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	66 store_encode = function (s)
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	67 return s and (s:gsub("[^_%w]", function (c) return format("%%%02x", c:byte()); end));
756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	68 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	69 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	70
8011 f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	71 if not atomic_append then
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	72 function atomic_append(f, data)
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	73 local pos = f:seek();
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	74 if not f:write(data) or not f:flush() then
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	75 f:seek("set", pos);
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	76 f:write((" "):rep(#data));
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	77 f:flush();
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	78 return nil, "write-failed";
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	79 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	80 return true;
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	81 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	82 end
f8ba814fe029 util.datamanager: Use pposix.atomic_append Kim Alvefur <zash@zash.se> parents: 7996 diff changeset	83
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	84 local _mkdir = {};
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	85 local function mkdir(path)
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	86 path = path:gsub("/", path_separator); -- TODO as an optimization, do this during path creation rather than here
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	87 if not _mkdir[path] then
2444 267d6482bac6 util.datamanager: Use pposix.mkdir if available Matthew Wild <mwild1@gmail.com> parents: 2328 diff changeset	88 raw_mkdir(path);
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	89 _mkdir[path] = true;
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	90 end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	91 return path;
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	92 end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	93
4108 e3e3aa286334 util.datamanager: Handle gracefully the lack of prosody.paths.data Matthew Wild <mwild1@gmail.com> parents: 4093 diff changeset	94 local data_path = (prosody and prosody.paths and prosody.paths.data) or ".";
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	95 local callbacks = {};
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	96
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	97 ------- API -------------
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	98
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	99 local function set_data_path(path)
1097 c5b33640a5f0 util.datamanager: Lower log level of 'Setting data path' to 'debug' Matthew Wild <mwild1@gmail.com> parents: 932 diff changeset	100 log("debug", "Setting data path to: %s", path);
452 613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	101 data_path = path;
613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	102 end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	103
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	104 local function callback(username, host, datastore, data)
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	105 for _, f in ipairs(callbacks) do
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	106 username, host, datastore, data = f(username, host, datastore, data);
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	107 if username == false then break; end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	108 end
4452 7de17ca4de14 util/datamanager: Use package.config to figure out directory seperator James Callahan <james@chatid.com> parents: 4112 diff changeset	109
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	110 return username, host, datastore, data;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	111 end
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	112 local function add_callback(func)
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	113 if not callbacks[func] then -- Would you really want to set the same callback more than once?
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	114 callbacks[func] = true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	115 callbacks[#callbacks+1] = func;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	116 return true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	117 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	118 end
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	119 local function remove_callback(func)
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	120 if callbacks[func] then
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	121 for i, f in ipairs(callbacks) do
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	122 if f == func then
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	123 callbacks[i] = nil;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	124 callbacks[f] = nil;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	125 return true;
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	126 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	127 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	128 end
932 1ed3e5fe165a Added: datamanager: Allow a callback to be installed which selectively prevents disk writes Waqas Hussain <waqas20@gmail.com> parents: 915 diff changeset	129 end
452 613c5c6bdce4 Added option core.data_path Waqas Hussain <waqas20@gmail.com> parents: 267 diff changeset	130
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	131 local function getpath(username, host, datastore, ext, create)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	132 ext = ext or "dat";
2328 2804d81206d8 util.datamanager: Store data stores with no host in '_global' folder Matthew Wild <mwild1@gmail.com> parents: 2159 diff changeset	133 host = (host and encode(host)) or "_global";
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	134 username = username and encode(username);
8299 756a2a00e7e7 util.datamanager: Encode the 'store' path component, preserving underscores Kim Alvefur <zash@zash.se> parents: 8092 diff changeset	135 datastore = store_encode(datastore);
84 d0a0bac6815e Added: Datastore support for hosts and global data in addition to users Waqas Hussain <waqas20@gmail.com> parents: 0 diff changeset	136 if username then
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	137 if create then mkdir(mkdir(mkdir(data_path).."/"..host).."/"..datastore); end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	138 return format("%s/%s/%s/%s.%s", data_path, host, datastore, username, ext);
5243 07e8256efcda util.datamanager: Remove dead code path Kim Alvefur <zash@zash.se> parents: 5153 diff changeset	139 else
643 8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	140 if create then mkdir(mkdir(data_path).."/"..host); end
8ff454831f7d Moved directory auto-creation to datamanager Waqas Hussain <waqas20@gmail.com> parents: 628 diff changeset	141 return format("%s/%s/%s.%s", data_path, host, datastore, ext);
84 d0a0bac6815e Added: Datastore support for hosts and global data in addition to users Waqas Hussain <waqas20@gmail.com> parents: 0 diff changeset	142 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	143 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	144
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	145 local function load(username, host, datastore)
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	146 local data, err, errno = envloadfile(getpath(username, host, datastore), {});
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	147 if not data then
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	148 if errno == ENOENT then
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	149 -- No such file, ok to ignore
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	150 return nil;
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	151 end
7996 f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	152 log("error", "Failed to load %s storage ('%s') for user: %s@%s", datastore, err, username or "nil", host or "nil");
f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	153 return nil, "Error reading storage";
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	154 end
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	155
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	156 local success, ret = pcall(data);
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	157 if not success then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	158 log("error", "Unable to load %s storage ('%s') for user: %s@%s", datastore, ret, username or "nil", host or "nil");
3086 931acb1188b1 util.datamanager: When failing to load a file, and the file exists, return nil, error. Waqas Hussain <waqas20@gmail.com> parents: 2925 diff changeset	159 return nil, "Error reading storage";
117 8e5c5e6a3240 Fixed: datamanager.store and datamanager.load could crash when username or host arguments were nil. (useful for server specific and global data). Waqas Hussain <waqas20@gmail.com> parents: 88 diff changeset	160 end
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	161 return ret;
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	162 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	163
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	164 local function atomic_store(filename, data)
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	165 local scratch = filename.."~";
10531 810abf34919a util.datamanager: Ignore unused 'errno' variable [luacheck] Kim Alvefur <zash@zash.se> parents: 9993 diff changeset	166 local f, ok, msg, errno; -- luacheck: ignore errno
810abf34919a util.datamanager: Ignore unused 'errno' variable [luacheck] Kim Alvefur <zash@zash.se> parents: 9993 diff changeset	167 -- TODO return util.error with code=errno?
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	168
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	169 f, msg, errno = io_open(scratch, "w");
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	170 if not f then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	171 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	172 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	173
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	174 ok, msg = f:write(data);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	175 if not ok then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	176 f:close();
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	177 os_remove(scratch);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	178 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	179 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	180
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	181 ok, msg = f:close();
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	182 if not ok then
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	183 os_remove(scratch);
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	184 return nil, msg;
5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	185 end
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	186
7202 5bf0ff3882aa util.datamanager: Explicit handling of each error condition (see #632) Kim Alvefur <zash@zash.se> parents: 7201 diff changeset	187 return os_rename(scratch, filename);
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	188 end
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	189
6575 bdaadf70a48f util.datamanager: Check that the global 'prosody' exists before using it (fixes nil indexing in use outside of prosody) Kim Alvefur <zash@zash.se> parents: 5441 diff changeset	190 if prosody and prosody.platform ~= "posix" then
5065 acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	191 -- os.rename does not overwrite existing files on Windows
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	192 -- TODO We could use Transactional NTFS on Vista and above
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	193 function atomic_store(filename, data)
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	194 local f, err = io_open(filename, "w");
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	195 if not f then return f, err; end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	196 local ok, msg = f:write(data);
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	197 if not ok then f:close(); return ok, msg; end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	198 return f:close();
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	199 end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	200 end
acfaf771f10e util.datamanager: Don't use os.rename on non-POSIX. It doesn't overwrite exisitng files on Windows. Waqas Hussain <waqas20@gmail.com> parents: 5057 diff changeset	201
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	202 local function store(username, host, datastore, data)
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	203 if not data then
e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	204 data = {};
e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	205 end
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	206
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	207 username, host, datastore, data = callback(username, host, datastore, data);
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	208 if username == false then
1381 46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	209 return true; -- Don't save this data at all
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	210 end
46a58df8557d util.datamanager: Allow multiple data storage callbacks, and allow them to modify parameters Matthew Wild <mwild1@gmail.com> parents: 1097 diff changeset	211
206 e30d0e30a0ff Datamanager now deletes files with no data Waqas Hussain <waqas20@gmail.com> parents: 182 diff changeset	212 -- save the datastore
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	213 local d = "return " .. serialize(data) .. ";\n";
5441 6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	214 local mkdir_cache_cleared;
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	215 repeat
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	216 local ok, msg = atomic_store(getpath(username, host, datastore, nil, true), d);
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	217 if not ok then
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	218 if not mkdir_cache_cleared then -- We may need to recreate a removed directory
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	219 _mkdir = {};
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	220 mkdir_cache_cleared = true;
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	221 else
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	222 log("error", "Unable to write to %s storage ('%s') for user: %s@%s", datastore, msg, username or "nil", host or "nil");
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	223 return nil, "Error saving to storage";
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	224 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	225 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	226 if next(data) == nil then -- try to delete empty datastore
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	227 log("debug", "Removing empty %s datastore for user %s@%s", datastore, username or "nil", host or "nil");
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	228 os_remove(getpath(username, host, datastore));
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	229 end
6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	230 -- we write data even when we are deleting because lua doesn't have a
12388 50fcd3879482 spelling: non-existing mistakes (thanks timeless) Kim Alvefur <zash@zash.se> parents: 12387 diff changeset	231 -- platform independent way of checking for nonexisting files
5441 6a5c622cc6d4 util.datamanager: Clear the cache of created directories on storage failure, and retry Matthew Wild <mwild1@gmail.com> parents: 5440 diff changeset	232 until ok;
0 3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	233 return true;
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	234 end
3e3171b59028 First commit, where do you want to go tomorrow? matthew parents: diff changeset	235
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	236 -- Append a blob of data to a file
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	237 local function append(username, host, datastore, ext, data)
7000 2b57f77985a3 util.datamanager: Make sure only strings are passed as data to append() Kim Alvefur <zash@zash.se> parents: 6999 diff changeset	238 if type(data) ~= "string" then return; end
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	239 local filename = getpath(username, host, datastore, ext, true);
6996 644b1bddc676 util.datamanager: No shadowing of variable [luacheck] Kim Alvefur <zash@zash.se> parents: 6995 diff changeset	240
8013 72cfbe377326 util.datamanager: Rearrange locals Kim Alvefur <zash@zash.se> parents: 8011 diff changeset	241 local f = io_open(filename, "r+");
5066 4be7093edde9 util.datamanager: Try to open in read+write mode, then retry with write mode if that fails (usually because it doesn't exist) Kim Alvefur <zash@zash.se> parents: 5057 diff changeset	242 if not f then
7928 9d014aca766f util.datamanager: Use atomic store function when writing the first entry of a list Kim Alvefur <zash@zash.se> parents: 7927 diff changeset	243 return atomic_store(filename, data);
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	244 -- File did probably not exist, let's create it
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	245 end
6999 0ad66d12113a util.datamanager: Add some comments about the append function Kim Alvefur <zash@zash.se> parents: 6998 diff changeset	246
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	247 local pos = f:seek("end");
13231 6a11d92ae436 util.datamanager: Disable block alignment Kim Alvefur <zash@zash.se> parents: 13186 diff changeset	248 --[[ TODO needs tests
13186 affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	249 if (blocksize-(pos%blocksize)) < (#data%blocksize) then
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	250 -- pad to blocksize with newlines so that the next item is both on a new
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	251 -- block and a new line
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	252 atomic_append(f, ("\n"):rep(blocksize-(pos%blocksize)));
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	253 pos = f:seek("end");
affaf6d08d26 util.datamanager: Pad list writes to avoid crossing block boundaries Kim Alvefur <zash@zash.se> parents: 13185 diff changeset	254 end
13231 6a11d92ae436 util.datamanager: Disable block alignment Kim Alvefur <zash@zash.se> parents: 13186 diff changeset	255 --]]
6997 0ab228bc21c6 util.datamanager: Handle potential issues from fallocate Kim Alvefur <zash@zash.se> parents: 6996 diff changeset	256
8013 72cfbe377326 util.datamanager: Rearrange locals Kim Alvefur <zash@zash.se> parents: 8011 diff changeset	257 local ok, msg = atomic_append(f, data);
6997 0ab228bc21c6 util.datamanager: Handle potential issues from fallocate Kim Alvefur <zash@zash.se> parents: 6996 diff changeset	258
6998 86607fe755b6 util.datamanager: Handle potential error from :write() call Kim Alvefur <zash@zash.se> parents: 6997 diff changeset	259 if not ok then
86607fe755b6 util.datamanager: Handle potential error from :write() call Kim Alvefur <zash@zash.se> parents: 6997 diff changeset	260 f:close();
7001 2743759ca1b5 util.datamanager: Return extra location info Kim Alvefur <zash@zash.se> parents: 7000 diff changeset	261 return ok, msg, "write";
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	262 end
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	263
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	264 ok, msg = f:close();
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	265 if not ok then
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	266 return ok, msg, "close";
6994 507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	267 end
507301531cf5 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) Kim Alvefur <zash@zash.se> parents: 6993 diff changeset	268
7001 2743759ca1b5 util.datamanager: Return extra location info Kim Alvefur <zash@zash.se> parents: 7000 diff changeset	269 return true, pos;
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	270 end
0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	271
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	272 local index_fmt, index_item_size, index_magic;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	273 if string.packsize then
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	274 index_fmt = "T"; -- offset to the end of the item, length can be derived from two index items
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	275 index_item_size = string.packsize(index_fmt);
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	276 index_magic = string.pack(index_fmt, 7767639 + 1); -- Magic string: T9 for "prosody", version number
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	277 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	278
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	279 local function list_append(username, host, datastore, data)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	280 if not data then return; end
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	281 if callback(username, host, datastore) == false then return true; end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	282 -- save the datastore
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	283
6993 dc0c6b8dc638 util.datamanager: Overwrite 'data' variable instead of shadownig it [luacheck] Kim Alvefur <zash@zash.se> parents: 6992 diff changeset	284 data = "item(" .. serialize(data) .. ");\n";
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	285 local ok, msg, where = append(username, host, datastore, "list", data);
6992 0622f2820d1d util.datamanager: Factor out code for appending bytes to a file Kim Alvefur <zash@zash.se> parents: 6777 diff changeset	286 if not ok then
8014 ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	287 log("error", "Unable to write to %s storage ('%s' in %s) for user: %s@%s",
ff3787033abb util.datamanager: Log where the error came from Kim Alvefur <zash@zash.se> parents: 8013 diff changeset	288 datastore, msg, where, username or "nil", host or "nil");
5045 4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	289 return ok, msg;
4ba6940deed0 util.datamanager: Use pposix.fallocate() to make sure appends succeed. Also add a fallback fallocate() Kim Alvefur <zash@zash.se> parents: 5038 diff changeset	290 end
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	291 if string.packsize then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	292 local offset = type(msg) == "number" and msg or 0;
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	293 local index_entry = string.pack(index_fmt, offset + #data);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	294 if offset == 0 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	295 index_entry = index_magic .. index_entry;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	296 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	297 local ok, off = append(username, host, datastore, "lidx", index_entry);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	298 off = off or 0;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	299 -- If this was the first item, then both the data and index offsets should
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	300 -- be zero, otherwise there's some kind of mismatch and we should drop the
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	301 -- index and recreate it from scratch
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	302 -- TODO Actually rebuild the index in this case?
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	303 if not ok or (off == 0 and offset ~= 0) or (off ~= 0 and offset == 0) then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	304 os_remove(getpath(username, host, datastore, "lidx"));
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	305 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	306 end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	307 return true;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	308 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	309
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	310 local function list_store(username, host, datastore, data)
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	311 if not data then
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	312 data = {};
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	313 end
1462 44780b856ce7 datamanager: Fixed incorrect callback result checking Waqas Hussain <waqas20@gmail.com> parents: 1381 diff changeset	314 if callback(username, host, datastore) == false then return true; end
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	315 -- save the datastore
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	316 local d = {};
8092 0a1c0f1107d2 util.datamanager: Use already known index instead of measuring length each iteration Kim Alvefur <zash@zash.se> parents: 8014 diff changeset	317 for i, item in ipairs(data) do
0a1c0f1107d2 util.datamanager: Use already known index instead of measuring length each iteration Kim Alvefur <zash@zash.se> parents: 8014 diff changeset	318 d[i] = "item(" .. serialize(item) .. ");\n";
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	319 end
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	320 os_remove(getpath(username, host, datastore, "lidx"));
5049 5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	321 local ok, msg = atomic_store(getpath(username, host, datastore, "list", true), t_concat(d));
5d685f123332 util.datamanager: Write to a temporary file and atomically move it into place Kim Alvefur <zash@zash.se> parents: 5045 diff changeset	322 if not ok then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	323 log("error", "Unable to write to %s storage ('%s') for user: %s@%s", datastore, msg, username or "nil", host or "nil");
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	324 return;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	325 end
915 0fe5bf7ab81d util.datamanager: Don't delete data when first entry in table is 'false'. My favourite bug so far. Matthew Wild <mwild1@gmail.com> parents: 896 diff changeset	326 if next(data) == nil then -- try to delete empty datastore
1732 f1282fad2f99 datamanager: Fixed logging errors on deletion of datastores not owned by a user@host Waqas Hussain <waqas20@gmail.com> parents: 1523 diff changeset	327 log("debug", "Removing empty %s datastore for user %s@%s", datastore, username or "nil", host or "nil");
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	328 os_remove(getpath(username, host, datastore, "list"));
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	329 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	330 -- we write data even when we are deleting because lua doesn't have a
12388 50fcd3879482 spelling: non-existing mistakes (thanks timeless) Kim Alvefur <zash@zash.se> parents: 12387 diff changeset	331 -- platform independent way of checking for nonexisting files
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	332 return true;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	333 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	334
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	335 local function build_list_index(username, host, datastore, items)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	336 log("debug", "Building index for (%s@%s/%s)", username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	337 local filename = getpath(username, host, datastore, "list");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	338 local fh, err, errno = io_open(filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	339 if not fh then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	340 return fh, err, errno;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	341 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	342 local prev_pos = 0; -- position before reading
13182 c48ae06e24d6 util.datamanager: Fix indexing first item if not at the very start Kim Alvefur <zash@zash.se> parents: 13181 diff changeset	343 local last_item_start = nil;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	344
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	345 if items and items[1] then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	346 local last_item = items[#items];
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	347 last_item_start = fh:seek("set", last_item.start + last_item.length);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	348 else
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	349 items = {};
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	350 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	351
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	352 for line in fh:lines() do
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	353 if line:sub(1, 4) == "item" then
13182 c48ae06e24d6 util.datamanager: Fix indexing first item if not at the very start Kim Alvefur <zash@zash.se> parents: 13181 diff changeset	354 if prev_pos ~= 0 and last_item_start then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	355 t_insert(items, { start = last_item_start; length = prev_pos - last_item_start });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	356 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	357 last_item_start = prev_pos
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	358 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	359 -- seek position is at the start of the next line within each loop iteration
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	360 -- so we need to collect the "current" position at the end of the previous
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	361 prev_pos = fh:seek()
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	362 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	363 if prev_pos ~= 0 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	364 t_insert(items, { start = last_item_start; length = prev_pos - last_item_start });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	365 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	366 return items;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	367 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	368
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	369 local function store_list_index(username, host, datastore, index)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	370 local data = { index_magic };
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	371 for i, v in ipairs(index) do
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	372 data[i + 1] = string.pack(index_fmt, v.start + v.length);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	373 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	374 local filename = getpath(username, host, datastore, "lidx");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	375 return atomic_store(filename, t_concat(data));
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	376 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	377
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	378 local index_mt = {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	379 __index = function(t, i)
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	380 if type(i) ~= "number" or i % 1 ~= 0 or i < 0 then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	381 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	382 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	383 if i <= 0 then
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	384 return 0
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	385 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	386 local fh = t.file;
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	387 local pos = (i - 1) * index_item_size;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	388 if fh:seek("set", pos) ~= pos then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	389 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	390 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	391 local data = fh:read(index_item_size * 2);
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	392 if not data or #data ~= index_item_size * 2 then
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	393 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	394 end
13137 b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	395 local start, next_pos = string.unpack(index_fmt .. index_fmt, data);
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	396 if pos == 0 then
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	397 start = 0
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	398 end
b417a49cc31b util.datamanager: Halve size of list index Kim Alvefur <zash@zash.se> parents: 13134 diff changeset	399 local length = next_pos - start;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	400 local v = { start = start; length = length };
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	401 t[i] = v;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	402 return v;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	403 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	404 __len = function(t)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	405 -- Account for both the header and the fence post error
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	406 return floor(t.file:seek("end") / index_item_size) - 1;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	407 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	408 }
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	409
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	410 local function get_list_index(username, host, datastore)
13180 48622b89f570 util.datamanager: Fix missing separator in log line Kim Alvefur <zash@zash.se> parents: 13137 diff changeset	411 log("debug", "Loading index for (%s@%s/%s)", username, host, datastore);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	412 local index_filename = getpath(username, host, datastore, "lidx");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	413 local ih = io_open(index_filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	414 if ih then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	415 local magic = ih:read(#index_magic);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	416 if magic ~= index_magic then
13181 87487056bccb util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) Kim Alvefur <zash@zash.se> parents: 13180 diff changeset	417 log("debug", "Index %q has wrong version number (got %q, expected %q), rebuilding...", index_filename, magic, index_magic);
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	418 -- wrong version or something
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	419 ih:close();
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	420 ih = nil;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	421 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	422 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	423
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	424 if ih then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	425 return setmetatable({ file = ih }, index_mt);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	426 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	427
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	428 local index, err = build_list_index(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	429 if not index then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	430 return index, err
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	431 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	432
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	433 -- TODO How to handle failure to store the index?
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	434 local dontcare = store_list_index(username, host, datastore, index); -- luacheck: ignore 211/dontcare
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	435 return index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	436 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	437
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	438 local function list_load_one(fh, start, length)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	439 if fh:seek("set", start) ~= start then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	440 return nil
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	441 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	442 local raw_data = fh:read(length)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	443 if not raw_data or #raw_data ~= length then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	444 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	445 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	446 local item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	447 local data, err, errno = envload(raw_data, "@list", {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	448 item = function(i)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	449 item = i;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	450 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	451 });
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	452 if not data then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	453 return data, err, errno
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	454 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	455 local success, ret = pcall(data);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	456 if not success then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	457 return success, ret;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	458 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	459 return item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	460 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	461
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	462 local indexed_list_mt = {
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	463 __index = function(t, i)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	464 if type(i) ~= "number" or i % 1 ~= 0 or i < 1 then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	465 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	466 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	467 local ix = t.index[i];
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	468 if not ix then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	469 return
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	470 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	471 local item = list_load_one(t.file, ix.start, ix.length);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	472 return item;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	473 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	474 __len = function(t)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	475 return #t.index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	476 end;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	477 }
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	478
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	479 local function list_load(username, host, datastore)
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	480 local items = {};
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	481 local data, err, errno = envloadfile(getpath(username, host, datastore, "list"), {item = function(i) t_insert(items, i); end});
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	482 if not data then
7925 209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	483 if errno == ENOENT then
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	484 -- No such file, ok to ignore
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	485 return nil;
209503ee3aaa util.datamanager: Ignore ENOENT (no such file) when loading data Kim Alvefur <zash@zash.se> parents: 7734 diff changeset	486 end
7996 f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	487 log("error", "Failed to load %s storage ('%s') for user: %s@%s", datastore, err, username or "nil", host or "nil");
f4c0fb54e16c util.datamanager: Remove check for file non-existence since ENOENT tells us that already Kim Alvefur <zash@zash.se> parents: 7995 diff changeset	488 return nil, "Error reading storage";
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	489 end
5021 85b2689dbcfe Eliminate direct setfenv usage Florian Zeitz <florob@babelmonkeys.de> parents: 4452 diff changeset	490
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	491 local success, ret = pcall(data);
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	492 if not success then
4993 5243b74a4cbb Hopefully inert commit to clean up logging across a number of modules, removing all cases of concatenation when building log messages Matthew Wild <mwild1@gmail.com> parents: 4452 diff changeset	493 log("error", "Unable to load %s storage ('%s') for user: %s@%s", datastore, ret, username or "nil", host or "nil");
3723 72a917d910a4 util.datamanager: Return an error string when pcall fails on a loaded list file. Waqas Hussain <waqas20@gmail.com> parents: 3722 diff changeset	494 return nil, "Error reading storage";
247 681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	495 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	496 return items;
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	497 end
681b29aa134f Added support for storing (and removing), loading and appending to lists of data to datamanager (for supporting offline messages) Waqas Hussain <waqas20@gmail.com> parents: 206 diff changeset	498
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	499 local function list_open(username, host, datastore)
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	500 if not index_magic then
13181 87487056bccb util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) Kim Alvefur <zash@zash.se> parents: 13180 diff changeset	501 log("debug", "Falling back from lazy loading to to loading full list for %s storage for user: %s@%s", datastore, username or "nil", host or "nil");
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	502 return list_load(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	503 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	504 local filename = getpath(username, host, datastore, "list");
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	505 local file, err, errno = io_open(filename);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	506 if not file then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	507 if errno == ENOENT then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	508 return nil;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	509 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	510 return file, err, errno;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	511 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	512 local index, err = get_list_index(username, host, datastore);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	513 if not index then
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	514 file:close()
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	515 return index, err;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	516 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	517 return setmetatable({ file = file; index = index }, indexed_list_mt);
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	518 end
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	519
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	520 local function shift_index(index_filename, index, trim_to, offset)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	521 local index_scratch = index_filename .. "~";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	522 local new_index, err = io_open(index_scratch, "w");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	523 if not new_index then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	524 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	525 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	526 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	527
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	528 local ok, err = new_index:write(index_magic);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	529 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	530 new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	531 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	532 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	533 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	534 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	535
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	536 if not index.file or not index.file:seek("set", index_item_size * trim_to) then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	537 new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	538 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	539 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	540 return "deleted";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	541 else
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	542 local pack, unpack = string.pack, string.unpack;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	543 for item in index.file:lines(index_item_size) do
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	544 local ok, err = new_index:write(pack(index_fmt, unpack(index_fmt, item) - offset));
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	545 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	546 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	547 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	548 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	549 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	550 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	551 local ok, err = new_index:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	552 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	553 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	554 os_remove(index_scratch);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	555 return "deleted", err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	556 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	557 return os_rename(index_scratch, index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	558 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	559 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	560
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	561 local function list_shift(username, host, datastore, trim_to)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	562 if trim_to == 1 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	563 return true
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	564 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	565 if type(trim_to) ~= "number" or trim_to < 1 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	566 return nil, "invalid-argument";
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	567 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	568 local list_filename = getpath(username, host, datastore, "list");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	569 local index_filename = getpath(username, host, datastore, "lidx");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	570 local index, err = get_list_index(username, host, datastore);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	571 if not index then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	572 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	573 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	574
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	575 local new_first = index[trim_to];
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	576 if not new_first then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	577 os_remove(index_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	578 return os_remove(list_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	579 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	580
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	581 local offset = new_first.start;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	582 if offset == 0 then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	583 return true;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	584 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	585
13185 b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	586 if remove_blocks then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	587 local f, err = io_open(list_filename, "r+");
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	588 if not f then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	589 return f, err;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	590 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	591
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	592 local diff = 0;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	593 local block_offset = 0;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	594 if offset % 0x1000 ~= 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	595 -- Not an even block boundary, we will have to overwrite
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	596 diff = offset % 0x1000;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	597 block_offset = offset - diff;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	598 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	599
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	600 if block_offset == 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	601 log("debug", "")
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	602 else
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	603 local ok, err = remove_blocks(f, 0, block_offset);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	604 log("debug", "remove_blocks(%s, 0, %d)", f, block_offset);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	605 if not ok then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	606 log("warn", "Could not remove blocks from %q[%d, %d]: %s", list_filename, 0, block_offset, err);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	607 else
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	608 if diff ~= 0 then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	609 -- overwrite unaligned leftovers
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	610 if f:seek("set", 0) then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	611 local wrote, err = f:write(string.rep("\n", diff));
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	612 if not wrote then
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	613 log("error", "Could not blank out %q[%d, %d]: %s", list_filename, 0, diff, err);
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	614 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	615 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	616 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	617 local ok, err = f:close();
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	618 shift_index(index_filename, index, trim_to, offset); -- Shift or delete the index
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	619 return ok, err;
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	620 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	621 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	622 end
b57f45165e1e util.datamanager: Efficiently remove whole blocks to shift lists Kim Alvefur <zash@zash.se> parents: 13183 diff changeset	623
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	624 local r, err = io_open(list_filename, "r");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	625 if not r then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	626 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	627 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	628 local w, err = io_open(list_filename .. "~", "w");
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	629 if not w then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	630 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	631 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	632 r:seek("set", offset);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	633 for block in r:lines(0x1000) do
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	634 local ok, err = w:write(block);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	635 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	636 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	637 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	638 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	639 r:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	640 local ok, err = w:close();
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	641 if not ok then
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	642 return nil, err;
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	643 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	644 shift_index(index_filename, index, trim_to, offset)
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	645 return os_rename(list_filename .. "~", list_filename);
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	646 end
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	647
33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	648
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	649 local type_map = {
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	650 keyval = "dat";
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	651 list = "list";
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	652 }
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	653
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	654 local function users(host, store, typ) -- luacheck: ignore 431/store
11369 87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	655 typ = "."..(type_map[typ or "keyval"] or typ);
10665 69acda92d5a9 util.datamanager: Fix iterating over "users" (thanks marc0s) Kim Alvefur <zash@zash.se> parents: 8555 diff changeset	656 local store_dir = format("%s/%s/%s", data_path, encode(host), store_encode(store));
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	657
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	658 local mode, err = lfs.attributes(store_dir, "mode");
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	659 if not mode then
6644 6cb6855f60df util.datamanager: Fix traceback due to %s in log message Kim Alvefur <zash@zash.se> parents: 6575 diff changeset	660 return function() log("debug", "%s", err or (store_dir .. " does not exist")) end
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	661 end
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	662 local next, state = lfs.dir(store_dir); -- luacheck: ignore 431/next 431/state
8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	663 return function(state) -- luacheck: ignore 431/state
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	664 for node in next, state do
11369 87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	665 if node:sub(-#typ, -1) == typ then
87105a9a11df util.datamanager: Support iterating over any file extension Kim Alvefur <zash@zash.se> parents: 11288 diff changeset	666 return decode(node:sub(1, -#typ-1));
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	667 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	668 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	669 end, state;
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	670 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5130 diff changeset	671
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	672 local function stores(username, host, typ)
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	673 typ = type_map[typ or "keyval"];
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	674 local store_dir = format("%s/%s/", data_path, encode(host));
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	675
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	676 local mode, err = lfs.attributes(store_dir, "mode");
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	677 if not mode then
11288 5fd1f1b544a0 util.datamanager: Fix not passing variables as format string to logger (thanks semgrep) Kim Alvefur <zash@zash.se> parents: 10666 diff changeset	678 return function() log("debug", "Could not iterate over stores in %s: %s", store_dir, err); end
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	679 end
7674 8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	680 local next, state = lfs.dir(store_dir); -- luacheck: ignore 431/next 431/state
8027eecc750f util.datamanager: Add annotations to ignore name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7673 diff changeset	681 return function(state) -- luacheck: ignore 431/state
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	682 for node in next, state do
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	683 if not node:match"^%." then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	684 if username == true then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	685 if lfs.attributes(store_dir..node, "mode") == "directory" then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	686 return decode(node);
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	687 end
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	688 elseif username then
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	689 local store_name = decode(node);
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	690 if lfs.attributes(getpath(username, host, store_name, typ), "mode") then
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	691 return store_name;
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	692 end
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	693 elseif lfs.attributes(node, "mode") == "file" then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	694 local file, ext = node:match("^(.*)%.([dalist]+)$");
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	695 if ext == typ then
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	696 return decode(file)
051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	697 end
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	698 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	699 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	700 end
5130 051d352ed03c storagemanager, datamanager, mod_storage_{internal,sql}: Replace list_stores() with an iterator version Kim Alvefur <zash@zash.se> parents: 5118 diff changeset	701 end, state;
5032 c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	702 end
c40ea227f8af util.datamanager: Add function for listing stores Kim Alvefur <zash@zash.se> parents: 5024 diff changeset	703
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	704 local function do_remove(path)
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	705 local ok, err = os_remove(path);
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	706 if not ok and lfs.attributes(path, "mode") then
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	707 return ok, err;
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	708 end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	709 return true
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	710 end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	711
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	712 local function purge(username, host)
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	713 local host_dir = format("%s/%s/", data_path, encode(host));
6681 0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	714 local ok, iter, state, var = pcall(lfs.dir, host_dir);
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	715 if not ok then
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	716 return ok, iter;
0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	717 end
5095 dddbcd62183a util.datamanager: Collect errors when deleting all stores of a user, but ignore "no such file" Kim Alvefur <zash@zash.se> parents: 5069 diff changeset	718 local errs = {};
6681 0217a04722c7 util.datamanager: Fix traceback from trying to purge when storage is empty or otherwise unaccessible (fixes #496) Kim Alvefur <zash@zash.se> parents: 6644 diff changeset	719 for file in iter, state, var do
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	720 if lfs.attributes(host_dir..file, "mode") == "directory" then
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	721 local store_name = decode(file);
177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	722 local ok, err = do_remove(getpath(username, host, store_name));
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	723 if not ok then errs[#errs+1] = err; end
5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	724
7673 177d569307fd util.datamanager: Rename variables to avoid name clashes [luacheck] Kim Alvefur <zash@zash.se> parents: 7432 diff changeset	725 local ok, err = do_remove(getpath(username, host, store_name, "list"));
5103 5a1488369c35 util.datamanager: Ignore errors if the file is gone after removing it Kim Alvefur <zash@zash.se> parents: 5095 diff changeset	726 if not ok then errs[#errs+1] = err; end
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	727 end
242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	728 end
5095 dddbcd62183a util.datamanager: Collect errors when deleting all stores of a user, but ignore "no such file" Kim Alvefur <zash@zash.se> parents: 5069 diff changeset	729 return #errs == 0, t_concat(errs, ", ");
5038 242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	730 end
242c62ff8e77 util.datamanager: Add function for removing all data belonging to a user Kim Alvefur <zash@zash.se> parents: 5032 diff changeset	731
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	732 return {
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	733 set_data_path = set_data_path;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	734 add_callback = add_callback;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	735 remove_callback = remove_callback;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	736 getpath = getpath;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	737 load = load;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	738 store = store;
7002 9ab0d5e69c41 util.datamanager: Add append to public api Kim Alvefur <zash@zash.se> parents: 7001 diff changeset	739 append_raw = append;
7929 4017ea6d2599 util.datamanager: Expose atomic_store as store_raw Kim Alvefur <zash@zash.se> parents: 7928 diff changeset	740 store_raw = atomic_store;
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	741 list_append = list_append;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	742 list_store = list_store;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	743 list_load = list_load;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	744 users = users;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	745 stores = stores;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	746 purge = purge;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	747 path_decode = decode;
5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	748 path_encode = encode;
13134 638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	749
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	750 build_list_index = build_list_index;
638f627e707f util.datamanager: Add O(1) list indexing with on-disk index Kim Alvefur <zash@zash.se> parents: 12975 diff changeset	751 list_open = list_open;
13183 33b114fbb5de util.datamanager: Add way to efficiently remove first items in a list Kim Alvefur <zash@zash.se> parents: 13182 diff changeset	752 list_shift = list_shift;
6777 5de6b93d0190 util.: Remove use of module() function, make all module functions local and return them in a table at the end Kim Alvefur <zash@zash.se>* parents: 6681 diff changeset	753 };

prosody

6a11d92ae436

util/datamanager.lua

Software / code / prosody

Annotate

util/datamanager.lua @ 13231:6a11d92ae436

Software `/` code `/` prosody