Software `/` code `/` prosody

Annotate

plugins/mod_http_file_share.lua @ 11564:60e31c9ece57

mod_http_file_share: Support download resumption via Range requests Only a starting point is supported due to the way response:send_file() sends everything it gets from the provided file handle but does not have any way to specify how much to read. This matches what Conversations appears to be doing.

author	Kim Alvefur <zash@zash.se>
date	Sun, 16 May 2021 16:52:59 +0200
parent	11503:7adda14945ad
child	11568:d5360307a99d

rev	line source
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	1 -- Prosody IM
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	2 -- Copyright (C) 2021 Kim Alvefur
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	3 --
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	4 -- This project is MIT/X11 licensed. Please see the
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	5 -- COPYING file in the source package for more information.
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	6 --
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	7 -- XEP-0363: HTTP File Upload
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	8 -- Again, from the top!
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	9
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	10 local t_insert = table.insert;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	11 local jid = require "util.jid";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	12 local st = require "util.stanza";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	13 local url = require "socket.url";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	14 local dm = require "core.storagemanager".olddm;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	15 local jwt = require "util.jwt";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	16 local errors = require "util.error";
11314 7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	17 local dataform = require "util.dataforms".new;
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	18 local dt = require "util.datetime";
15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	19 local hi = require "util.human.units";
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	20 local cache = require "util.cache";
11495 6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	21 local lfs = require "lfs";
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	22
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	23 local namespace = "urn:xmpp:http:upload:0";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	24
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	25 module:depends("disco");
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	26
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	27 module:add_identity("store", "file", module:get_option_string("name", "HTTP File Upload"));
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	28 module:add_feature(namespace);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	29
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	30 local uploads = module:open_store("uploads", "archive");
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	31 -- id, <request>, time, owner
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	32
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	33 local secret = module:get_option_string(module.name.."_secret", require"util.id".long());
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	34 local external_base_url = module:get_option_string(module.name .. "_base_url");
11314 7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	35 local file_size_limit = module:get_option_number(module.name .. "_size_limit", 10 * 1024 * 1024); -- 10 MB
11315 c52fcea39c8e mod_http_file_share: Add file type filter Kim Alvefur <zash@zash.se> parents: 11314 diff changeset	36 local file_types = module:get_option_set(module.name .. "_allowed_file_types", {});
11333 f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	37 local safe_types = module:get_option_set(module.name .. "_safe_file_types", {"image/","video/","audio/*","text/plain"});
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	38 local expiry = module:get_option_number(module.name .. "_expires_after", 7 * 86400);
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	39 local daily_quota = module:get_option_number(module.name .. "_daily_quota", file_size_limit*10); -- 100 MB / day
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	40
11311 9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	41 local access = module:get_option_set(module.name .. "_access", {});
9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	42
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	43 if not external_base_url then
d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	44 module:depends("http");
d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	45 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	46
11314 7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	47 module:add_extension(dataform {
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	48 { name = "FORM_TYPE", type = "hidden", value = namespace },
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	49 { name = "max-file-size", type = "text-single" },
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	50 }:form({ ["max-file-size"] = tostring(file_size_limit) }, "result"));
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	51
11312 aade4a6179a3 mod_http_file_share: Return proper error if unauthorized Kim Alvefur <zash@zash.se> parents: 11311 diff changeset	52 local upload_errors = errors.init(module.name, namespace, {
11317 79e1f407b6f5 mod_http_file_share: Expand registry to fix extra tag Kim Alvefur <zash@zash.se> parents: 11316 diff changeset	53 access = { type = "auth"; condition = "forbidden" };
79e1f407b6f5 mod_http_file_share: Expand registry to fix extra tag Kim Alvefur <zash@zash.se> parents: 11316 diff changeset	54 filename = { type = "modify"; condition = "bad-request"; text = "Invalid filename" };
79e1f407b6f5 mod_http_file_share: Expand registry to fix extra tag Kim Alvefur <zash@zash.se> parents: 11316 diff changeset	55 filetype = { type = "modify"; condition = "not-acceptable"; text = "File type not allowed" };
79e1f407b6f5 mod_http_file_share: Expand registry to fix extra tag Kim Alvefur <zash@zash.se> parents: 11316 diff changeset	56 filesize = { type = "modify"; condition = "not-acceptable"; text = "File too large";
11318 3b16aba6285f mod_http_file_share: Fix name of max-file-size tag Kim Alvefur <zash@zash.se> parents: 11317 diff changeset	57 extra = {tag = st.stanza("file-too-large", {xmlns = namespace}):tag("max-file-size"):text(tostring(file_size_limit)) };
11330 f2c9492e3d25 mod_http_file_share: Fix the obligatory misplaced closing bracket (thanks scansion) Kim Alvefur <zash@zash.se> parents: 11329 diff changeset	58 };
11345 0fec04b64a49 mod_http_file_share: Add missing semicolon Kim Alvefur <zash@zash.se> parents: 11343 diff changeset	59 filesizefmt = { type = "modify"; condition = "bad-request"; text = "File size must be positive integer"; };
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	60 quota = { type = "wait"; condition = "resource-constraint"; text = "Daily quota reached"; };
11312 aade4a6179a3 mod_http_file_share: Return proper error if unauthorized Kim Alvefur <zash@zash.se> parents: 11311 diff changeset	61 });
aade4a6179a3 mod_http_file_share: Return proper error if unauthorized Kim Alvefur <zash@zash.se> parents: 11311 diff changeset	62
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	63 local upload_cache = cache.new(1024);
11348 f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	64 local quota_cache = cache.new(1024);
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	65
11491 c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	66 local measure_upload_cache_size = module:measure("upload_cache", "amount");
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	67 local measure_quota_cache_size = module:measure("quota_cache", "amount");
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	68
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	69 module:hook_global("stats-update", function ()
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	70 measure_upload_cache_size(upload_cache:count());
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	71 measure_quota_cache_size(quota_cache:count());
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	72 end);
c3fb802f9e45 mod_http_file_share: Report number of items in caches to statsmanager Kim Alvefur <zash@zash.se> parents: 11406 diff changeset	73
11355 89efa3f2966b mod_http_file_share: Collect statistics of files uploaded Kim Alvefur <zash@zash.se> parents: 11350 diff changeset	74 local measure_uploads = module:measure("upload", "sizes");
89efa3f2966b mod_http_file_share: Collect statistics of files uploaded Kim Alvefur <zash@zash.se> parents: 11350 diff changeset	75
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	76 -- Convenience wrapper for logging file sizes
15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	77 local function B(bytes) return hi.format(bytes, "B", "b"); end
15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	78
11325 76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	79 local function get_filename(slot, create)
76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	80 return dm.getpath(slot, module.host, module.name, "bin", create)
76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	81 end
76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	82
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	83 function get_daily_quota(uploader)
11347 5b3ad3c7fe47 mod_http_file_share: Split out some variables for later reuse Kim Alvefur <zash@zash.se> parents: 11346 diff changeset	84 local now = os.time();
5b3ad3c7fe47 mod_http_file_share: Split out some variables for later reuse Kim Alvefur <zash@zash.se> parents: 11346 diff changeset	85 local max_age = now - 86400;
11348 f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	86 local cached = quota_cache:get(uploader);
f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	87 if cached and cached.time > max_age then
f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	88 return cached.size;
f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	89 end
11347 5b3ad3c7fe47 mod_http_file_share: Split out some variables for later reuse Kim Alvefur <zash@zash.se> parents: 11346 diff changeset	90 local iter, err = uploads:find(nil, {with = uploader; start = max_age });
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	91 if not iter then return iter, err; end
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	92 local total_bytes = 0;
11349 a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	93 local oldest_upload = now;
11348 f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	94 for _, slot, when in iter do
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	95 local size = tonumber(slot.attr.size);
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	96 if size then total_bytes = total_bytes + size; end
11349 a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	97 if when < oldest_upload then oldest_upload = when; end
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	98 end
11349 a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	99 -- If there were no uploads then we end up caching [now, 0], which is fine
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	100 -- since we increase the size on new uploads
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	101 quota_cache:set(uploader, { time = oldest_upload, size = total_bytes });
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	102 return total_bytes;
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	103 end
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	104
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	105 function may_upload(uploader, filename, filesize, filetype) -- > boolean, error
11311 9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	106 local uploader_host = jid.host(uploader);
9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	107 if not ((access:empty() and prosody.hosts[uploader_host]) or access:contains(uploader) or access:contains(uploader_host)) then
11312 aade4a6179a3 mod_http_file_share: Return proper error if unauthorized Kim Alvefur <zash@zash.se> parents: 11311 diff changeset	108 return false, upload_errors.new("access");
11311 9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	109 end
9edda2026e57 mod_http_file_share: Add basic access control Kim Alvefur <zash@zash.se> parents: 11310 diff changeset	110
11313 e53894d26092 mod_http_file_share: Validate that filename does not contain '/' Kim Alvefur <zash@zash.se> parents: 11312 diff changeset	111 if not filename or filename:find"/" then
e53894d26092 mod_http_file_share: Validate that filename does not contain '/' Kim Alvefur <zash@zash.se> parents: 11312 diff changeset	112 -- On Linux, only '/' and '\0' are invalid in filenames and NUL can't be in XML
e53894d26092 mod_http_file_share: Validate that filename does not contain '/' Kim Alvefur <zash@zash.se> parents: 11312 diff changeset	113 return false, upload_errors.new("filename");
e53894d26092 mod_http_file_share: Validate that filename does not contain '/' Kim Alvefur <zash@zash.se> parents: 11312 diff changeset	114 end
e53894d26092 mod_http_file_share: Validate that filename does not contain '/' Kim Alvefur <zash@zash.se> parents: 11312 diff changeset	115
11319 a4b299e37909 mod_http_file_share: Reject invalid file sizes Kim Alvefur <zash@zash.se> parents: 11318 diff changeset	116 if not filesize or filesize < 0 or filesize % 1 ~= 0 then
a4b299e37909 mod_http_file_share: Reject invalid file sizes Kim Alvefur <zash@zash.se> parents: 11318 diff changeset	117 return false, upload_errors.new("filesizefmt");
a4b299e37909 mod_http_file_share: Reject invalid file sizes Kim Alvefur <zash@zash.se> parents: 11318 diff changeset	118 end
11314 7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	119 if filesize > file_size_limit then
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	120 return false, upload_errors.new("filesize");
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	121 end
7c8b02c5a335 mod_http_file_share: Add file size limit (default 10M) Kim Alvefur <zash@zash.se> parents: 11313 diff changeset	122
11346 315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	123 local uploader_quota = get_daily_quota(uploader);
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	124 if uploader_quota + filesize > daily_quota then
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	125 return false, upload_errors.new("quota");
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	126 end
315faec1a920 mod_http_file_share: Add support for daily upload quotas. Kim Alvefur <zash@zash.se> parents: 11345 diff changeset	127
11315 c52fcea39c8e mod_http_file_share: Add file type filter Kim Alvefur <zash@zash.se> parents: 11314 diff changeset	128 if not ( file_types:empty() or file_types:contains(filetype) or file_types:contains(filetype:gsub("/.", "/")) ) then
c52fcea39c8e mod_http_file_share: Add file type filter Kim Alvefur <zash@zash.se> parents: 11314 diff changeset	129 return false, upload_errors.new("filetype");
c52fcea39c8e mod_http_file_share: Add file type filter Kim Alvefur <zash@zash.se> parents: 11314 diff changeset	130 end
c52fcea39c8e mod_http_file_share: Add file type filter Kim Alvefur <zash@zash.se> parents: 11314 diff changeset	131
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	132 return true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	133 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	134
11350 3287dbdde33e mod_http_file_share: Reorder arguments Kim Alvefur <zash@zash.se> parents: 11349 diff changeset	135 function get_authz(slot, uploader, filename, filesize, filetype)
11502 8fd760c04cdf mod_http_file_share: Include time of issuance in auth token Kim Alvefur <zash@zash.se> parents: 11501 diff changeset	136 local now = os.time();
11322 4ade9810ce35 mod_http_file_share: Move Authorization type string Kim Alvefur <zash@zash.se> parents: 11321 diff changeset	137 return jwt.sign(secret, {
11501 2c9db2278fed mod_http_file_share: Group related properties for readability Kim Alvefur <zash@zash.se> parents: 11500 diff changeset	138 -- token properties
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	139 sub = uploader;
11502 8fd760c04cdf mod_http_file_share: Include time of issuance in auth token Kim Alvefur <zash@zash.se> parents: 11501 diff changeset	140 iat = now;
8fd760c04cdf mod_http_file_share: Include time of issuance in auth token Kim Alvefur <zash@zash.se> parents: 11501 diff changeset	141 exp = now+300;
11501 2c9db2278fed mod_http_file_share: Group related properties for readability Kim Alvefur <zash@zash.se> parents: 11500 diff changeset	142
2c9db2278fed mod_http_file_share: Group related properties for readability Kim Alvefur <zash@zash.se> parents: 11500 diff changeset	143 -- slot properties
2c9db2278fed mod_http_file_share: Group related properties for readability Kim Alvefur <zash@zash.se> parents: 11500 diff changeset	144 slot = slot;
11503 7adda14945ad mod_http_file_share: Include expiry time of the upload itself in token Kim Alvefur <zash@zash.se> parents: 11502 diff changeset	145 expires = expiry >= 0 and (now+expiry) or nil;
11501 2c9db2278fed mod_http_file_share: Group related properties for readability Kim Alvefur <zash@zash.se> parents: 11500 diff changeset	146 -- file properties
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	147 filename = filename;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	148 filesize = filesize;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	149 filetype = filetype;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	150 });
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	151 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	152
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	153 function get_url(slot, filename)
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	154 local base_url = external_base_url or module:http_url();
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	155 local slot_url = url.parse(base_url);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	156 slot_url.path = url.parse_path(slot_url.path or "/");
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	157 t_insert(slot_url.path, slot);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	158 if filename then
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	159 t_insert(slot_url.path, filename);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	160 slot_url.path.is_directory = false;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	161 else
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	162 slot_url.path.is_directory = true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	163 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	164 slot_url.path = url.build_path(slot_url.path);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	165 return url.build(slot_url);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	166 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	167
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	168 function handle_slot_request(event)
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	169 local stanza, origin = event.stanza, event.origin;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	170
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	171 local request = st.clone(stanza.tags[1], true);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	172 local filename = request.attr.filename;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	173 local filesize = tonumber(request.attr.size);
11320 817cadf6be92 mod_http_file_share: Handle content-type being optional Kim Alvefur <zash@zash.se> parents: 11319 diff changeset	174 local filetype = request.attr["content-type"] or "application/octet-stream";
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	175 local uploader = jid.bare(stanza.attr.from);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	176
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	177 local may, why_not = may_upload(uploader, filename, filesize, filetype);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	178 if not may then
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	179 origin.send(st.error_reply(stanza, why_not));
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	180 return true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	181 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	182
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	183 module:log("info", "Issuing upload slot to %s for %s", uploader, B(filesize));
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	184 local slot, storage_err = errors.coerce(uploads:append(nil, nil, request, os.time(), uploader))
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	185 if not slot then
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	186 origin.send(st.error_reply(stanza, storage_err));
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	187 return true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	188 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	189
11349 a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	190 local cached_quota = quota_cache:get(uploader);
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	191 if cached_quota and cached_quota.time > os.time()-86400 then
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	192 cached_quota.size = cached_quota.size + filesize;
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	193 quota_cache:set(uploader, cached_quota);
a219001b449d mod_http_file_share: Update cached value while it is reasonably fresh Kim Alvefur <zash@zash.se> parents: 11348 diff changeset	194 end
11348 f076199b4d38 mod_http_file_share: Cache quotas to avoid hitting storage Kim Alvefur <zash@zash.se> parents: 11347 diff changeset	195
11350 3287dbdde33e mod_http_file_share: Reorder arguments Kim Alvefur <zash@zash.se> parents: 11349 diff changeset	196 local authz = get_authz(slot, uploader, filename, filesize, filetype);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	197 local slot_url = get_url(slot, filename);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	198 local upload_url = slot_url;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	199
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	200 local reply = st.reply(stanza)
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	201 :tag("slot", { xmlns = namespace })
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	202 :tag("get", { url = slot_url }):up()
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	203 :tag("put", { url = upload_url })
11322 4ade9810ce35 mod_http_file_share: Move Authorization type string Kim Alvefur <zash@zash.se> parents: 11321 diff changeset	204 :text_tag("header", "Bearer "..authz, {name="Authorization"})
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	205 :reset();
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	206
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	207 origin.send(reply);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	208 return true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	209 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	210
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	211 function handle_upload(event, path) -- PUT /upload/:slot
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	212 local request = event.request;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	213 local authz = request.headers.authorization;
11326 1ecda954fe97 mod_http_file_share: Strip authorization type prefix a bit earlier Kim Alvefur <zash@zash.se> parents: 11325 diff changeset	214 if authz then
1ecda954fe97 mod_http_file_share: Strip authorization type prefix a bit earlier Kim Alvefur <zash@zash.se> parents: 11325 diff changeset	215 authz = authz:match("^Bearer (.*)")
1ecda954fe97 mod_http_file_share: Strip authorization type prefix a bit earlier Kim Alvefur <zash@zash.se> parents: 11325 diff changeset	216 end
1ecda954fe97 mod_http_file_share: Strip authorization type prefix a bit earlier Kim Alvefur <zash@zash.se> parents: 11325 diff changeset	217 if not authz then
11335 b7acab5e7f57 mod_http_file_share: Clarify message about missing Authorization header Kim Alvefur <zash@zash.se> parents: 11334 diff changeset	218 module:log("debug", "Missing or malformed Authorization header");
11336 b05331cff47a mod_http_file_share: Indicate missing token via WWW-Authenticate header Kim Alvefur <zash@zash.se> parents: 11335 diff changeset	219 event.response.headers.www_authenticate = "Bearer";
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	220 return 403;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	221 end
11326 1ecda954fe97 mod_http_file_share: Strip authorization type prefix a bit earlier Kim Alvefur <zash@zash.se> parents: 11325 diff changeset	222 local authed, upload_info = jwt.verify(secret, authz);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	223 if not (authed and type(upload_info) == "table" and type(upload_info.exp) == "number") then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	224 module:log("debug", "Unauthorized or invalid token: %s, %q", authed, upload_info);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	225 return 401;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	226 end
11334 dbba2d44fda2 mod_http_file_share: Allow started uploads to complete after token expired Kim Alvefur <zash@zash.se> parents: 11333 diff changeset	227 if not request.body_sink and upload_info.exp < os.time() then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	228 module:log("debug", "Authorization token expired on %s", dt.datetime(upload_info.exp));
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	229 return 410;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	230 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	231 if not path or upload_info.slot ~= path:match("^[^/]+") then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	232 module:log("debug", "Invalid upload slot: %q, path: %q", upload_info.slot, path);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	233 return 400;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	234 end
11323 a853a018eede mod_http_file_share: Validate file size early in HTTP PUT request Kim Alvefur <zash@zash.se> parents: 11322 diff changeset	235 if request.headers.content_length and tonumber(request.headers.content_length) ~= upload_info.filesize then
a853a018eede mod_http_file_share: Validate file size early in HTTP PUT request Kim Alvefur <zash@zash.se> parents: 11322 diff changeset	236 return 413;
a853a018eede mod_http_file_share: Validate file size early in HTTP PUT request Kim Alvefur <zash@zash.se> parents: 11322 diff changeset	237 -- Note: We don't know the size if the upload is streamed in chunked encoding,
a853a018eede mod_http_file_share: Validate file size early in HTTP PUT request Kim Alvefur <zash@zash.se> parents: 11322 diff changeset	238 -- so we also check the final file size on completion.
a853a018eede mod_http_file_share: Validate file size early in HTTP PUT request Kim Alvefur <zash@zash.se> parents: 11322 diff changeset	239 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	240
11325 76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	241 local filename = get_filename(upload_info.slot, true);
11324 494761f5d7da mod_http_file_share: Use '.bin' file extension Kim Alvefur <zash@zash.se> parents: 11323 diff changeset	242
11375 6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	243 do
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	244 -- check if upload has been completed already
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	245 -- we want to allow retry of a failed upload attempt, but not after it's been completed
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	246 local f = io.open(filename, "r");
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	247 if f then
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	248 f:close();
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	249 return 409;
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	250 end
6b687210975b mod_http_file_share: Prevent attempt to upload again after completion Kim Alvefur <zash@zash.se> parents: 11374 diff changeset	251 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	252
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	253 if not request.body_sink then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	254 module:log("debug", "Preparing to receive upload into %q, expecting %s", filename, B(upload_info.filesize));
11500 21706a581b8a mod_http_file_share: Log error opening file for writing Kim Alvefur <zash@zash.se> parents: 11499 diff changeset	255 local fh, err = io.open(filename.."~", "w");
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	256 if not fh then
11500 21706a581b8a mod_http_file_share: Log error opening file for writing Kim Alvefur <zash@zash.se> parents: 11499 diff changeset	257 module:log("error", "Could not open file for writing: %s", err);
21706a581b8a mod_http_file_share: Log error opening file for writing Kim Alvefur <zash@zash.se> parents: 11499 diff changeset	258 return 500;
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	259 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	260 request.body_sink = fh;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	261 if request.body == false then
11374 5b8aec0609f0 mod_http_file_share: Support sending 100 Continue Kim Alvefur <zash@zash.se> parents: 11357 diff changeset	262 if request.headers.expect == "100-continue" then
5b8aec0609f0 mod_http_file_share: Support sending 100 Continue Kim Alvefur <zash@zash.se> parents: 11357 diff changeset	263 request.conn:write("HTTP/1.1 100 Continue\r\n\r\n");
5b8aec0609f0 mod_http_file_share: Support sending 100 Continue Kim Alvefur <zash@zash.se> parents: 11357 diff changeset	264 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	265 return true;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	266 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	267 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	268
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	269 if request.body then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	270 module:log("debug", "Complete upload available, %s", B(#request.body));
15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	271 -- Small enough to have been uploaded already
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	272 local written, err = errors.coerce(request.body_sink:write(request.body));
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	273 if not written then
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	274 return err;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	275 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	276 request.body = nil;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	277 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	278
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	279 if request.body_sink then
11316 ae0461b37fbe mod_http_file_share: Verify final file size on completion of upload Kim Alvefur <zash@zash.se> parents: 11315 diff changeset	280 local final_size = request.body_sink:seek();
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	281 local uploaded, err = errors.coerce(request.body_sink:close());
11316 ae0461b37fbe mod_http_file_share: Verify final file size on completion of upload Kim Alvefur <zash@zash.se> parents: 11315 diff changeset	282 if final_size ~= upload_info.filesize then
ae0461b37fbe mod_http_file_share: Verify final file size on completion of upload Kim Alvefur <zash@zash.se> parents: 11315 diff changeset	283 -- Could be too short as well, but we say the same thing
ae0461b37fbe mod_http_file_share: Verify final file size on completion of upload Kim Alvefur <zash@zash.se> parents: 11315 diff changeset	284 uploaded, err = false, 413;
ae0461b37fbe mod_http_file_share: Verify final file size on completion of upload Kim Alvefur <zash@zash.se> parents: 11315 diff changeset	285 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	286 if uploaded then
11321 15ab878a7d23 mod_http_file_share: Add some logging Kim Alvefur <zash@zash.se> parents: 11320 diff changeset	287 module:log("debug", "Upload of %q completed, %s", filename, B(final_size));
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	288 assert(os.rename(filename.."~", filename));
11355 89efa3f2966b mod_http_file_share: Collect statistics of files uploaded Kim Alvefur <zash@zash.se> parents: 11350 diff changeset	289 measure_uploads(final_size);
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	290
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	291 upload_cache:set(upload_info.slot, {
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	292 name = upload_info.filename;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	293 size = tostring(upload_info.filesize);
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	294 type = upload_info.filetype;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	295 time = os.time();
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	296 });
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	297 return 201;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	298 else
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	299 assert(os.remove(filename.."~"));
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	300 return err;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	301 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	302 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	303
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	304 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	305
11357 8cb2a64b15da mod_http_file_share: Collect cache hit/miss statistics for downloads Kim Alvefur <zash@zash.se> parents: 11356 diff changeset	306 local download_cache_hit = module:measure("download_cache_hit", "rate");
8cb2a64b15da mod_http_file_share: Collect cache hit/miss statistics for downloads Kim Alvefur <zash@zash.se> parents: 11356 diff changeset	307 local download_cache_miss = module:measure("download_cache_miss", "rate");
8cb2a64b15da mod_http_file_share: Collect cache hit/miss statistics for downloads Kim Alvefur <zash@zash.se> parents: 11356 diff changeset	308
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	309 function handle_download(event, path) -- GET /uploads/:slot+filename
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	310 local request, response = event.request, event.response;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	311 local slot_id = path:match("^[^/]+");
11331 7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	312 local basename, filetime, filetype, filesize;
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	313 local cached = upload_cache:get(slot_id);
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	314 if cached then
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	315 module:log("debug", "Cache hit");
11357 8cb2a64b15da mod_http_file_share: Collect cache hit/miss statistics for downloads Kim Alvefur <zash@zash.se> parents: 11356 diff changeset	316 download_cache_hit();
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	317 basename = cached.name;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	318 filesize = cached.size;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	319 filetype = cached.type;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	320 filetime = cached.time;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	321 upload_cache:set(slot_id, cached);
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	322 -- TODO cache negative hits?
11331 7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	323 else
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	324 module:log("debug", "Cache miss");
11357 8cb2a64b15da mod_http_file_share: Collect cache hit/miss statistics for downloads Kim Alvefur <zash@zash.se> parents: 11356 diff changeset	325 download_cache_miss();
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	326 local slot, when = errors.coerce(uploads:get(nil, slot_id));
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	327 if not slot then
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	328 module:log("debug", "uploads:get(%q) --> not-found, %s", slot_id, when);
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	329 else
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	330 module:log("debug", "uploads:get(%q) --> %s, %d", slot_id, slot, when);
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	331 basename = slot.attr.filename;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	332 filesize = slot.attr.size;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	333 filetype = slot.attr["content-type"];
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	334 filetime = when;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	335 upload_cache:set(slot_id, {
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	336 name = basename;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	337 size = slot.attr.size;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	338 type = filetype;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	339 time = when;
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	340 });
3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	341 end
11331 7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	342 end
7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	343 if not basename then
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	344 return 404;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	345 end
11331 7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	346 local last_modified = os.date('!%a, %d %b %Y %H:%M:%S GMT', filetime);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	347 if request.headers.if_modified_since == last_modified then
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	348 return 304;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	349 end
11325 76fc73d39092 mod_http_file_share: Factor out function for generating full filename Kim Alvefur <zash@zash.se> parents: 11324 diff changeset	350 local filename = get_filename(slot_id);
11493 77f2d45799ed mod_http_file_share: Fix reporting of missing files Kim Alvefur <zash@zash.se> parents: 11491 diff changeset	351 local handle, ferr = io.open(filename);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	352 if not handle then
11499 a8cbbbb1f165 mod_http_file_share: Fix logging of error opening file Kim Alvefur <zash@zash.se> parents: 11496 diff changeset	353 module:log("error", "Could not open file for reading: %s", ferr);
11493 77f2d45799ed mod_http_file_share: Fix reporting of missing files Kim Alvefur <zash@zash.se> parents: 11491 diff changeset	354 -- This can be because the upload slot wasn't used, or the file disappeared
77f2d45799ed mod_http_file_share: Fix reporting of missing files Kim Alvefur <zash@zash.se> parents: 11491 diff changeset	355 -- somehow, or permission issues.
77f2d45799ed mod_http_file_share: Fix reporting of missing files Kim Alvefur <zash@zash.se> parents: 11491 diff changeset	356 return 410;
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	357 end
11333 f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	358
11564 60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	359 local request_range = request.headers.range;
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	360 local response_range;
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	361 if request_range then
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	362 local range_start, range_end = request_range:match("^bytes=(%d+)%-(%d*)$")
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	363 -- Only support resumption, ie ranges from somewhere in the middle until the end of the file.
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	364 if (range_start and range_start ~= "0" and range_start ~= filesize) and (range_end == "" or range_end == filesize) then
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	365 if handle:seek("set", tonumber(range_start)) then
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	366 response_range = "bytes "..range_start.."-"..filesize.."/"..filesize;
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	367 filesize = string.format("%d", tonumber(filesize)-tonumber(range_start));
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	368 end
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	369 end
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	370 end
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	371
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	372
11402 a3be7b3cf1e1 mod_http_file_share: Fix traceback on missing file-type Kim Alvefur <zash@zash.se> parents: 11398 diff changeset	373 if not filetype then
a3be7b3cf1e1 mod_http_file_share: Fix traceback on missing file-type Kim Alvefur <zash@zash.se> parents: 11398 diff changeset	374 filetype = "application/octet-stream";
a3be7b3cf1e1 mod_http_file_share: Fix traceback on missing file-type Kim Alvefur <zash@zash.se> parents: 11398 diff changeset	375 end
11333 f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	376 local disposition = "attachment";
f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	377 if safe_types:contains(filetype) or safe_types:contains(filetype:gsub("/.", "/")) then
f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	378 disposition = "inline";
f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	379 end
f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	380
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	381 response.headers.last_modified = last_modified;
11331 7a915fa49373 mod_http_file_share: Extract all file properties into variables earlier Kim Alvefur <zash@zash.se> parents: 11330 diff changeset	382 response.headers.content_length = filesize;
11402 a3be7b3cf1e1 mod_http_file_share: Fix traceback on missing file-type Kim Alvefur <zash@zash.se> parents: 11398 diff changeset	383 response.headers.content_type = filetype;
11333 f80056b97cf0 mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’) Kim Alvefur <zash@zash.se> parents: 11332 diff changeset	384 response.headers.content_disposition = string.format("%s; filename=%q", disposition, basename);
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	385
11564 60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	386 if response_range then
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	387 response.status_code = 206;
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	388 response.headers.content_range = response_range;
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	389 end
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	390 response.headers.accept_ranges = "bytes";
60e31c9ece57 mod_http_file_share: Support download resumption via Range requests Kim Alvefur <zash@zash.se> parents: 11503 diff changeset	391
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	392 response.headers.cache_control = "max-age=31556952, immutable";
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	393 response.headers.content_security_policy = "default-src 'none'; frame-ancestors 'none';"
11327 6f2b69469060 mod_http_file_share: More security headers Kim Alvefur <zash@zash.se> parents: 11326 diff changeset	394 response.headers.strict_transport_security = "max-age=31556952";
6f2b69469060 mod_http_file_share: More security headers Kim Alvefur <zash@zash.se> parents: 11326 diff changeset	395 response.headers.x_content_type_options = "nosniff";
6f2b69469060 mod_http_file_share: More security headers Kim Alvefur <zash@zash.se> parents: 11326 diff changeset	396 response.headers.x_frame_options = "DENY"; -- replaced by frame-ancestors in CSP?
6f2b69469060 mod_http_file_share: More security headers Kim Alvefur <zash@zash.se> parents: 11326 diff changeset	397 response.headers.x_xss_protection = "1; mode=block";
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	398
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	399 return response:send_file(handle);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	400 end
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	401
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	402 if expiry >= 0 and not external_base_url then
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	403 -- TODO HTTP DELETE to the external endpoint?
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	404 local array = require "util.array";
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	405 local async = require "util.async";
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	406 local ENOENT = require "util.pposix".ENOENT;
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	407
11329 2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	408 local function sleep(t)
2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	409 local wait, done = async.waiter();
2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	410 module:add_timer(t, done)
2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	411 wait();
2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	412 end
2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	413
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	414 local reaper_task = async.runner(function(boundary_time)
11496 8bf632540197 mod_http_file_share: Include storage in prune time measurement Kim Alvefur <zash@zash.se> parents: 11495 diff changeset	415 local prune_done = module:measure("prune", "times");
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	416 local iter, total = assert(uploads:find(nil, {["end"] = boundary_time; total = true}));
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	417
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	418 if total == 0 then
11343 f125ac529c22 mod_http_file_share: Clarify log message Kim Alvefur <zash@zash.se> parents: 11336 diff changeset	419 module:log("info", "No expired uploaded files to prune");
11356 43e5429ab355 mod_http_file_share: Measure how long it takes to prune expired files Kim Alvefur <zash@zash.se> parents: 11355 diff changeset	420 prune_done();
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	421 return;
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	422 end
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	423
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	424 module:log("info", "Pruning expired files uploaded earlier than %s", dt.datetime(boundary_time));
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	425
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	426 local obsolete_uploads = array();
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	427 local i = 0;
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	428 for slot_id in iter do
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	429 i = i + 1;
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	430 obsolete_uploads:push(slot_id);
11332 3e0dcdf6283e mod_http_file_share: Cache file metadata Kim Alvefur <zash@zash.se> parents: 11331 diff changeset	431 upload_cache:set(slot_id, nil);
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	432 end
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	433
11329 2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	434 sleep(0.1);
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	435 local n = 0;
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	436 local problem_deleting = false;
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	437 obsolete_uploads:filter(function(slot_id)
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	438 n = n + 1;
11329 2a431d3ad8f1 mod_http_file_share: Insert pauses to avoid blocknig for long periods Kim Alvefur <zash@zash.se> parents: 11328 diff changeset	439 if i % 100 == 0 then sleep(0.1); end
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	440 local filename = get_filename(slot_id);
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	441 local deleted, err, errno = os.remove(filename);
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	442 if deleted or errno == ENOENT then
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	443 return true;
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	444 else
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	445 module:log("error", "Could not delete file %q: %s", filename, err);
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	446 problem_deleting = true;
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	447 return false;
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	448 end
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	449 end);
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	450 -- obsolete_uploads now contains slot ids for which the files have been
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	451 -- deleted and that needs to be cleared from the database
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	452
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	453 local deletion_query = {["end"] = boundary_time};
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	454 if not problem_deleting then
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	455 module:log("info", "All (%d) expired files successfully deleted", n);
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	456 -- we can delete based on time
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	457 else
11405 ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	458 module:log("warn", "%d out of %d expired files could not be deleted", n-#obsolete_uploads, n);
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	459 -- we'll need to delete only those entries where the files were
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	460 -- successfully deleted, and then try again with the failed ones.
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	461 -- eventually the admin ought to notice and fix the permissions or
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	462 -- whatever the problem is.
ce8291e89d67 mod_http_file_share: Remove correct entries when not all expired files were deleted Kim Alvefur <zash@zash.se> parents: 11402 diff changeset	463 deletion_query = {ids = obsolete_uploads};
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	464 end
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	465
11406 9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	466 if #obsolete_uploads == 0 then
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	467 module:log("debug", "No metadata to remove");
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	468 else
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	469 local removed, err = uploads:delete(nil, deletion_query);
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	470
11406 9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	471 if removed == true or removed == n or removed == #obsolete_uploads then
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	472 module:log("debug", "Removed all metadata for expired uploaded files");
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	473 else
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	474 module:log("error", "Problem removing metadata for deleted files: %s", err);
9d6545a7d483 mod_http_file_share: Skip removal of nothing Kim Alvefur <zash@zash.se> parents: 11405 diff changeset	475 end
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	476 end
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	477
11356 43e5429ab355 mod_http_file_share: Measure how long it takes to prune expired files Kim Alvefur <zash@zash.se> parents: 11355 diff changeset	478 prune_done();
11328 ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	479 end);
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	480
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	481 module:add_timer(1, function ()
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	482 reaper_task:run(os.time()-expiry);
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	483 return 60*60;
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	484 end);
ceaa3cebf28b mod_http_file_share: Add support for removing old files (default 2 weeks) Kim Alvefur <zash@zash.se> parents: 11327 diff changeset	485 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	486
11495 6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	487 -- Reachable from the console
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	488 function check_files(query)
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	489 local issues = {};
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	490 local iter = assert(uploads:find(nil, query));
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	491 for slot_id, file in iter do
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	492 local filename = get_filename(slot_id);
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	493 local size, err = lfs.attributes(filename, "size");
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	494 if not size then
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	495 issues[filename] = err;
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	496 elseif tonumber(file.attr.size) ~= size then
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	497 issues[filename] = "file size mismatch";
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	498 end
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	499 end
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	500
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	501 return next(issues) == nil, issues;
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	502 end
6d3f84148729 mod_http_file_share: Add internal command to check files consistency Kim Alvefur <zash@zash.se> parents: 11493 diff changeset	503
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	504 module:hook("iq-get/host/urn:xmpp:http:upload:0:request", handle_slot_request);
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	505
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	506 if not external_base_url then
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	507 module:provides("http", {
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	508 streaming_uploads = true;
11398 a1f26d17d70f mod_http_file_share: Allow credentials via CORS (needed for auth token) Kim Alvefur <zash@zash.se> parents: 11394 diff changeset	509 cors = {
a1f26d17d70f mod_http_file_share: Allow credentials via CORS (needed for auth token) Kim Alvefur <zash@zash.se> parents: 11394 diff changeset	510 credentials = true;
a1f26d17d70f mod_http_file_share: Allow credentials via CORS (needed for auth token) Kim Alvefur <zash@zash.se> parents: 11394 diff changeset	511 };
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	512 route = {
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	513 ["PUT /*"] = handle_upload;
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	514 ["GET /*"] = handle_download;
11394 420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	515 ["GET /"] = function (event)
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	516 return prosody.events.fire_event("http-message", {
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	517 response = event.response;
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	518 ---
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	519 title = "Prosody HTTP Upload endpoint";
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	520 message = "This is where files will be uploaded to, and served from.";
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	521 warning = not (event.request.secure) and "This endpoint is not considered secure!" or nil;
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	522 }) or "This is the Prosody HTTP Upload endpoint.";
420787340339 mod_http_file_share: Return a message from the base URL Kim Alvefur <zash@zash.se> parents: 11375 diff changeset	523 end
11309 b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	524 }
b59aed75dc5e mod_http_file_share: Let's write another XEP-0363 implementation Kim Alvefur <zash@zash.se> parents: diff changeset	525 });
11310 d1a0f2e918c0 mod_http_file_share: Add support for external file upload service Kim Alvefur <zash@zash.se> parents: 11309 diff changeset	526 end

prosody

60e31c9ece57

plugins/mod_http_file_share.lua

Software / code / prosody

Annotate

plugins/mod_http_file_share.lua @ 11564:60e31c9ece57

Software `/` code `/` prosody