prosody
ddc07fb8dcd4
util/http.lua

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody

Annotate

util/http.lua @ 9879:ddc07fb8dcd4 0.11

mod_mam: Perform message expiry based on building an index by date (backport of 39ee70fbb009 from trunk) For each day, store a set of users that have new messages. To expire messages, we collect the union of sets of users from dates that fall outside the cleanup range. The previous algoritm did not work well with many users, especially with the default settings.
author Kim Alvefur <zash@zash.se>
date Fri, 22 Mar 2019 17:32:56 +0100
parent 9504:cfbea3064aa9
child 9759:1af5106a2c34
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5299
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
1 -- Prosody IM
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
2 -- Copyright (C) 2013 Florian Zeitz
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
3 --
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
4 -- This project is MIT/X11 licensed. Please see the
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
5 -- COPYING file in the source package for more information.
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
6 --
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
7
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
8 local format, char = string.format, string.char;
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
9 local pairs, ipairs, tonumber = pairs, ipairs, tonumber;
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
10 local t_insert, t_concat = table.insert, table.concat;
5299
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
11
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
12 local function urlencode(s)
5458
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
13 return s and (s:gsub("[^a-zA-Z0-9.~_-]", function (c) return format("%%%02x", c:byte()); end));
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
14 end
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
15 local function urldecode(s)
5458
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
16 return s and (s:gsub("%%(%x%x)", function (c) return char(tonumber(c,16)); end));
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
17 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
18
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
19 local function _formencodepart(s)
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
20 return s and (s:gsub("%W", function (c)
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
21 if c ~= " " then
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
22 return format("%%%02x", c:byte());
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
23 else
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
24 return "+";
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
25 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
26 end));
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
27 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
28
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
29 local function formencode(form)
5458
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
30 local result = {};
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
31 if form[1] then -- Array of ordered { name, value }
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
32 for _, field in ipairs(form) do
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
33 t_insert(result, _formencodepart(field.name).."=".._formencodepart(field.value));
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
34 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
35 else -- Unordered map of name -> value
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
36 for name, value in pairs(form) do
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
37 t_insert(result, _formencodepart(name).."=".._formencodepart(value));
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
38 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
39 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
40 return t_concat(result, "&");
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
41 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
42
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
43 local function formdecode(s)
5458
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
44 if not s:match("=") then return urldecode(s); end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
45 local r = {};
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
46 for k, v in s:gmatch("([^=&]*)=([^&]*)") do
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
47 k, v = k:gsub("%+", "%%20"), v:gsub("%+", "%%20");
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
48 k, v = urldecode(k), urldecode(v);
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
49 t_insert(r, { name = k, value = v });
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
50 r[k] = v;
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
51 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
52 return r;
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
53 end
84162b81c863 net.http, util.http: Move definitions of urlencode/decode and formencode/decode to util.http (possible to use them without unnecessary network-related dependencies)
Matthew Wild <mwild1@gmail.com>
parents: 5299
diff changeset
54
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
55 local function contains_token(field, token)
5299
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
56 field = ","..field:gsub("[ \t]", ""):lower()..",";
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
57 return field:find(","..token:lower()..",", 1, true) ~= nil;
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
58 end
cc9d460aa779 util.http: New module for HTTP helper functions
Florian Zeitz <florob@babelmonkeys.de>
parents:
diff changeset
59
9504
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
60 local function normalize_path(path, is_dir)
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
61 if is_dir then
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
62 if path:sub(-1,-1) ~= "/" then path = path.."/"; end
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
63 else
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
64 if path:sub(-1,-1) == "/" then path = path:sub(1, -2); end
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
65 end
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
66 if path:sub(1,1) ~= "/" then path = "/"..path; end
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
67 return path;
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
68 end
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
69
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
70 return {
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
71 urlencode = urlencode, urldecode = urldecode;
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
72 formencode = formencode, formdecode = formdecode;
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
73 contains_token = contains_token;
9504
cfbea3064aa9 mod_http: Move normalize_path to util.http
Kim Alvefur <zash@zash.se>
parents: 5471
diff changeset
74 normalize_path = normalize_path;
5471
34bfd26525f5 util.http: Refactor and import all necessary functions
Matthew Wild <mwild1@gmail.com>
parents: 5458
diff changeset
75 };