prosody
39ee70fbb009
net/cqueues.lua

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody

Annotate

net/cqueues.lua @ 9751:39ee70fbb009

mod_mam: Perform message expiry based on building an index by date For each day, store a set of users that have new messages. To expire messages, we collect the union of sets of users from dates that fall outside the cleanup range. The previous algoritm did not work well with many users, especially with the default settings.
author Kim Alvefur <zash@zash.se>
date Thu, 03 Jan 2019 17:25:43 +0100
parent 6538:f1eb66288f60
child 10996:d742095046f9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6514
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
1 -- Prosody IM
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
2 -- Copyright (C) 2014 Daurnimator
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
3 --
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
4 -- This project is MIT/X11 licensed. Please see the
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
5 -- COPYING file in the source package for more information.
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
6 --
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
7 -- This module allows you to use cqueues with a net.server mainloop
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
8 --
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
9
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
10 local server = require "net.server";
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
11 local cqueues = require "cqueues";
6538
f1eb66288f60 net.cqueues: Fix incorrect version check
daurnimator <quae@daurnimator.com>
parents: 6537
diff changeset
12 assert(cqueues.VERSION >= 20150113, "cqueues newer than 20150113 required")
6514
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
13
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
14 -- Create a single top level cqueue
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
15 local cq;
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
16
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
17 if server.cq then -- server provides cqueues object
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
18 cq = server.cq;
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
19 elseif server.get_backend() == "select" and server._addtimer then -- server_select
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
20 cq = cqueues.new();
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
21 local function step()
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
22 assert(cq:loop(0));
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
23 end
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
24
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
25 -- Use wrapclient (as wrapconnection isn't exported) to get server_select to watch cq fd
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
26 local handler = server.wrapclient({
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
27 getfd = function() return cq:pollfd(); end;
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
28 settimeout = function() end; -- Method just needs to exist
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
29 close = function() end; -- Need close method for 'closeall'
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
30 }, nil, nil, {});
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
31
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
32 -- Only need to listen for readable; cqueues handles everything under the hood
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
33 -- readbuffer is called when `select` notes an fd as readable
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
34 handler.readbuffer = step;
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
35
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
36 -- Use server_select low lever timer facility,
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
37 -- this callback gets called *every* time there is a timeout in the main loop
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
38 server._addtimer(function(current_time)
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
39 -- This may end up in extra step()'s, but cqueues handles it for us.
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
40 step();
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
41 return cq:timeout();
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
42 end);
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
43 elseif server.event and server.base then -- server_event
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
44 cq = cqueues.new();
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
45 -- Only need to listen for readable; cqueues handles everything under the hood
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
46 local EV_READ = server.event.EV_READ;
6537
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
47 -- Convert a cqueues timeout to an acceptable timeout for luaevent
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
48 local function luaevent_safe_timeout(cq)
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
49 local t = cq:timeout();
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
50 -- if you give luaevent 0 or nil, it re-uses the previous timeout.
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
51 if t == 0 then
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
52 t = 0.000001; -- 1 microsecond is the smallest that works (goes into a `struct timeval`)
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
53 elseif t == nil then -- pick something big if we don't have one
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
54 t = 0x7FFFFFFF; -- largest 32bit int
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
55 end
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
56 return t
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
57 end
6536
605a87e90e27 net.cqueues: Add workaround for luaevent callback getting collected
daurnimator <quae@daurnimator.com>
parents: 6514
diff changeset
58 local event_handle;
605a87e90e27 net.cqueues: Add workaround for luaevent callback getting collected
daurnimator <quae@daurnimator.com>
parents: 6514
diff changeset
59 event_handle = server.base:addevent(cq:pollfd(), EV_READ, function(e)
605a87e90e27 net.cqueues: Add workaround for luaevent callback getting collected
daurnimator <quae@daurnimator.com>
parents: 6514
diff changeset
60 -- Need to reference event_handle or this callback will get collected
605a87e90e27 net.cqueues: Add workaround for luaevent callback getting collected
daurnimator <quae@daurnimator.com>
parents: 6514
diff changeset
61 -- This creates a circular reference that can only be broken if event_handle is manually :close()'d
605a87e90e27 net.cqueues: Add workaround for luaevent callback getting collected
daurnimator <quae@daurnimator.com>
parents: 6514
diff changeset
62 local _ = event_handle;
6537
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
63 -- Run as many cqueues things as possible (with a timeout of 0)
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
64 -- If an error is thrown, it will break the libevent loop; but prosody resumes after logging a top level error
6514
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
65 assert(cq:loop(0));
6537
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
66 return EV_READ, luaevent_safe_timeout(cq);
e4d443d05626 net.cqueues: Fixes hardcoded timeout for first iteration
daurnimator <quae@daurnimator.com>
parents: 6536
diff changeset
67 end, luaevent_safe_timeout(cq));
6514
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
68 else
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
69 error "NYI"
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
70 end
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
71
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
72 return {
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
73 cq = cq;
d425fc41e59f net.cqueues: Add module that allows use of cqueues while still using net.server as main loop
daurnimator <quae@daurnimator.com>
parents:
diff changeset
74 }