Software `/` code `/` prosody-modules

Annotate

mod_pubsub_feeds/feeds.lib.lua @ 5571:ca3c2d11823c

mod_pubsub_feeds: Track latest timestamp seen in feeds instead of last poll This should ensure that an entry that has a publish timestmap after the previously oldest post, but before the time of the last poll check, is published to the node. Previously if an entry would be skipped if it was published at 13:00 with a timestamp of 12:30, where the last poll was at 12:45. For feeds that lack a timestamp, it now looks for the first post that is not published, assuming that the feed is in reverse chronological order, then iterates back up from there.

author	Kim Alvefur <zash@zash.se>
date	Sun, 25 Jun 2023 16:27:55 +0200
parent	4428:b328ca621ba6

rev	line source
2132 b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	1 local st = require "util.stanza";
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	2 -- RSS->Atom translator
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	3 -- http://code.matthewwild.co.uk/lua-feeds/
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	4
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	5 -- Helpers to translate item child elements
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	6 local rss2atom = {};
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	7 function rss2atom.title(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	8 atom_entry:tag("title"):text(tag:get_text()):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	9 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	10
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	11 function rss2atom.link(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	12 atom_entry:tag("link", { href = tag:get_text() }):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	13 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	14
4428 b328ca621ba6 mod_pubsub_feeds: Translate RSS <enclosure> to Atom for podcasts Kim Alvefur <zash@zash.se> parents: 4427 diff changeset	15 function rss2atom.enclosure(atom_entry, tag)
b328ca621ba6 mod_pubsub_feeds: Translate RSS <enclosure> to Atom for podcasts Kim Alvefur <zash@zash.se> parents: 4427 diff changeset	16 atom_entry:tag("link", { rel = "enclosure", href = tag.attr.url, type = tag.attr.type, length = tag.attr.length }):up();
b328ca621ba6 mod_pubsub_feeds: Translate RSS <enclosure> to Atom for podcasts Kim Alvefur <zash@zash.se> parents: 4427 diff changeset	17 end
b328ca621ba6 mod_pubsub_feeds: Translate RSS <enclosure> to Atom for podcasts Kim Alvefur <zash@zash.se> parents: 4427 diff changeset	18
2132 b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	19 function rss2atom.author(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	20 atom_entry:tag("author")
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	21 :tag("email"):text(tag:get_text()):up()
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	22 :up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	23 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	24
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	25 function rss2atom.guid(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	26 atom_entry:tag("id"):text(tag:get_text()):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	27 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	28
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	29 function rss2atom.category(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	30 atom_entry:tag("category", { term = tag:get_text(), scheme = tag.attr.domain }):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	31 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	32
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	33 function rss2atom.description(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	34 atom_entry:tag("summary"):text(tag:get_text()):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	35 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	36
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	37 local months = {
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	38 jan = "01", feb = "02", mar = "03", apr = "04", may = "05", jun = "06";
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	39 jul = "07", aug = "08", sep = "09", oct = "10", nov = "11", dec = "12";
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	40 };
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	41
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	42 function rss2atom.pubDate(atom_entry, tag)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	43 local pubdate = tag:get_text():gsub("^%a+,", ""):gsub("^%s*", "");
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	44 local date, month, year, hour, minute, second, zone =
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	45 pubdate:match("^(%d%d?) (%a+) (%d+) (%d+):(%d+):?(%d) ?(.)$");
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	46 if not date then return; end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	47 if #date == 1 then
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	48 date = "0"..date;
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	49 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	50 month = months[month:sub(1,3):lower()];
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	51 if #year == 2 then -- GAH!
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	52 if tonumber(year) > 80 then
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	53 year = "19"..year;
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	54 else
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	55 year = "20"..year;
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	56 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	57 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	58 if zone == "UT" or zone == "GMT" then zone = "Z"; end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	59 if #second == 0 then
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	60 second = "00";
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	61 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	62 local date_string = string.format("%s-%s-%sT%s:%s:%s%s", year, month, date, hour, minute, second, zone);
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	63 atom_entry:tag("published"):text(date_string):up();
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	64 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	65
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	66 -- Translate a single item to atom
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	67 local function translate_rss(rss_feed)
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	68 local feed = st.stanza("feed", { xmlns = "http://www.w3.org/2005/Atom" });
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	69 local channel = rss_feed:get_child("channel");
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	70 -- TODO channel properties
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	71 for item in channel:childtags("item") do
2378 649f733aa3dc mod_pubsub_feeds/feeds.lib: Fix converting RSS items to more than one Atom entry (thanks walduhu) Kim Alvefur <zash@zash.se> parents: 2133 diff changeset	72 feed:tag("entry");
2133 85762420a2c0 mod_pubsub_feeds: Use correct loop variable Kim Alvefur <zash@zash.se> parents: 2132 diff changeset	73 for tag in item:childtags() do
2132 b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	74 local translator = rss2atom[tag.name];
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	75 if translator then
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	76 translator(feed, tag);
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	77 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	78 end
4427 c402b273f2e3 mod_pubsub_feeds: Copy Atom tags from RSS as-is Kim Alvefur <zash@zash.se> parents: 2378 diff changeset	79 -- Preserve Atom-namespaced items
c402b273f2e3 mod_pubsub_feeds: Copy Atom tags from RSS as-is Kim Alvefur <zash@zash.se> parents: 2378 diff changeset	80 for atomtag in item:childtags(nil, "http://www.w3.org/2005/Atom") do
c402b273f2e3 mod_pubsub_feeds: Copy Atom tags from RSS as-is Kim Alvefur <zash@zash.se> parents: 2378 diff changeset	81 feed:add_child(st.clone(atomtag));
c402b273f2e3 mod_pubsub_feeds: Copy Atom tags from RSS as-is Kim Alvefur <zash@zash.se> parents: 2378 diff changeset	82 end
2378 649f733aa3dc mod_pubsub_feeds/feeds.lib: Fix converting RSS items to more than one Atom entry (thanks walduhu) Kim Alvefur <zash@zash.se> parents: 2133 diff changeset	83 feed:reset();
2132 b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	84 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	85 return feed;
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	86 end
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	87
b149ea428b81 mod_pubsub_feeds: Switch to use util.xml for parsing feeds and include RSS to Atom translation code from lua-feeds Kim Alvefur <zash@zash.se> parents: diff changeset	88 return { translate_rss = translate_rss }

prosody-modules

ca3c2d11823c

mod_pubsub_feeds/feeds.lib.lua

Software / code / prosody-modules

Annotate

mod_pubsub_feeds/feeds.lib.lua @ 5571:ca3c2d11823c

Software `/` code `/` prosody-modules