prosody-modules
73a610c3c7a9
mod_http_upload_external/README.markdown

Find changesets by keywords by author, files, the commit message, revision number or hash, or revset expression.

Software / code / prosody-modules

Annotate

mod_http_upload_external/README.markdown @ 3168:73a610c3c7a9

mod_http_external: Link to prosody-filer (Go implementation)
author Matthew Wild <mwild1@gmail.com>
date Mon, 02 Jul 2018 13:13:23 +0100
parent 2823:f14bea5da323
child 3189:57332ea0c1c7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2334
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
1 ---
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
2 description: HTTP File Upload (external service)
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
3 labels: 'Stage-Alpha'
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
4 ---
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
5
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
6 Introduction
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
7 ============
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
8
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
9 This module implements [XEP-0363], which lets clients upload files
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
10 over HTTP to an external web server.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
11
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
12 This module generates URLs that are signed using a HMAC. Any web service that can authenticate
2823
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
13 these URLs can be used.
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
14
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
15 Implementations
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
16 ---------------
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
17
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
18 * [PHP implementation](https://hg.prosody.im/prosody-modules/raw-file/tip/mod_http_upload_external/share.php)
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
19 * [Python3+Flask implementation](https://github.com/horazont/xmpp-http-upload)
3168
73a610c3c7a9 mod_http_external: Link to prosody-filer (Go implementation)
Matthew Wild <mwild1@gmail.com>
parents: 2823
diff changeset
20 * [Go implementation, Prosody Filer](https://github.com/ThomasLeister/prosody-filer)
2823
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
21
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
22 To implement your own service compatible with this module, check out the implementation notes below
f14bea5da323 mod_http_upload_external: add Python service implementation
Jonas Wielicki <jonas@wielicki.name>
parents: 2334
diff changeset
23 (and if you publish your implementation - let us know!).
2334
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
24
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
25 Configuration
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
26 =============
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
27
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
28 Add `"http_upload_external"` to modules_enabled in your global section, or under the host(s) you wish
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
29 to use it on.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
30
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
31 External URL
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
32 ------------
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
33
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
34 You need to provide the path to the external service. Ensure it ends with '/'.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
35
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
36 For example, to use the PHP implementation linked above, you might set it to:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
37
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
38 ``` {.lua}
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
39 http_upload_external_base_url = "https://your.example.com/path/to/share.php/"
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
40 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
41
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
42 Secret
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
43 ------
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
44
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
45 Set a long and unpredictable string as your secret. This is so the upload service can verify that
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
46 the upload comes from mod_http_upload_external, and random strangers can't upload to your server.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
47
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
48 ``` {.lua}
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
49 http_upload_external_secret = "this is a secret string!"
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
50 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
51
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
52 You need to set exactly the same secret string in your external service.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
53
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
54 Limits
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
55 ------
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
56
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
57 A maximum file size can be set by:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
58
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
59 ``` {.lua}
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
60 http_upload_external_file_size_limit = 123 -- bytes
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
61 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
62
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
63 Default is 100MB (100\*1024\*1024).
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
64
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
65 Compatibility
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
66 =============
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
67
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
68 Works with Prosody 0.9.x and later.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
69
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
70 Implementation
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
71 ==============
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
72
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
73 To implement your own external service that is compatible with this module, you need to expose a
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
74 simple API that allows the HTTP GET, HEAD and PUT methods on arbitrary URLs located on your service.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
75
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
76 For example, if http_upload_external_base_url is set to `https://example.com/upload/` then your service
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
77 might receive the following requests:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
78
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
79 Upload a new file:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
80
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
81 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
82 PUT https://example.com/upload/foo/bar.jpg?v=49e9309ff543ace93d25be90635ba8e9965c4f23fc885b2d86c947a5d59e55b2
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
83 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
84
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
85 Recipient checks the file size and other headers:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
86
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
87 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
88 HEAD https://example.com/upload/foo/bar.jpg
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
89 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
90
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
91 Recipient downloads the file:
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
92
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
93 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
94 GET https://example.com/upload/foo/bar.jpg
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
95 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
96
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
97 The only tricky logic is in validation of the PUT request. Firstly, don't overwrite existing files (return 409 Conflict).
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
98
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
99 Then you need to validate the auth token. This will be in the URL query parameter 'v'. If it is absent, fail with 403 Forbidden.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
100
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
101 Calculate the expected auth token by reading the value of the Content-Length header of the PUT request. E.g. for a 1MB file
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
102 will have a Content-Length of '1048576'. Append this to the uploaded file name, separated by a space (0x20) character.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
103
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
104 For the above example, you would end up with the following string: "foo/bar.jpg 1048576"
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
105
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
106 The auth token is a SHA256 HMAC of this string, using the configured secret as the key. E.g.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
107
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
108 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
109 calculated_auth_token = hmac_sha256("foo/bar.jpg 1048576", "secret string")
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
110 ```
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
111
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
112 If this is not equal to the 'v' parameter provided in the upload URL, reject the upload with 403 Forbidden.
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
113
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
114 Note: your language/environment may provide a function for doing a constant-time comparison of these, to guard against
c728b2f77c7c mod_http_upload_external: Add README
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
115 timing attacks that may be used to discover the secret key.