servers/ttrss
1
0
Fork 0
mirror of https://tt-rss.org/git/tt-rss.git synced 2024-07-05 13:20:55 +02:00
Code Issues Releases Wiki Activity
11,017 Commits 34 Branches 0 Tags 149 MiB
87a30d88d3
Commit Graph

161 Commits

Author SHA1 Message Date
Andrew Dolgov
8a02a728c8 add DAEMON_UNSUCCESSFUL_DAYS_LIMIT tunable (defaults to 30 days) 2020-09-30 17:03:16 +03:00
Andrew Dolgov
e641547d37 set ttrss_feeds.last_successful_update as needed 2020-09-30 16:35:50 +03:00
Andrew Dolgov
15e8ee3471 housekeeping: add task to cleanup orphan feed icons 2020-09-29 12:46:55 +03:00
Andrew Dolgov
82bc740363 Logger::log - allow specifying errno 
bump severity of PDO exception log messages to E_USER_WARNING
 2020-09-29 10:08:54 +03:00
Andrew Dolgov
23d20847a3 update_rss_feed: fallback to previous method if passthru() is not available 2020-09-28 21:19:53 +03:00
Andrew Dolgov
7c8bed0524 accept -1 as a valid exit code for per-feed update processes 2020-09-28 16:02:59 +03:00
Andrew Dolgov
335dcd3bf9 don't mention last_updated in non-zero failure error message because that's not what it means 2020-09-28 08:32:14 +03:00
Andrew Dolgov
3534b8dfa7 improve logging for per-feed update task failures 2020-09-28 08:02:38 +03:00
Andrew Dolgov
74cd60d7cc update_rss_feed: don't return as if failed on http 304 2020-09-27 17:13:36 +03:00
Andrew Dolgov
d4d0e976dc update-feed: exit with non-zero exit code if update_rss_feed() failed 
daemon: log if per-feed update task terminated with non-zero exit code
 2020-09-27 16:42:45 +03:00
Andrew Dolgov
0761533d0a lock per-feed update processes based on feed ID to reduce possibilty 
of concurrent updates
 2020-09-27 16:01:39 +03:00
Andrew Dolgov
528b387563 update individual feed in a separate process to prevent PHP fatal errors 
(for example, OOM) from stopping the entire batch
this should also slightly increase memory budget for update processes
 2020-09-27 15:58:13 +03:00
Andrew Dolgov
05744bb474 fix updater never scheduling feeds for update if they never been updated before while having default update interval set 2020-09-22 20:33:51 +03:00
Andrew Dolgov
6811d0bde2 use self:: in some places to invoke static methods from the same class 2020-09-22 14:54:15 +03:00
Andrew Dolgov
74568df4ff remove a lot of stuff from global context (functions.php), add a few helper classes instead 2020-09-22 09:04:33 +03:00
Andrew Dolgov
3dd4169b5f clarify some URL validation-related error messages 2020-09-21 20:35:24 +03:00
Andrew Dolgov
4785f21316 update_rss_feed: log effective URL after fetching 
validate_url: treat scheme as case-insensitive
 2020-09-21 20:26:57 +03:00
Andrew Dolgov
a4525d31b2 replace FALSE with false so that static analyzer shuts up about it 2020-09-17 19:02:27 +03:00
Andrew Dolgov
afa0023c51 don't try to update manually disabled feeds even if they haven't been updated before or are marked for a manual update 2020-09-17 15:40:50 +03:00
Andrew Dolgov
c352e872e9 core: pass found enclosures to HOOK_ARTICLE_FILTER 
af_redditimgur: remove enclosures if we found something to embed because it's going to be a low-res thumbnail
 2020-06-24 22:54:14 +03:00
Andrew Dolgov
6eb94f1e13 better support for image srcset attributes as discussed in https://community.tt-rss.org/t/problem-with-img-srcset/3519 2020-06-15 11:58:59 +03:00
Andrew Dolgov
06d2c65193 calculate_article_hash: don't die() on previous, woops 2020-05-17 17:44:32 +03:00
Andrew Dolgov
3a142cbf58 calculate_article_hash: ignore some useless or read-only fields (i.e. GUID) when calculating hash 2020-05-17 17:42:37 +03:00
Andrew Dolgov
cd1f3cb8cc * store UID in article hashed GUID separately so it could be migrated cleanly to a different instance 
* store resulting GUID as a JSON object so it could be extended easier if needed
 2020-05-17 14:01:16 +03:00
Andrew Dolgov
3a4b9249a9 DiskCache: properly deal with srcset attributes 2020-04-29 19:29:36 +03:00
Andrew Dolgov
4a00f96733 remove unneeded var_dump() 2020-04-29 11:35:02 +03:00
Andrew Dolgov
6573541873 * add HOOK_ENCLOSURE_IMPORTED 
* pass feed id to HOOK_FEED_PARSED
 2020-04-29 11:33:39 +03:00
lllusion3418
ec1b0befc7 add support for video[@src] in media cache 
it's a valid alternative to a source[@src] child element:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video
 2020-03-12 11:08:39 +01:00
lllusion3418
cdde23b4dc actually download <video> posters to media cache 
video[@poster] is already supported in the rewriting logic but never
actually downloaded
 2020-03-12 11:08:33 +01:00
Andrew Dolgov
f24ece85a6 add validationtextarea control, use it for filter match editor 2020-02-28 13:53:45 +03:00
Andrew Dolgov
6080cca9ca scrap counter cache system; rework counters to sum() booleans instead 2020-01-24 14:25:31 +03:00
Andrew Dolgov
e5b7b145e5 cache media: set referrer to source URL when fetching images 2019-11-25 09:48:24 +03:00
Andrew Dolgov
304d3a0b88 tag-related fixes 
1. move tag sanitization to feedparser common item class
2. enforce length limit on tags when parsing
3. support multiple tags passed via one dc:subject and other such elements, parse them as a comma-separated list
4. sort resulting tag list to prevent different order between feed updates
5. remove some duplicate code related to tag validation
6. allow + symbol in tags
 2019-11-20 18:56:34 +03:00
Andrew Dolgov
8c3efd51ec reset domain hit quota on feed update start 2019-11-17 13:17:21 +03:00
Andrew Dolgov
0d7b10469b update_rss_feed: add specific logging for HOOK_FETCH_FEED, HOOK_FEED_FETCHED, HOOK_FEED_PARSED handlers 2019-11-14 06:39:45 +03:00
Andrew Dolgov
5bb8dad631 is_gzipped: don't try to strpos() over entire buffer 2019-11-12 07:11:10 +03:00
Andrew Dolgov
647c7c45eb allow article filters to modify num_comments 2019-10-25 14:37:00 +03:00
Andrew Dolgov
4e05008aac update_rss_feed: force cast initial timestamp value to integer 2019-09-30 11:41:07 +03:00
Andrew Dolgov
b0d67cd3d0 rework previous to pass unformatted timestamp to plugin, and deal with formatting later 
also, move timestamp-related debugging output after plugin handler
 2019-09-11 14:04:59 +03:00
Andrew Dolgov
94a12b9674 pass formatted entry timestamp to article filters and allow them to modify it 2019-09-11 11:43:40 +03:00
Andrew Dolgov
6914ad1f74 retire MIN_CACHE_FILE_SIZE 2019-08-14 12:44:50 +03:00
Andrew Dolgov
84974c60a7 RSSUtils::cache_media, cache_enclosures: use DiskCache 2019-08-14 12:15:56 +03:00
Andrew Dolgov
fdb6066bf6 * HOOK_ENCLOSURE_ENTRY: pass article_id to handler 
* DiskCache: multiple fixes; support isWritable() for cache entries, set content-disposition for send()
* public/cached_url: allow selecting files from sub-caches other than images
* plugins/Cache_Starred_Images: rework to use DiskCache, can be enabled per-user, properly handles article enclosures, etc
 2019-08-13 16:40:21 +03:00
Andrew Dolgov
19b9b27662 expire_cached_files to DiskCache::expire() 2019-08-13 14:13:42 +03:00
Andrew Dolgov
088fcf8131 move more globals to more appropriate places 
set libxml to always use internal errors
 2019-06-20 08:40:02 +03:00
Andrew Dolgov
4fa9aee4e7 move several more global functions to more appropriate classes 2019-06-20 08:14:06 +03:00
Andrew Dolgov
9423d72f6c parser: force libxml error messages to valid utf8 2019-05-12 10:13:22 +03:00
Andrew Dolgov
c936cc3a1f use DEFAULT_SEARCH_LANGUAGE to generate tsvector index if per-feed language is not specified, also use it as default value on search form for convenience 2019-04-10 13:03:26 +03:00
Andrew Dolgov
671f4cee65 domdocument: remove old meta charset unicode hacks, replace with shorter xml preamble utf8 hack (on loadhtml where it makes sense) 
af_readability: better (?) charset hack for non-unicode pages
 2019-03-21 21:08:02 +03:00
Andrew Dolgov
33a2d5f8e4 update_rss_feed: set basic feed info if site_url is blank 2019-03-15 14:00:09 +03:00
Andrew Dolgov
69a691f4e1 cleanup old feed browser cache 2019-03-06 20:12:44 +03:00
Andrew Dolgov
0b74db5ad7 remove feedbrowser (other feeds) 2019-03-06 20:02:06 +03:00
Andrew Dolgov
38e01270d8 archived feeds: expire old entries (schema bump) 2019-03-06 19:06:05 +03:00
Andrew Dolgov
13e7e775a3 update_rss_feed: mark_unread_on_update should take into account catchup filter action and entry_force_catchup 2019-02-06 22:56:14 +03:00
Andrew Dolgov
949bfa3457 add minor clean()-ing on some rss feed values 2018-12-26 09:58:28 +03:00
Andrew Dolgov
eedd402807 rssutils: don't gzdecode() stuff 2018-12-21 17:52:41 +03:00
Andrew Dolgov
a5517fe857 fetch_file_contents: decompress gzipped data 
af_readability: remove utf8 preamble hack
 2018-12-21 17:50:16 +03:00
Andrew Dolgov
958fbfedb6 rssutils: check if returned data is in gzip format before trying to decode it 2018-12-14 14:55:36 +03:00
JustAMacUser
4b2f3039d2 Properly report filter plugin time (re-fixes PR 98). 2018-12-12 21:30:16 -05:00
JustAMacUser
53602096b9 Fixed misplaced bracket. 2018-12-12 11:47:36 -05:00
Andrew Dolgov
f3737c0b24 update_rss_feed: add log message if article is filtered out 
combine filters: fix crash on missing global function
 2018-12-08 17:01:30 +03:00
Andrew Dolgov
1e3a53c037 do not try to update filter triggers if nothing was triggered (properly this time) 2018-12-06 23:12:20 +03:00
Andrew Dolgov
5780a5d501 do not try to update filter triggers if nothing was triggered 2018-12-06 23:05:02 +03:00
Andrew Dolgov
3e4326e34d add ttrss_filters2.last_triggered (bump schema version) 2018-12-06 19:37:20 +03:00
Andrew Dolgov
a01c33d654 add HOOK_FILTER_TRIGGERED (for filter debugging) 2018-12-06 19:15:00 +03:00
Andrew Dolgov
3ad9944d5e fix missing sprintf() argument 2018-11-30 09:19:00 +03:00
Andrew Dolgov
c10a43069e debug logging system rework: 
* support various logging levels per-message
 * remove hacks like debug_suppress, DAEMON_EXTENDED_DEBUG, etc
 * _debug() is kept as a compatibility shim for plugins
 2018-11-30 08:34:29 +03:00
Andrew Dolgov
2d54eb1a87 remove cache/simplepie 2018-11-23 12:31:47 +03:00
Andrew Dolgov
2c940c4861 better handle PDOExceptions during open transaction in feed update 2018-11-03 15:00:58 +03:00
Andrew Dolgov
665495b94b cache_media: only touch() local file if it's writable 2018-09-10 16:17:12 +03:00
Andrew Dolgov
62d0060aa1 update_daemon_common: do not abort entire batch if PDOException happens when processing individual feeds 2018-09-03 13:03:35 +03:00
fox
8ab77d19ef Merge branch 'pullreq-enclosure-content-type' of tkappe/tt-rss into master 2018-08-21 03:47:39 +00:00
Tobias Kappé
ac8a0e7dc6 Differentiate enclosures based on content type. 
Some RSS feeds contain multiple enclosures with the same URL. When the first of
these is not recognized as an image, later entries are not added to the
database as rows in ttrss_enclosures. This change differentiates enclosures
based on their content type, so an entry can have multiple enclosure types with
the same URL (but possibly a different content type).
 2018-08-18 17:17:30 +01:00
Andrew Dolgov
163b50b15f cache_media: only show downloading debug message when actually downloading 2018-08-16 11:56:51 +03:00
Andrew Dolgov
069aea5989 remove FEED_CRYPT_KEY and everything related to it 
always assume auth_pass_encrypted is false
 2018-08-13 15:59:24 +03:00
Tobias Kappé
3bbaf902ab Sanitize language obtained for an entry. 2018-08-12 16:12:34 +01:00
Tobias Kappé
22a866edb5 Store language of entries as indicated by the feed. 2018-08-12 15:27:26 +01:00
BtbN
2b8afd4942 Only strip utf8mb4 if mysql_charset != utf8mb4 
If a user has fixed their database properly utf8mb4 works just fine allowing emoji and other 4 byte unicode characters to work.
 2018-08-04 13:07:32 +00:00
Andrew Dolgov
6e6c3a878d update_rss_feed: limit maximum length of tsvector data because of pgsql limitations 2018-07-10 14:08:55 +03:00
Andrew Dolgov
66fe33e769 bump date_updated when updated article data is saved to exclude it from purging (because it is still present in the originating feed) 2018-05-20 10:41:08 +03:00
Andrew Dolgov
963c22646b pass tsvector data as a named parameter on article update, remove escaping hacks 2018-04-03 13:57:27 +03:00
Andrew Dolgov
5edf4b73a4 add a workaround to support numeric tags 2018-02-11 10:24:24 +03:00
Andrew Dolgov
7f4a404566 include: convert some spaces to tabs 2018-01-30 10:44:31 +03:00
Andrew Dolgov
102a01354b strip utf8mb4 characters in enclosures on mysql 2018-01-29 23:24:11 +03:00
jsoares
26ad257de5 Fixed time stamping of new unmarked/unpublished articles 2018-01-22 22:41:24 +00:00
Andrew Dolgov
d4c05d0be2 update_rss_feed: don't try to use quoted NOW() in query 2017-12-18 18:27:23 +03:00
Richard Mortimer
aa16334f1f Include NOW() in prepared SQL for rssutils.php 2017-12-10 23:05:52 +00:00
Andrew Dolgov
e6532439d6 force strip_tags() on all user input unless explicitly allowed 2017-12-03 23:35:38 +03:00
Andrew Dolgov
7c6f7bb0aa fix some minor issues found by code analyzer 2017-12-03 23:08:04 +03:00
Andrew Dolgov
342e8a9eeb move feeds cache directory to cache/feeds 2017-12-03 21:01:35 +03:00
Andrew Dolgov
93e70e36c2 force article content/etc to string when updating to avoid failing null constraint check 2017-12-03 13:32:24 +03:00
Andrew Dolgov
49a888ecce rssutils: forbid question marks in tsvector data, PDO gets confused sometimes even by quoted ?s 2017-12-03 10:10:01 +03:00
Andrew Dolgov
187abfe732 main classes: remove sql_bool_to_bool() kludge 2017-12-03 09:35:59 +03:00
Andrew Dolgov
0500e14cc2 update_rss_feed: transaction lock article processing 2017-12-02 11:16:33 +03:00
Andrew Dolgov
0567016b40 rssutils: PDO 2017-12-02 09:56:34 +03:00
Andrew Dolgov
afcb105f4e rssutils: start PDO switch 2017-12-02 08:38:57 +03:00
Andrew Dolgov
e50c8eaa4e enforce unconditional requests every 6 hours even if server claims data is not modified 2017-11-30 13:12:28 +03:00
Andrew Dolgov
9d930af9e1 fetch_file_contents: improve error handling 
1. if request fails get error string from http  response status line
2. do not override http error with possible CURL/php specific last error
3. fix silent php error generated while processing response headers to get last modified value
 2017-10-30 13:13:10 +03:00
Gilles Grandou
f9ad33c2d8 allows favicons to be in Windows PC BMP format 2017-10-09 22:58:00 +02:00
wn_
3476690cbf Only require an array of basic info from 'HOOK_FEED_BASIC_INFO'. 
Removes the need for the plugin to provide feed content.

Gives plugins a chance to provide 'title' and 'site_url' basic info.
Falls back to attempting retrieval+parsing of the fetch URL if needed.
 2017-09-26 20:42:33 -05:00