compare urls irrespective of downloader

importfeed --force: Avoid creating duplicates of existing already
downloaded files when yt-dlp or a special remote was used.

diff --git a/CHANGELOG b/CHANGELOG
index 6eca06eba1b3d6c93aba5c62dfb96cf135a04a2a..d0f759cc6afd6094c046f603fa5a943fcf8cd035 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,8 @@ git-annex (10.20240130) UNRELEASED; urgency=medium
 
   * importfeed: Added --scrape option, which uses yt-dlp to screen scrape
     the equivilant of an RSS feed.
+  * importfeed --force: Avoid creating duplicates of existing
+    already downloaded files when yt-dlp or a special remote was used.
 
  -- Joey Hess <id@joeyh.name>  Mon, 29 Jan 2024 15:59:33 -0400
 

diff --git a/Command/ImportFeed.hs b/Command/ImportFeed.hs
index e4e78818cea6b064eeb2982c9e43cfbb4197e634..dbc8cb8bf0f056de6ad1ed822deff13c270be3f9 100644 (file)
--- a/Command/ImportFeed.hs
+++ b/Command/ImportFeed.hs
@@ -462,7 +462,7 @@ runDownload todownload url extension cache cv getter = do
                                in d </> show n ++ "_" ++ base
                tryanother = makeunique (n + 1) file
                alreadyexists = liftIO $ isJust <$> catchMaybeIO (R.getSymbolicLinkStatus (toRawFilePath f))
-               checksameurl k = ifM (elem url <$> getUrls k)
+               checksameurl k = ifM (elem url . map fst . map getDownloader <$> getUrls k)
                        ( return Nothing
                        , tryanother
                        )

diff --git a/doc/forum/importfeed_on_multiple___34__overlapping__34___playlists/comment_4_0bf6d5b03032c0b9fd35c8410aea832c._comment b/doc/forum/importfeed_on_multiple___34__overlapping__34___playlists/comment_4_0bf6d5b03032c0b9fd35c8410aea832c._comment
new file mode 100644 (file)
index 0000000..2b79698
--- /dev/null
+++ b/doc/forum/importfeed_on_multiple___34__overlapping__34___playlists/comment_4_0bf6d5b03032c0b9fd35c8410aea832c._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2024-02-02T19:27:41Z"
+ content="""
+Oh, but it seems that `importfeed --force` already has code to handle this.
+It checks if the file already exists and has the same url recorded as the url
+being imported. For some reason in that case, it says it failed to import
+the url, which seems a bit odd behavior to me, but in any case it doesn't
+add a "2_" file.
+
+That works for regular rss feeds, but does not work for yt-dlp urls.
+
+The reason is the url mangling done for yt-dlp urls (and other special
+remote urls). Fixed this bug.
+"""]]