Merge branch 'develop'

.travis.yml

@@ -2,7 +2,7 @@ language: c
 
 install:
     - sudo apt-get update
-    - sudo apt-get install python3-sphinx gettext python3-setuptools
+    - sudo apt-get install scons python3-sphinx gettext python3-setuptools
     - sudo apt-get install libblkid-dev libelf-dev libglib2.0-dev libjson-glib-dev
     - sudo apt-get install clang
     - sudo easy_install3 $(cat test-requirements.txt)

.version

@@ -1 +1 @@
-2.8.0 Maidenly Moose
+2.9.0 Odd Olm

CHANGELOG.md

@@ -4,6 +4,48 @@ All notable changes to this project will be documented in this file.
 
 The format follows [keepachangelog.com]. Please stick to it.
 
+## [2.9.0 Odd Olm] -- 2019-08-20
+
+### Added
+
+* An up-to-date COPR package for Fedora (thanks eclipseo).
+* Add --xattr as --xattr-write, --xattr-read & --write-unfinished to enable easy
+  checksum caching for the next run.
+* Unique option for json and csv formatters (-c json:unique)
+* New -k option for rmlint.sh to keep directory timestamps like before deletion.
+
+### Changed
+
+* Warn when using -j without -D.
+* The exist status is now EXIT_FAILURE when rmlint was interrupted.
+* Slightly changed progressbar ETA algorithm to deliver more stable results.
+* docs: added a few scripting examples to underline the possibility to use 3rd-party tools.
+* --dedupe options now checks if the files are already reflinks to reduce disk thrashing.
+* Made sure that -T dd is the same as specifying -D.
+
+### Deprecated
+
+Nothing was deprecated.
+
+### Removed
+
+Nothing was removed.
+
+### Fixed
+
+* Dry-run was not honoured when a user-defined command was used.
+* Fix crash when compiled with -msse4.2
+* Relative symbolic links were compared wrong when using -f (See issue #333)
+* gui: The size options wer enot correctly passed to rmlint (only Megabytes)
+* xattr: timestamps where often compared wrongly.
+* Fiemap (and thus reflink detection) code was improved.
+* --replay did not replay duplicate directories correctly.
+* --replay did not honour --hidden and --size correctly.
+* Various build issues.
+* gui: fix "render from selected" feature.
+* gui: generated script now correctly removes itself.
+* gui: fix display of paths with ampersands in them.
+
 ## [2.8.0 Maidenly Moose] -- 2018-10-30
 
 Mostly a bugfix release with smaller functional changes.

README.rst

@@ -38,11 +38,14 @@ Finds…
 
 **Differences to other duplicate finders:**
 
-- Extremely fast (no exaggeration, we promise!).
+- Extremely fast (no exaggeration, we promise!)
 - Paranoia mode for those who do not trust hashsums.
 - Many output formats.
 - No interactivity.
-- Search for files only newer than a certain ``mtime``. 
+- Search for files only newer than a certain ``mtime``.
+- Many ways to handle duplicates.
+- Caching and replaying.
+- ``btrfs`` support.
 - ...
 
 It runs and compiles under most Unices, including Linux, FreeBSD and Darwin.

SConstruct

@@ -287,7 +287,7 @@ def check_faccessat(context):
 def check_xattr(context):
     rc = 1
 
-    for func in ['getxattr', 'setxattr', 'removexattr']:
+    for func in ['getxattr', 'setxattr', 'removexattr', 'listxattr']:
         if tests.CheckFunc(
             context, func,
             header=
@@ -304,6 +304,27 @@ def check_xattr(context):
     return rc
 
 
+
+def check_lxattr(context):
+    rc = 1
+
+    for func in ['lgetxattr', 'lsetxattr', 'lremovexattr', 'llistxattr']:
+        if tests.CheckFunc(
+            context, func,
+            header=
+                '#include <sys/types.h>'
+                '#include <sys/xattr.h>'
+        ):
+            rc = 0
+            break
+
+    conf.env['HAVE_LXATTR'] = rc
+
+    context.did_show_result = True
+    context.Result(rc)
+    return rc
+
+
 def check_sha512(context):
     rc = 1
     if tests.CheckDeclaration(context, 'G_CHECKSUM_SHA512', includes='#include <glib.h>\n'):
@@ -562,6 +583,7 @@ conf = Configure(env, custom_tests={
     'check_libelf': check_libelf,
     'check_fiemap': check_fiemap,
     'check_xattr': check_xattr,
+    'check_lxattr': check_lxattr,
     'check_sha512': check_sha512,
     'check_blkid': check_blkid,
     'check_posix_fadvise': check_posix_fadvise,
@@ -695,6 +717,7 @@ conf.check_sys_block()
 conf.check_libelf()
 conf.check_fiemap()
 conf.check_xattr()
+conf.check_lxattr()
 conf.check_bigfiles()
 conf.check_sha512()
 conf.check_gettext()

docs/_static/benchmarks/found_items.html

@@ -28,4 +28,13 @@
                 #table-7e93a246-3029-4e5b-83fc-e25d48bf01fd tr:nth-child(2n-1) td {
                     background-color: #f2f2f2;
                 }
-                </style><table id="table-7e93a246-3029-4e5b-83fc-e25d48bf01fd"><thead><tr><th></th><th>rdfind</th><th>fdupes</th><th>rmlint</th><th>rmlint-paranoid</th><th>rmlint-replay</th><th>rmlint-v2.2.2</th><th>rmlint-v2.2.2-paranoid</th><th>rmlint-xxhash</th><th>rmlint-old</th><th>dupd</th><th>baseline.py</th></tr></thead><tbody><tr><td>Duplicates</td><td>0</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>39.656k</td><td>43.217k</td><td>67.931k</td></tr><tr><td>Originals</td><td>0</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>15.133k</td><td>16.109k</td><td>22.848k</td></tr></tbody></table>
+                </style>
+                <table id="table-7e93a246-3029-4e5b-83fc-e25d48bf01fd">
+                    <thead>
+                        <tr><th></th><th>fdupes</th><th>rmlint</th><th>rmlint-paranoid</th><th>rmlint-replay</th><th>rmlint-v2.2.2</th><th>rmlint-v2.2.2-paranoid</th><th>rmlint-xxhash</th><th>rmlint-old</th><th>dupd</th><th>baseline.py</th></tr>
+                    </thead>
+                    <tbody>
+                        <tr><td>Duplicates<td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>27.203k</td><td>39.656k</td><td>43.217k</td><td>67.931k</td></tr>
+                        <tr><td>Originals</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>16.115k</td><td>15.133k</td><td>16.109k</td><td>22.848k</td></tr>
+                    </tbody>
+                </table>

docs/benchmarks.rst

@@ -13,6 +13,12 @@ on ``rmlint``. You're of course free to interpret something different or re-run_
 the benchmarks on your own machine. The exact version of each program is given
 in the plots.
 
+.. warning::
+
+    This page is a little out of date. Help in updating it would be appreciated.
+    The performance characteristics of rmlint have improved overall, but
+    so might have other tools.
+
 It should be noted that it is very hard to compare these tools, since *each*
 tool investigated a slightly different amount of data and produces different
 results on the dataset below. This is partly due to the fact that some tools
@@ -95,7 +101,6 @@ case.
 .. raw:: html
    :file: _static/benchmarks/found_items.html
 
-|
 
 Surprisingly each tool found a different set of files. As stated above, direct
 comparison may not be possible here. For most tools except ``rdfind`` and

docs/conf.py

@@ -66,7 +66,7 @@
 
 # General information about the project.
 project = 'rmlint documentation'
-copyright = '2014-2015, Christopher Pahl & Daniel Thomas'
+copyright = '2014-2019, Christopher Pahl & Daniel Thomas'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/index.rst

@@ -129,8 +129,8 @@ Authors
 ``rmlint`` was and is written by: 
 
 ===================================  ============================= ===========================================
-*Christopher Pahl*                   https://github.com/sahib      2010-2018
-*Daniel Thomas*                      https://github.com/SeeSpotRun 2014-2018
+*Christopher Pahl*                   https://github.com/sahib      2010-2019
+*Daniel Thomas*                      https://github.com/SeeSpotRun 2014-2019
 ===================================  ============================= ===========================================
 
 Additional thanks to:

docs/install.rst

@@ -2,9 +2,9 @@ Installation
 ============
 
 Many major Linux distribution might already package ``rmlint`` -- but watch out for
-the version. This manual describes the rewrite of ``rmlint`` (i.e. version :math:`\geq 2`).
-Old versions before this might contain bugs, have design flaws or might eat your
-hamster. We recommend using the newest version.
+the version. If possible, we recommend using the `newest version`_ available.
+
+.. _`newest version`: https://github.com/sahib/rmlint/releases
 
 If there is no package yet or you want to try a development version, you gonna
 need to compile ``rmlint`` from source.
@@ -38,7 +38,7 @@ Here's a list of readily prepared commands for known operating systems:
 
   .. code-block:: bash
 
-    $ yum -y install git scons python3-sphinx gettext json-glib-devel
+    $ yum -y install pkgconf git scons python3-sphinx gettext json-glib-devel
     $ yum -y install glib2-devel libblkid-devel elfutils-libelf-devel
     # Optional dependencies for the GUI:
     $ yum -y install pygobject3 gtk3 librsvg2
@@ -50,6 +50,10 @@ Here's a list of readily prepared commands for known operating systems:
     $ dnf copr enable eclipseo/rmlint
     $ dnf install rmlint
 
+  Since **Fedora 29** we also have an `official package`_.
+
+.. _`official package`: https://bugzilla.redhat.com/show_bug.cgi?id=1655338
+
 .. _`Fedora Copr`: https://copr.fedorainfracloud.org/coprs/eclipseo/rmlint/
 
 * **ArchLinux:**
@@ -64,7 +68,7 @@ Here's a list of readily prepared commands for known operating systems:
 
   .. code-block:: bash
 
-    $ pacman -S git scons python-sphinx
+    $ pacman -S pkgconf git scons python-sphinx
     $ pacman -S glib2 libutil-linux elfutils json-glib
     # Optional dependencies for the GUI:
     $ pacman -S gtk3 python-gobject librsvg
@@ -82,20 +86,25 @@ Here's a list of readily prepared commands for known operating systems:
 .. _`PKGBUILD`: https://aur.archlinux.org/packages/rm/rmlint-git/PKGBUILD
 .. _`ArchLinux AUR`: https://aur.archlinux.org/packages/rmlint-git
 
-* **Ubuntu** :math:`\geq 12.04`:
+* **Debian** / **Ubuntu** :math:`\geq 12.04`:
+
+  Note: Debian also `ships an official package`_.
+  Use the below instructions if you need a more recent version.
 
   This most likely applies to most distributions that are derived from Ubuntu.
   Note that the ``GUI`` depends on ``GTK+ >= 3.12``! 
   Ubuntu 14.04 LTS and earlier `still ships`_  with ``3.10``.
 
   .. code-block:: bash
 
-    $ apt-get install git scons python3-sphinx python3-nose gettext build-essential
+    $ apt-get install pkg-config git scons python3-sphinx python3-nose gettext build-essential
     # Optional dependencies for more features:
     $ apt-get install libelf-dev libglib2.0-dev libblkid-dev libjson-glib-1.0 libjson-glib-dev
     # Optional dependencies for the GUI:
     $ apt-get install python3-gi gir1.2-rsvg gir1.2-gtk-3.0 python-cairo gir1.2-polkit-1.0 gir1.2-gtksource-3.0 
 
+ 
+.. _`ships an official package`: https://packages.debian.org/de/sid/rmlint
 .. _`still ships`: https://github.com/sahib/rmlint/issues/171#issuecomment-199070974
 
 * **macOS**
@@ -130,8 +139,7 @@ Here's a list of readily prepared commands for known operating systems:
 -----
 
 Send us a note if you want to see your distribution here or the instructions
-need an update.
-The commands above install the full dependencies, therefore
+need an update. The commands above install the full dependencies, therefore
 some packages might be stripped if you do not need the feature
 they enable. Only hard requirement for the commandline is ``glib``.