perl-Data-Unique - Module to check for duplicate item with time expiration and disk persistence

Property Value
Distribution ALT Linux Sisyphus
Repository Autoimports noarch
Package filename perl-Data-Unique-0.02-alt1.noarch.rpm
Package name perl-Data-Unique
Package version 0.02
Package release alt1
Package architecture noarch
Package type rpm
Category Development/Perl
Homepage -
License -
Maintainer -
Download size 8.36 KB
Installed size 8.36 KB
Create a data structure that avoid duplicate entries (key) whith any data and add expiration time to clean old entries.
This module use  Storable::AMF0 for the persistence.
After some benchmark of various serialisation it is best compromise in read and write for huge quantity of data.
_use strict;
_use warnings;
_use Data::Dumper;
_use feature qw( say );
_use Time::HiRes qw(gettimeofday usleep );
_use Data::Unique;
_my $filename = '/tmp/dedup.test';
_my @dup;
_my $dedup = Data::Unique->new( { expiration => 10, file => $filename, gc => 5 } );
_for my $idx ( 1 .. 6 ) {
_    my ( $seconds, $microseconds ) = gettimeofday;
_    my $time = ( $seconds * 1000000 ) + $microseconds;
_    say "$idx -> $time";
_    $dedup->item( $time, { T => $idx } ) or say "no insertion ($$time already present)";
_    push @dup, $time if ( ( $idx % 2 ) == 0 );
_    usleep 10;
_say Data::Dumper::Dumper $dedup;
_say "Number of item=".$dedup->scalar;
_say "Expiration time ".$dedup->expiration;
_say "Number of item=".$dedup->scalar;
_say $dedup->expiration(6);
_sleep 15;
_#say "deleted item number=".$dedup->gc();
_say "Number of item=".$dedup->scalar;
_foreach my $ins (@dup) {
_   say  $dedup->item($ins, { T => time }) ? "inserting $ins" : "no insertion ($ins already present)";
_say "Expiration time ".$dedup->expiration;
_say "Number of item=".$dedup->scalar. '  =>  '.scalar( keys( %{ $dedup->{data} }));


Package Version Architecture Repository
perl-Data-Unique - - -


Name Value
/usr/share/perl5 -
perl(Storable/ -
rpmlib(PayloadIsLzma) -


Name Value
perl(Data/ = 0.020
perl-Data-Unique = 0.02-alt1


Type URL
Binary Package perl-Data-Unique-0.02-alt1.noarch.rpm
Source Package perl-Data-Unique-0.02-alt1.src.rpm

Install Howto

  1. Add the following line to /etc/apt/sources.list:
    rpm [Sisyphus] noarch autoimports
    rpm [Sisyphus] noarch autoimports
  2. Update the package index:
    # sudo apt-get update
  3. Install perl-Data-Unique rpm package:
    # sudo apt-get install perl-Data-Unique



See Also

Package Description
perl-Data-Unique-Name-20170810-alt1.noarch.rpm Generate a unique but stable name from a string
perl-Data-Unixish-1.570-alt1.noarch.rpm Implementation for Unixish, a data transformation framework
perl-Data-Unixish-ANSI-0.08-alt1.noarch.rpm dux functions related to text containing ANSI codes
perl-Data-Unixish-DNS-0.001-alt1.noarch.rpm dux functions related to DNS
perl-Data-Unixish-Perl-0.001-alt1.noarch.rpm dux functions related to Perl
perl-Data-Unixish-ansi-0.02-alt1.noarch.rpm dux functions related to text containing ANSI codes
perl-Data-Utilities-0.04-alt1.noarch.rpm perl module Data-Utilities
perl-Data-VRM-GB-0.03-alt1.noarch.rpm Extract data about British vehicle registration marks
perl-Data-VString-0.000004-alt1.noarch.rpm Perl extension to handle v-strings (often used as version strings)
perl-Data-Validate-Common-0.3-alt1.noarch.rpm Some common validator methods
perl-Data-Validate-Currency-0.1.0-alt1.noarch.rpm Module to validate if data is valid currency
perl-Data-Validate-DNS-CAA-0.04-alt1.noarch.rpm Validate DNS Certification Authority Authorization (CAA) values
perl-Data-Validate-DNS-NAPTR-Regexp-0.007-alt1.noarch.rpm Validate the NAPTR Regexp field per RFC 2915 / RFC 3403 Section 4
perl-Data-Validate-DNS-SSHFP-0.01-alt1.noarch.rpm Validate DNS SSH Fingerprint (SSHFP) Record Values
perl-Data-Validate-DNS-TLSA-0.02-alt1.noarch.rpm Validate DNS Transport Layer Security Association (TLSA) Record Values