Duplicate Removal Method of Large Data under Two Level Fingerprins Flow Based on Multi CPU Calculation
DOI:
CSTR:
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    This paper analyzes the importance of data removal. According to the existing data of the removal techniques and algorithms, it improves MD5 code fingerprint algorithm to calculate and optimize it, analyzes and recombines the fingerprint calculation of water level. Using the cache group to replace the single cache, we propose a new method to calculate the two water level fingerprints based on multi CPU to study and analyze the method. At the same time, it supports effectiveness of the method through the relevant tests and test data.

    Reference
    Related
    Cited by
Get Citation

贺建英,袁小艳,唐青松.大数据下基于多CPU的两级指纹流水计算去重方法.计算机系统应用,2015,24(8):206-211

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:January 28,2015
  • Revised:March 18,2015
  • Adopted:
  • Online: September 03,2015
  • Published:
Article QR Code
You are the firstVisitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063