Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Evolutionary Relationships through Genome Comparisons02:54

Evolutionary Relationships through Genome Comparisons

5.9K
Genome comparison is one of the excellent ways to interpret the evolutionary relationships between organisms. The basic principle of genome comparison is that if two species share a common feature, it is likely encoded by the DNA sequence conserved between both species. The advent of genome sequencing technologies in the late 20th century enabled scientists to understand the concept of conservation of domains between species and helped them to deduce evolutionary relationships across diverse...
5.9K
Survival Tree01:19

Survival Tree

125
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
 Building a Survival Tree
Constructing a...
125

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same journal

Constructing an Artificial Intelligence-Driven Multilingual Medical Health Education Chatbot with Domain-Specific Medical Knowledge.

Big data·2026
Same journal

Explainable Machine Learning-Based Prediction of Postoperative Hypoxemia in Elderly Patients Undergoing General Anesthesia.

Big data·2026
Same journal

Big Data-Driven Video Anomaly Detection Using VideoMAE for Visual Analytics in CCTV Surveillance.

Big data·2026
Same journal

Agentic Artificial Intelligence-Driven Explainable Deep Learning for Deciphering Noncoding Pathogenic Mechanisms of Delirium Through Genomic Big Data Integration.

Big data·2026
Same journal

Personalized Driven Instruction Through Explainable Agentic AI in Multicultural Higher Education Environments.

Big data·2026
Same journal

Big Data-Driven Explainable Agentic AI Decision Frameworks for Enterprise Innovation in FinTech Ecosystems.

Big data·2026

Related Experiment Video

Updated: Jul 30, 2025

High-resolution Single Particle Analysis from Electron Cryo-microscopy Images Using SPHIRE
13:28

High-resolution Single Particle Analysis from Electron Cryo-microscopy Images Using SPHIRE

Published on: May 16, 2017

50.4K

Small Files Problem Resolution via Hierarchical Clustering Algorithm.

Oded Koren1, Aviel Shamalov1, Nir Perel2

  • 1School of Industrial Engineering and Management, Shenkar College of Engineering, Design and Art, Ramat Gan, Israel.

Big Data
|May 17, 2023
PubMed
Summary
This summary is machine-generated.

The Small Files Problem in Hadoop Distributed File System (HDFS) is addressed by a new Hierarchical Clustering Algorithm. This method efficiently merges small CSV files, reducing memory usage and improving HDFS performance.

Keywords:
Big DataHDFSblockshierarchical clusteringmachine learningsmall files

More Related Videos

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations
12:27

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations

Published on: February 15, 2017

7.0K
ExCYT: A Graphical User Interface for Streamlining Analysis of High-Dimensional Cytometry Data
05:12

ExCYT: A Graphical User Interface for Streamlining Analysis of High-Dimensional Cytometry Data

Published on: January 16, 2019

11.5K

Related Experiment Videos

Last Updated: Jul 30, 2025

High-resolution Single Particle Analysis from Electron Cryo-microscopy Images Using SPHIRE
13:28

High-resolution Single Particle Analysis from Electron Cryo-microscopy Images Using SPHIRE

Published on: May 16, 2017

50.4K
Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations
12:27

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations

Published on: February 15, 2017

7.0K
ExCYT: A Graphical User Interface for Streamlining Analysis of High-Dimensional Cytometry Data
05:12

ExCYT: A Graphical User Interface for Streamlining Analysis of High-Dimensional Cytometry Data

Published on: January 16, 2019

11.5K

Area of Science:

  • Computer Science
  • Data Management

Background:

  • The Small Files Problem in Hadoop Distributed File System (HDFS) poses significant challenges to memory and computing efficiency.
  • Existing solutions for managing file sizes in HDFS have limitations.

Purpose of the Study:

  • To propose a novel approach for mitigating the Small Files Problem in HDFS.
  • To introduce a method for identifying and merging small CSV files to optimize storage.

Main Methods:

  • A Hierarchical Clustering Algorithm was developed to analyze file structures.
  • Dendrogram analysis was employed to determine file merge suitability.
  • The algorithm was simulated using 100 CSV files and 20 non-CSV files.

Main Results:

  • The algorithm successfully identified seven appropriate CSV files for merging.
  • Merging these files led to a reduction in memory space within HDFS.
  • The simulation demonstrated the algorithm's effectiveness in managing CSV file sizes.

Conclusions:

  • The proposed Hierarchical Clustering Algorithm offers an efficient solution for the Small Files Problem in HDFS.
  • This approach optimizes memory usage and enhances overall file management in distributed systems.
  • The method shows promise for improving the performance of HDFS with large volumes of small files.