ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Poster for ISGC

•
•
D
Data Finder

The DataFinder is a data management application that supports organizing, describing, and automating access to large datasets produced during experiments and stored across grids and clouds. It provides a unified interface for various backend data stores, allowing easy management and transfer of data between grid and cloud resources. The DataFinder supports various backends including webDAV, FTP, local file systems, cloud storage services like Amazon S3, and gridFTP servers, giving users flexibility in storing data. It helps researchers and small companies archive and access technical and scientific data generated by simulations distributed across computational resources.

1 of 1
Download to read offline
DataFinder - A Data Management Application for Grids and Clouds Why develop the How are Cloud and Grid DataFinder? connected with the Department DataFinder? Employee When conducting experiments Simulation Use Cases: large datasets are produced. The same user interface is Using a grid to The DataFinder supports Geometry provided for all data stores, so calculate and execute therefore: users have uniform access to Grid Generation Geometry, Flow • organizing large data sets backends for Grid and Cloud Flow Solution Simulation,… and also • handling describing meta storage. This allows easy data storing Data in Grids or Visualisation Clouds data management throughout the • execution of scripts for the whole virtual storage space, automation of common for example copying and procedures, like simulations moving data between Grid and visualizations of and Cloud resources. Wizards: Help to configure Use experiment conditions Cases and manage the With the DataFinder the What kind of backends Data Access produced datasets are easily does the Datafinder accessible and processable. support? Who could use the A major concept of DataFinder? DataFinder is to store data on different backends: Application: A typical use case is • WebDAV server Datafinder (User archiving of data on Interface) • FTP und GridFTP server For executing scripts in Cloud storage. Even if the • local file system grids and for accessing computation producing the • Tivoli Storage Manager and managing data data is performed locally (TSM) stored on different or in the Grid, storing on • for higher flexibility: backends a lightweighted storage Amazon Simple Storage backend is possible. Service (S3) The DataFinder could be The user has the option especially interesting for to either store his data on smaller companies or a classical server, on Grid institutions which do not resources, or in the Amazon want to maintain their own Cloud. External Medias (CD, DVD,…) Meta Data Server archiving infrastructure. Storage System: All possible Example Use Case Backends, that can D-Grid Project: „AeroGrid“ be used, including • Technical and scientific data File System Grids and Clouds management on distributed resources ���� (Grid and Cloud) • Documentation of CFD simulations ���� FTP Server • Assure quality of results and trace S3 Cloud- Storage System errors WebDAV Server • Repeatability of simulations • User Front End „Datafinder“ (Python/ PyQT) • SimulationCode: Cluster optimized Storage Resource CFD code „Trace“ (C++) GridFTP Server Broker • Jobs with Globus and UNICORE Tivoli Storage Manager Deutsches Zentrum Miriam Ney für Luft- und Raumfahrt e.V. Miriam.Ney@dlr.de in der Helmholtz-Gemeinschaft Andreas Schreiber Simulations- und Softwaretechnik Andreas.Schreiber@dlr.de Köln/ Braunschweig/ Berlin www.dlr.de/sc/datafinder Simulation and Software Technology

More Related Content

Poster for ISGC

  • 1. DataFinder - A Data Management Application for Grids and Clouds Why develop the How are Cloud and Grid DataFinder? connected with the Department DataFinder? Employee When conducting experiments Simulation Use Cases: large datasets are produced. The same user interface is Using a grid to The DataFinder supports Geometry provided for all data stores, so calculate and execute therefore: users have uniform access to Grid Generation Geometry, Flow • organizing large data sets backends for Grid and Cloud Flow Solution Simulation,… and also • handling describing meta storage. This allows easy data storing Data in Grids or Visualisation Clouds data management throughout the • execution of scripts for the whole virtual storage space, automation of common for example copying and procedures, like simulations moving data between Grid and visualizations of and Cloud resources. Wizards: Help to configure Use experiment conditions Cases and manage the With the DataFinder the What kind of backends Data Access produced datasets are easily does the Datafinder accessible and processable. support? Who could use the A major concept of DataFinder? DataFinder is to store data on different backends: Application: A typical use case is • WebDAV server Datafinder (User archiving of data on Interface) • FTP und GridFTP server For executing scripts in Cloud storage. Even if the • local file system grids and for accessing computation producing the • Tivoli Storage Manager and managing data data is performed locally (TSM) stored on different or in the Grid, storing on • for higher flexibility: backends a lightweighted storage Amazon Simple Storage backend is possible. Service (S3) The DataFinder could be The user has the option especially interesting for to either store his data on smaller companies or a classical server, on Grid institutions which do not resources, or in the Amazon want to maintain their own Cloud. External Medias (CD, DVD,…) Meta Data Server archiving infrastructure. Storage System: All possible Example Use Case Backends, that can D-Grid Project: „AeroGrid“ be used, including • Technical and scientific data File System Grids and Clouds management on distributed resources ���� (Grid and Cloud) • Documentation of CFD simulations ���� FTP Server • Assure quality of results and trace S3 Cloud- Storage System errors WebDAV Server • Repeatability of simulations • User Front End „Datafinder“ (Python/ PyQT) • SimulationCode: Cluster optimized Storage Resource CFD code „Trace“ (C++) GridFTP Server Broker • Jobs with Globus and UNICORE Tivoli Storage Manager Deutsches Zentrum Miriam Ney für Luft- und Raumfahrt e.V. Miriam.Ney@dlr.de in der Helmholtz-Gemeinschaft Andreas Schreiber Simulations- und Softwaretechnik Andreas.Schreiber@dlr.de Köln/ Braunschweig/ Berlin www.dlr.de/sc/datafinder Simulation and Software Technology