iRODS 4.0 and Beyond (DDN UK User Group Meeting, September 2014)
1 like609 views
These slides present a vision of iRODS configuration management and the technologies that have been implemented to this point in order to realize that vision.
iRODS 4.0 and Beyond (DDN UK User Group Meeting, September 2014)
- 1. iRODS 4.0 and Beyond Presented at the iRODS & DDN User Group Meeting 2014 September 8, 2014 iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 1
- 2. Motivation and Goals iRODS is open source middleware for: Data Discovery, Workflow Automation, Secure Collaboration, and Data Virtualization We want it to be sustainable. iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 2 That means it has to be: Reliable Supportable Usable Scalable
- 3. A Framework for Sustainable iRODS iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 3 Consortium Messaging Technology
- 4. The Vision for iRODS Administration Knowing Your Requirements, Map Out Your Grid: Where does the user interact with the system? Where does the system interact with your storage? What functionality is required from each node? Translate Your Map into a Configuration Updating the Configuration Updates Your Map iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 4 When You Need Help
- 5. A Technology Framework for Sustainable iRODS iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 5 Pluggable Architecture Configuration Management Grid Introspection Plugin Dependency Model A Feature Set That Remains Relevant
- 6. iRODS 4.0: A First Step Authen5ca5on Mechanisms Network Services Microservices iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 6 Pluggable Architecture Simplifies development, testing, support Broadens developer community Makes long-term relevance more likely. Databases Storage Resources APIs
- 7. iRODS 4.0: Plugins, Presently iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 7 Resources: Compound Deferred Load Balanced MSO Non-Blocking Passthru Random Replication Round Robin Universal MSS Unix File System WOS HPSS S3 Authentication: Native PAM OSAuth GSI Kerberos Network: TCP SSL Database: PostgreSQL Oracle MySQL
- 8. iRODS 4.0: Look What You Can Do with Composable Resources! iadmin mkresc $newResc unixfilesystem $newPath! iadmin mkresc replResc replication! iadmin addchildtoresc replResc $demoResc! iadmin addchildtoresc replResc $newResc! iadmin modresc replResc rebalance! iadmin rmchildfromresc replResc $demoResc! iadmin rmchildfromresc replResc $newResc! iadmin rmresc replResc! itrim -rM -N 1 -S $demoResc /$zoneName! iadmin rmresc $demoResc! iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 8 Live storage migration
- 9. iRODS 4.0: Continuous Integration iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 9 Why Build and Test? Transparency (in both process and product) Use existing industry best practices Coverage -> Confidence in Refactoring Packaging -> Ease of installation and upgrade Test framework idempotency Test independence Topology awareness Automation, Automation, Automation
- 10. iRODS 4.0: Continuous Integration iRODS 4.0 transitioned legacy Perl-based test framework to a Python-based iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 10 framework using unittest. We have increased code coverage from ~20% to ~57% Status always visible at http://ci-dev.renci.org/hudson/view/iRODS Currently in Continuous Integration OSes: Ubuntu 10 and 12, CentOS 5 and 6, SuSE 11 and 12 Databases: PostgreSQL, MySQL, Oracle Microservice plugins Authentication plugins: Native, PAM, OSAuth Network plugins: TCP, SSL Resource plugins: Compound, Random, Replication, RoundRobin, Passthru, Deferred, Unixfilesystem, UnivMSS, S3, WOS, MockArchive, NonBlocking
- 11. Continuous Integration: Coverity Enterprise code defect analysis tool Catches memory leaks, potential attack vectors, and unintended iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 11 behavior Assigns severity Not all detected defects are necessarily defective We have addressed all high severity issues. Evaluating and eliminating remaining defects. Status available at https://scan.coverity.com/projects/2605
- 12. iRODS 4.0.x: Point Releases iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 12 Fixing things Security issues Memory leaks Bugs Implementing useful features Run-in-place and OSX support Additional options on initial configuration Microservice templates C API support Review and testing of system microservices
- 13. iRODS 4.1: More Groundwork Operating roadmap: https://github.com/irods/irods/issues?q=is%3Aopen+is%3Aissue +milestone%3A4.1.0 First Hints of Configuration Management and Grid Introspection JSON-based configuration describe a grid Infrastructure to support grid report iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 13 New API Interface Easier implementation, support for more languages Specific User Community Requests Data/metadata ingest as an atomic operation Key-value passthrough btw. iCommands and plugins Improved support for file streaming and PEPs Additional Useful Features Pluggable parallel transfer Inter-zone metadata copying Making configuration more straightforward
- 14. iRODS 4.1: Enabling Grid Report Query the Entire Grid (Privileges Determine Level of Detail) What nodes are connected? What storage resources are connected? Are the storage resources alive? What plugins are installed? What iRODS component versions are installed? Goes Hand-in-Hand with Configuration Management Grid report can be used to replicate the grid iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 14
- 15. Beyond iRODS 4.1 iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 15 Further steps toward the vision: Registry to track installed plugins Plugin dependency model Visual interfaces for configuration and reporting Additional important features: Infrastructure to support metadata templating Bugfixes
- 16. The iRODS Ecosystem: DFC Contributions The DataNet Federation Consortium is creating national-scale research data federations. Ongoing iRODS Development: Clients and Client Interfaces iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 16 Jargon and iDrop Web 3 Modeshape (plus WebDAV) plugin Messaging Interface Integration with Elastic Search Metadata Templating and Ontology Discovery This material is based upon work supported by the National Science Foundation under Cooperative Agreement OCI-0940841.
- 17. Beyond Technical: Documentation By Spring 2015 Reference implementations: Genomics Research Library/Archive System Architecture and Administration Manuals System Administration Videos Training and Certification Curriculum System Architect Datagrid Administrator Support Technician Developer iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 17
- 18. Beyond Technical: Consortium Activities iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 18 Products: Membership Professional Services Support Services Training and Certification
- 19. Thank you! Dan Bedard iRODS Market Development Manager danb@renci.org iRODS 4.0 and Beyond -- iRODS & DDN User Group Meeting 2014 19