Is the quickest route to successful AI an optimized data path? Sponsored Content by DDN
AI and DL are drivers of data complexity
The center of gravity in high performance computing (HPC) and enterprise processing has shifted to data. Artificial Intelligence (AI), Deep Learning (DL) and Machine Learning (ML) are essential business and research tools giving organizations valuable insights into their data and doing so with unprecedented velocity and accuracy. Enterprises, universities and government are investing tremendous resources to further develop AI, DL and ML. AI is used by many organizations and across many industries such as healthcare, data security, personalized marketing, and financial services as well as new areas including autonomous vehicles, augmented reality, and natural language recognition.
AI, DL and ML create some of the toughest workloads in computing history—they are increasing amounts of data to be analyzed and stored in terabyte up to petabyte ranges typically used in HPC. IT infrastructure must scale rapidly and efficiently as workloads grow. According to data collected by 451 Research 1], “A key to managing and aggregating volume of data lies in managing storage tiers. 63% of survey respondents want to improve storage efficiency. With the adoption of AI/ML workloads, the storage layer is expected to play a crucial role in the IT environment.”
IT infrastructure and storage challenges with AI and DL
Current IT datacenter infrastructures are designed for business application workloads and are inadequate in handling the demanding needs of AI and DL. On-premise datacenters often have minimal scalability and are designed to handle limited, modest workloads and small data volumes. Their networks are often low performing using poorly optimized filesystems with processing performed on central processing units (CPUs) and storage done mostly on hard disk drives (HDDs). This infrastructure leads to processing pipeline congestion and I/O bottlenecks.
Existing storage solutions in datacenters may also face limitations. AI and DL run best on Graphical Processing Units (GPUs) which are more scalable and faster than CPUs. However, without the right data storage platform, a GPU-based computing platform is just as bottlenecked and ineffectual as an antiquated non-AI-enabled datacenter.
Meeting the needs of AI-enabled storage
The proper selection of a data storage platform and its efficient integration in the datacenter infrastructure are key to eliminating AI bottlenecks and truly accelerating time to insight. AI/DL storage requirements include:
- Scalability: Infrastructure that is flexible and scalable in implementation and enables efficient handling of a wide breadth of data sizes and types.
- High performance parallelism: High parallelized architecture that delivers data simultaneously to all processes running within the GPUs to eliminate waiting for data transactions.
- Highly concurrent random streaming: Flexibility in data flows using a shared storage system as a common file repository.
- Parallelized Filesystem: Architecture must use parallelized file system versus NFS-based file storage system prone to severe traffic contention when engaged from multiple clients.
- Security: Storage system must provide high data availability, maximum system uptime and be integrated as a fully redundant system.
- Low Latency: Network must be optimized to process a very high volume of data messages with minimal delay (latency).
Introducing DDN –AI-enabled storage
DDN is the world’s largest privately held data storage company and the leading supplier to data-intensive, global organizations. DDN has successfully deployed data-at-scale systems across all areas of AI and DL with their parallel storage solutions such as DDN A3I (Accelerated, Any-Scale AI) whose value is shown in Figure 1. “DDN has done extensive work to simplify scalable performance storage that really delivers for production AI storage workloads,” said William Beaudin, Director of AI Architecture at DDN. “Not only can an A3I storage solution be deployed in a matter of hours, but they also scale easily to meet performance and capacity requirements no matter how large or diverse.”
Using on-premise storage versus public, hybrid or private cloud
DDN designs optimized solutions enable organizations to generate value and accelerate time to insight from their data, both on premise and in the cloud (public, hybrid or private cloud). In many cases companies can take advantage of AI performed against their existing cloud-based applications, like Salesforce and SAP to support sales, finance, procurement, and supply chain optimization. In other cases, companies can apply ready-made applications for occasions where they may lack internal data science expertise. These applications can take in moderate amounts of data for workloads from cyber security to drug discovery. Yet, most of these instances will be limited to smaller sets of relatively structured data. The process of gathering, transferring and storing the massive amounts of data required for larger AI projects will continue to be a barrier for HPC-like AI and Deep Learning.
However, there are valid reasons for using on-premise storage versus cloud options. Industries such as financial services and healthcare are subject to stringent government regulations which may prohibit storing data on the cloud. Organizations should consider using on-premise storage for these instances 2]:
- Locality of data and compute: On-premise storage works best when data needs to be accessed frequently and quickly
- Data gravity: AI processing involves large amounts of data which may not be movable over cloud
- SLA or Response Times: Organizations with stringent SLAs or certain response times
Summary: Benefits of an optimized data storage path for AI
Artificial Intelligence (AI) and Deep Learning (DL) are essential business and research tools which provide organizations with valuable insights into their data. However, the complexity of processing and storing this data requires an optimized data path and a storage solution capable of handling AI and DL workloads.
DDN is a leading storage system provider that can meet AI and DL storage needs. “DDN has over twenty years of experience in dealing with at-scale data challenges and has been at the forefront of the resurging interest in AI and DL,” said Kurt Kuckein, senior director of marketing at DDN. “Our engineering and services organizations have thousands of hours working on the largest data challenges in the world, and we have managed to compile all that knowledge into infrastructure that makes customer projects faster and more successful.”
References
1] 451 Research – Business Impact Brief: Automating Storage Tiers Can Drive Faster, Deeper Analytical Insight
2] Cloud Storage vs. In-House: 12 Reasons to Stay On-Prem, Enterprise Storage Forum, https://www.enterprisestorageforum.com/backup-recovery/cloud-storage-vs.-in-house-12-reasons-to-stay-on-prem.html
Automating Storage Tiers Can Drive Faster, Deeper Analytical Insight, https://www.ddn.com/download/automating-storage-tiers-can-drive-faster-deeper-analytical-insight/
About DDN
DDN is the world’s leading data management supplier to data-intensive, global organizations. The rapidly evolving competitive landscape makes it essential to ensure projects like AI initiatives can move quickly from investigation to production. For more than 20 years, DDN has focused on designing, deploying and optimizing solutions for production level AI, HPC and Big Data. DDN enables businesses to generate more value and accelerate time to insight from their data, on-premise and in multicloud environments. Organizations leverage the power of DDN technology and technical expertise to capture, store, process, analyze, collaborate and distribute information and content in the most efficient, reliable and cost-effective manner. DDN customers include many of the world’s leading financial services firms, banks, healthcare and life science organizations, manufacturing and energy companies, government and research facilities, and service providers who use their data to develop everything from innovative treatments for disease to new paths to revenue.
Contact DDN
DDN has long been a partner of choice for organizations pursuing data-intensive projects at any scale. DDN provides significant technical expertise through its global research and development and field technical organizations. A worldwide team with hundreds of engineers and technical experts can be called upon to optimize every phase of a project: initial inception, solution architecture, systems deployment, customer support and future scaling needs. DDN laboratories are also equipped with leading GPU compute platforms to provide unique benchmarking and testing capabilities for AI and DL applications.
Strong customer focus coupled with technical excellence and deep field experience ensures that DDN delivers the best possible solution for any challenge. Taking a consultative approach, DDN experts perform an in-depth evaluation of requirements and provide application-level optimization of data workflows for a project. They will then design and propose an optimized, highly reliable and easy-to-use solution that accelerates the customer’s effort.
Contact DDN today and engage our team of experts to unleash the power of your AI projects.
Copyright 2019
