Re: [ANNOUNCE] TONITE(Thurs)= dsub & Docker to batch Google Cloud (GCP) @St. L Linux Users Group (STLLINUX)

TOPIC: dsub & Docker to batch Google Cloud Platform (GCP) Presenter: Robert Citek (https://www.sluug.org/bio/Robert_Citek <https://www.google.com/url?q=https://www.sluug.org/bio/Robert_Citek&sa=D&source=calendar&usd=2&usg=AOvVaw1LFKPI0lZ7mn0ZFQulj6HD> ) *From Months to Hours: * * Batch Processing with Docker & dsub on Google Cloud Platform* This presentation will showcase how dsub, a powerful command-line tool leveraging Docker and Google Cloud Platform (GCP), enables highly parallel and cost-effective execution of batch processing jobs in the cloud. We will explore a real-world case study involving the conversion of 10,000 images from a proprietary format to TIFF. This project, which would have been prohibitively time-consuming and expensive on a single system or even a small in-house cluster, was completed in hours for a small fraction of the cost using dsub on GCP. The talk will provide an overview of the project's challenges and how dsub's capabilities in job submission, resource management, and parallelization were utilized to achieve significant time and cost savings. Following the project overview, a live demonstration of a small-scale dsub workflow on GCP will illustrate the tool's ease of use and potential for accelerating various computational tasks. Here are the links: https://github.com/DataBiosphere/dsub <https://www.google.com/url?q=https://github.com/DataBiosphere/dsub&sa=D&source=calendar&usd=2&usg=AOvVaw0dYM0CTtzG4fIlziZ8iEOC> https://hub.docker.com/r/rwcitek/dsub/tags <https://www.google.com/url?q=https://hub.docker.com/r/rwcitek/dsub/tags&sa=D&source=calendar&usd=2&usg=AOvVaw3j2HJ7Fa2v7Ipxe0RLf8sy> ========================================= From GENERIC DESCRIPTIONS: dsub currently supports the Cloud Life Sciences v2beta (https://cloud.google.com/life-sciences/docs/reference/rest <https://www.google.com/url?q=https://cloud.google.com/life-sciences/docs/reference/rest&sa=D&source=calendar&usd=2&usg=AOvVaw3QhnLoo7-C5ZsXkrYtgfEm>) API from Google Cloud and is is developing support for the Batch ( https://cloud.google.com/batch/docs/reference/rest <https://www.google.com/url?q=https://cloud.google.com/batch/docs/reference/rest&sa=D&source=calendar&usd=2&usg=AOvVaw0MnSNLNq8ZL7DClrb46mKm>) API from Google Cloud. dsub: simple batch jobs with Docker ( https://github.com/DataBiosphere/dsub#dsub-simple-batch-jobs-with-docker) <https://www.google.com/url?q=https://github.com/DataBiosphere/dsub%23dsub-simple-batch-jobs-with-docker)License&sa=D&source=calendar&usd=2&usg=AOvVaw386m7s6H_-TUSs82VAOHBy> License <https://www.google.com/url?q=https://github.com/DataBiosphere/dsub%23dsub-simple-batch-jobs-with-docker)License&sa=D&source=calendar&usd=2&usg=AOvVaw386m7s6H_-TUSs82VAOHBy> (https://github.com/DataBiosphere/dsub/blob/main/LICENSE <https://www.google.com/url?q=https://github.com/DataBiosphere/dsub/blob/main/LICENSE&sa=D&source=calendar&usd=2&usg=AOvVaw1iT94JaRJXr4TTHpO8jouX>) Overview (https://github.com/DataBiosphere/dsub#overview)dsub <https://www.google.com/url?q=https://github.com/DataBiosphere/dsub%23overview)dsub&sa=D&source=calendar&usd=2&usg=AOvVaw0_BMP7aJ3xqDaTPqrUMoxl> is a command-line tool that makes it easy to submit and run batch scripts in the cloud. The dsub user experience is modeled after traditional high-performance computing job schedulers like Grid Engine and Slurm. You write a script and then submit it to a job scheduler from a shell prompt on your local machine. Today dsub supports Google Cloud as the backend batch job runner, along with a local provider for development and testing. With help from the community, we'd like to add other backends, such as a Grid Engine, Slurm, Amazon Batch, and Azure Batch. ====================================== On Thu, Mar 20, 2025 at 9:12 AM SLUUG Announcement List < announce(a)sluug.org&gt; wrote: