PWV Portfolio Feed

5/ Key Highlights contd:
☁️ Simplified cloud-native data access for scientific researchers and data engineers
💰 Potential to save millions in storage and compute costs annually when scaled to a multi-Petabyte data archive

4/ Key Highlights:
🧊 Transformed 1.4 million NetCDF files into a single, analysis-ready data cube *without* duplicating any data
⚡ Reduced time series extraction time from 5 minutes to 3 seconds

3/ With this approach, we were able to extract a month-long time series from GPM IMERG precipitation data stored in S3 in 3 seconds, compared to 5 minutes in the previous cloud-based method.

2/ In the pilot, we used our new open source tensor storage engine #Icechunk and #VirtualiZarr to present archival NetCDF data stored in S3 as a single analysis-ready cloud-optimized (ARCO) dataset.

Amazing! Would be super cool if those additional data points could be funneled back into MusicBrainz somehow :)

He’s right, your rebase also strips extra headers. Do you know where the whitelist of headers is hard coded?

do it i hear it's lovely!

every day i try to not be the CTO that everyone complains about to their coworkers

Any feedback is welcome and encouraged. Anything that isn't obvious and/or not yet documented are good candidates.

Re license: Still deciding on the details. I want to evaluate whether free for non-commercial use and license for commercial use is an option.

Happy to discuss more via Discord DMs.

Looking forward to your feedback! Happy to help! 🙏

my one shard of hope in politics is that mamdani is the next nyc mayor, if cuomo wins idk what is left

Glad you like it! Yes, LiveStore supports syncing via CF durable objects and persists to D1. Give it a try!

Syncing with an existing DB can really feel like hitting a WAL.